TSDB enabled release for kafka #6277

agithomas · 2023-05-22T08:36:47Z

Enhancement

What does this PR do?

Added TSDB related changes as beta release for Redis package

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2023-05-22T08:50:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-23T14:21:58.383+0000
Duration: 16 min 7 sec

Test stats 🧪

Test	Results
Failed	0
Passed	31
Skipped	0
Total	31

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-22T08:51:02Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	72.222% (`13/18`)	👎 -26.679
Lines	77.778% (`56/72`)	👎 -15.998
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-06-14T05:35:28Z

💚 Build Succeeded

Buildkite Build
Commit: 5b465fb

ritalwar

LGTM!

…rations into issue-6152-kafka

ritalwar

LGTM!

lalit-satapathy

LGTM

elasticmachine · 2023-06-27T09:45:14Z

Package kafka - 1.8.0 containing this change is available at https://epr.elastic.co/search?package=kafka

…ifest Add a comment noting that secret handling should be considered once Fleet Server resolves issue #6277. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com>

@strawgate

* Add Prometheus OpenTelemetry Input Package This package allows scraping Prometheus-compatible metrics endpoints using the OpenTelemetry Collector's Prometheus receiver. Features: - Scrape multiple Prometheus endpoints - Configurable scrape interval and timeout - Support for HTTP and HTTPS schemes - Honor labels and timestamps options * fix issue number * add tests * fix prometheus policy and system tests * fix review * fix review * Update packages/prometheus_input_otel/_dev/deploy/docker/docker-compose.yml Co-authored-by: Mykola Kmet <mykola.kmet@elastic.co> * add new logo and fix docs * upgrade prometheus version used in docker, update readme examples * add basic auth * fix handle bar indentation * adding sample event * Update .github/CODEOWNERS * Update packages/prometheus_input_otel/_dev/test/system/test-default-config.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Update packages/prometheus_input_otel/_dev/deploy/docker/docker-compose.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Update packages/prometheus_input_otel/manifest.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Add Prometheus scrape config migration example to README Shows how an existing Prometheus scrape config can be migrated to the package configuration with a one-to-one mapping table. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * move test asset under the config folder * Update README to show literal receiver config and link to upstream docs - Replace mapping table with actual receiver config output - Link to upstream OTel Prometheus receiver documentation - Streamline documentation following other OTel input packages Addresses feedback from @strawgate and @ishleenk17 Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Add OTel telescope badge to logo and Raw Config policy template - Updated package logo to include OpenTelemetry telescope badge overlay, following the pattern used by system_otel and mysql_otel packages - Added "Prometheus Metrics (Raw Config)" policy template that allows users to paste their existing Prometheus scrape_configs YAML directly - Renamed existing template to "Prometheus Metrics (Guided)" for clarity - Added input_raw.yml.hbs template for raw config mode - Added policy test for the raw config template - Updated README to document both configuration modes Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Add discovery field to prometheus_input_otel manifest Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Remove discovery field from prometheus_input_otel manifest The discovery field should only be used in content packages, not input packages. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Fix system test by specifying policy_template The system test was failing because there are multiple policy templates (prometheusreceiver and prometheusreceiver_raw) with the same input type 'otelcol'. Adding 'policy_template: prometheusreceiver' disambiguates which template the test should use. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Split prometheus_input_otel into two packages - prometheus_input_otel: Guided configuration with individual settings - prometheus_input_otel_raw: Raw config mode for pasting existing scrape configs Both packages are owned by elastic/ecosystem team. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Fix YAML indentation in prometheus_input_otel_raw template The scrape_configs variable was being rendered on a new line at column 0, causing invalid YAML. Fixed by: 1. Placing {{scrape_configs}} on the same line as 'scrape_configs:' 2. Updating test files to use proper YAML array format instead of string literal 3. Adding policy_template field to system test config Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Update packages/prometheus_input_otel_raw/_dev/test/system/test-default-config.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Update packages/prometheus_input_otel_raw/_dev/test/policy/test-default.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Update packages/prometheus_input_otel_raw/manifest.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Update packages/prometheus_input_otel_raw/_dev/test/system/test-default-config.yml Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> * Update packages/prometheus_input_otel_raw/_dev/test/system/test-default-config.yml Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update packages/prometheus_input_otel_raw/_dev/test/policy/test-default.expected Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update packages/prometheus_input_otel_raw/sample_event.json Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add additional policy test coverage for Prometheus OTel packages - prometheus_input_otel: - test-tls.yml: Tests HTTPS with TLS configuration - test-basic-auth.yml: Tests basic authentication - test-multiple-targets.yml: Tests multiple scrape targets with custom metrics path - prometheus_input_otel_raw: - test-multiple-jobs.yml: Tests multiple Prometheus scrape jobs - test-advanced-config.yml: Tests HTTPS with TLS and basic auth in raw config Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Fix policy test expected output for secret password handling The password field has secret: true in manifest.yml, which causes Fleet to replace passwords with secret references (${SECRET_0}) rather than literal values. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Update package titles for Prometheus OTel packages - prometheus_input_otel: 'Prometheus OTel Scrape (Guided)' - prometheus_input_otel_raw: 'Prometheus OTel Scrape (Bring Your Own Config)' - Updated cross-references in documentation Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Add system tests with basic auth for Prometheus OTel packages Add system tests that exercise basic authentication with secret password handling: prometheus_input_otel (Guided): - Add prometheus_basic_auth docker service with basic auth enabled - Add web-basic-auth.yml config with bcrypt password hash - Add test-basic-auth-config.yml system test with username/password prometheus_input_otel_raw (BYOC): - Add prometheus_basic_auth docker service - Add test-basic-auth-config.yml testing basic_auth in raw scrape config This addresses the request to test the secret: true handling for password fields in OTel input packages. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Add skip annotation to basic auth system test for secret password issue The `secret: true` setting on password fields causes authentication to fail in OTel input packages. The password is not resolved correctly by the OTel receiver, appearing as `$co.elastic.secret{...}` instead of the actual password value. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Disable secret handling for password field due to Fleet Server bug The `secret: true` setting on password fields causes authentication to fail in OTel input packages because Fleet Server doesn't correctly resolve the secrets for the OTel collector. Changed to `secret: false` with a TODO comment to re-enable once elastic/fleet-server#6277 is resolved. - Updated manifest.yml to set secret: false with TODO comment - Updated test-basic-auth.expected to expect plain password (not ${SECRET_0}) - Removed skip annotation from test-basic-auth-config.yml system test Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> * Add TODO comment for secret handling in prometheus_input_otel_raw manifest Add a comment noting that secret handling should be considered once Fleet Server resolves issue #6277. Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> --------- Co-authored-by: Mykola Kmet <mykola.kmet@elastic.co> Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Julien Lind <jlind23@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

agithomas added 2 commits May 22, 2023 08:36

TSDB based beta release for kafka

33016c1

Updated PR link

916b502

agithomas mentioned this pull request May 22, 2023

[Meta][TSDB] TSDB Release for multiple packages #6152

Closed

agithomas added 2 commits May 24, 2023 06:29

Updated changelog, dimension field limit

3a356fc

Updated the manifest version

9fb33ea

agithomas changed the title ~~TSDB based beta release for kafka~~ TSDB enabled release for kafka Jun 12, 2023

agithomas and others added 3 commits June 12, 2023 10:14

Updated changelog for TSDB enablement

0354e6a

Merge branch 'main' into issue-6152-kafka

c2a8edf

Merge branch 'main' into issue-6152-kafka

5b465fb

ritalwar approved these changes Jun 14, 2023

View reviewed changes

agithomas added 2 commits June 23, 2023 14:20

Updated changelog with TSDB related description

68db8c4

Merge branch 'issue-6152-kafka' of https://github.com/agithomas/integ…

505528f

…rations into issue-6152-kafka

agithomas marked this pull request as ready for review June 26, 2023 05:00

agithomas requested a review from a team as a code owner June 26, 2023 05:00

ritalwar approved these changes Jun 26, 2023

View reviewed changes

lalit-satapathy approved these changes Jun 26, 2023

View reviewed changes

agithomas merged commit 1e93b7c into elastic:main Jun 27, 2023

andrewkroh added the Integration:kafka Kafka label Jul 22, 2024

agithomas deleted the issue-6152-kafka branch February 12, 2025 05:01

github-actions bot mentioned this pull request Feb 3, 2026

Add Prometheus OpenTelemetry Input Package #17117

Merged

This was referenced Feb 6, 2026

Add new Redis OTel input package #17281

Closed

Add redis otel input package #17285

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TSDB enabled release for kafka #6277

TSDB enabled release for kafka #6277

Uh oh!

agithomas commented May 22, 2023

Uh oh!

elasticmachine commented May 22, 2023 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented May 22, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Jun 14, 2023

Uh oh!

ritalwar left a comment

Uh oh!

ritalwar left a comment

Uh oh!

lalit-satapathy left a comment

Uh oh!

elasticmachine commented Jun 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

TSDB enabled release for kafka #6277

TSDB enabled release for kafka #6277

Uh oh!

Conversation

agithomas commented May 22, 2023

What does this PR do?

Checklist

Uh oh!

elasticmachine commented May 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented May 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

elasticmachine commented Jun 14, 2023

💚 Build Succeeded

Uh oh!

ritalwar left a comment

Choose a reason for hiding this comment

Uh oh!

ritalwar left a comment

Choose a reason for hiding this comment

Uh oh!

lalit-satapathy left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented May 22, 2023 •

edited

Loading

elasticmachine commented May 22, 2023 •

edited

Loading