Skip to content

Add end-to-end Test_ExtendedHeartbeat for app-extended-heartbeat telemetry event#6338

Draft
khanayan123 wants to merge 53 commits intomainfrom
ayan.khan/app-extended-heartbeat
Draft

Add end-to-end Test_ExtendedHeartbeat for app-extended-heartbeat telemetry event#6338
khanayan123 wants to merge 53 commits intomainfrom
ayan.khan/app-extended-heartbeat

Conversation

@khanayan123
Copy link
Copy Markdown
Contributor

@khanayan123 khanayan123 commented Feb 18, 2026

Summary

Adds a new TELEMETRY_EXTENDED_HEARTBEAT end-to-end scenario and Test_ExtendedHeartbeat test that validates the app-extended-heartbeat telemetry event includes every config previously reported in app-started or app-client-configuration-change.

Changes

New scenario

  • utils/_context/_scenarios/__init__.py: TELEMETRY_EXTENDED_HEARTBEAT scenario with shortened intervals (DD_TELEMETRY_HEARTBEAT_INTERVAL=1, DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL=2, plus _DD_* alias) so the extended heartbeat event fires within the test window.

New test

  • tests/test_telemetry.py: Test_ExtendedHeartbeat — gated by the existing @features.app_extended_heartbeat_event decorator. Collects config names from all app-started and app-client-configuration-change events, then asserts every one appears in at least one app-extended-heartbeat event across the stream (order-agnostic, to handle forked workers and remote-config re-reports).

Manifest version gates

Enabled at the following tracer versions:

  • cpp_httpd: >1.0.4
  • cpp_nginx: >1.14.0
  • dotnet: v3.39.0
  • golang: v2.6.1
  • java: v1.23.0
  • nodejs: >=5.97.0 (new ref_5_97_0 anchor added)
  • php: v1.19.0
  • python: v4.6.5
  • ruby: >2.30.0

Add comprehensive parametric test suite for app-extended-heartbeat
telemetry event across all SDK languages. Tests verify timing,
payload structure, consistency, and compliance with API spec.

Tests added:
- test_extended_heartbeat_emission: Verifies interval timing
- test_extended_heartbeat_sequence: Validates multiple events
- test_extended_heartbeat_payload_content: Checks required fields
- test_extended_heartbeat_matches_app_started: Ensures consistency
- test_extended_heartbeat_excludes_products_and_install_signature
- test_extended_heartbeat_default_interval: Validates 24h default

All tests run parametrically across 9 SDK languages (Go, Java, .NET,
C++, PHP, Ruby, Rust, Python, Node.js).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 18, 2026

CODEOWNERS have been resolved as:

.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/test_telemetry.py                                                 @DataDog/libdatadog-telemetry @DataDog/apm-sdk-capabilities @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

khanayan123 and others added 6 commits February 18, 2026 14:45
Register comprehensive parametric tests for app-extended-heartbeat event
validation across all SDK languages. Tests verify:
- Periodic emission timing (24h default, configurable)
- Payload structure (config, deps, integrations)
- Consistency with app-started event
- Proper field exclusions (products, install_signature)

Language version markers:
- Go: v1.73.0-dev
- Java: v1.40.0
- Node.js: v5.0.0
- Python: v2.0.0
- Ruby: v2.1.0-dev
- C++: v0.2.0-dev
- PHP: v1.0.0-dev
- .NET: missing_feature (intentionally skipped - no forked runtime-id issues)

Tests use fast intervals (0.3-0.5s) for CI efficiency.

Test location: tests/parametric/test_telemetry.py::Test_ExtendedHeartbeat (lines 1256-1514)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed Test_ExtendedHeartbeat manifest entries from development
version strings (e.g., v1.73.0-dev, v2.1.0-dev) to missing_feature
to follow repository conventions.

Development versions should not be specified in manifests. Tests
for unreleased features should use missing_feature until the
feature is actually released in a production version.

Changes:
- cpp.yml: v0.2.0-dev → missing_feature
- golang.yml: v1.73.0-dev → missing_feature
- nodejs.yml: '>=5.0.0' → missing_feature
- php.yml: v1.0.0-dev → missing_feature
- ruby.yml: v2.1.0-dev → missing_feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…tionality

Replace 6 complex test methods with a single simplified test that focuses
on the core requirement: comparing configurations across telemetry events.

The new test:
- Grabs app-started, app-extended-heartbeat, and app-client-configuration-change events
- Extracts configuration data from each
- Asserts configs match between app-started and app-extended-heartbeat
- Optionally validates config-change event configs match as well

This reduces test complexity from ~260 lines to ~40 lines while maintaining
coverage of the critical functionality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Configure manifests to run app-extended-heartbeat tests only on Node.js
for initial validation:

- Enable Node.js: *ref_5_0_0 (>=5.0.0)
- Disable Python: v2.0.0 -> missing_feature
- Disable Java: v1.40.0 -> missing_feature
- Keep all others disabled: missing_feature (cpp, dotnet, php, golang, ruby)

This allows focused testing of Node.js implementation before enabling
other languages.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add SLF001 exception for test_telemetry.py to allow _get_telemetry_event usage
- Fix nodejs.yml manifest to use '>=5.0.0' instead of non-existent anchor
- All linting checks now pass (mypy, ruff, yamlfmt, yamllint, shellcheck)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@datadog-official
Copy link
Copy Markdown

datadog-official Bot commented Feb 24, 2026

Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 4 Tests failed

tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction.test_lfi_get[php-fpm-8.5] from system_tests_suite   View in Datadog   (Fix with Cursor)
assert 505 == 302
 +  where 505 = HttpResponse(status_code:505, headers:{'Date': 'Tue, 28 Apr 2026 02:26:00 GMT', 'Server': 'Apache/2.4.66 (Ubuntu)', 'T...com/product/security-platform/application-security-monitoring/" target="_blank">Datadog</a></p></footer></body></html>).status_code
 +    where HttpResponse(status_code:505, headers:{'Date': 'Tue, 28 Apr 2026 02:26:00 GMT', 'Server': 'Apache/2.4.66 (Ubuntu)', 'T...com/product/security-platform/application-security-monitoring/" target="_blank">Datadog</a></p></footer></body></html>) = <tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction object at 0x7f1010e44200>.r3

self = <tests.appsec.rasp.test_lfi.Test_Lfi_RC_CustomAction object at 0x7f1010e44200>

    def test_lfi_get(self):
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
        assert self.config_state_1b.state == rc.ApplyState.ACKNOWLEDGED
        assert self.r1.status_code == 403
...
tests.appsec.test_automated_login_events.Test_V3_Login_Events_RC.test_rc[php-fpm-8.5] from system_tests_suite   View in Datadog   (Fix with Cursor)
KeyError: '_dd.appsec.events.users.login.success.auto.mode'

self = <tests.appsec.test_automated_login_events.Test_V3_Login_Events_RC object at 0x7f7fc43027b0>

    def test_rc(self):
        def validate_disabled(meta: dict):
            assert "_dd.appsec.events.users.login.success.auto.mode" not in meta
    
        def validate_anon(meta: dict):
            assert meta["_dd.appsec.events.users.login.success.auto.mode"] == "anonymization"
...
tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig.test_block_405[php-fpm-8.5] from system_tests_suite   View in Datadog   (Fix with Cursor)
assert 405 == 505
 +  where 405 = HttpResponse(status_code:405, headers:{'Date': 'Tue, 28 Apr 2026 02:19:04 GMT', 'Server': 'Apache/2.4.66 (Ubuntu)', 'T...customer service team. Security provided by Datadog."}],"security_response_id":"3ed0c9c7-f216-4293-1b3a-19afa190a345"}).status_code
 +    where HttpResponse(status_code:405, headers:{'Date': 'Tue, 28 Apr 2026 02:19:04 GMT', 'Server': 'Apache/2.4.66 (Ubuntu)', 'T...customer service team. Security provided by Datadog."}],"security_response_id":"3ed0c9c7-f216-4293-1b3a-19afa190a345"}) = <tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig object at 0x7f1010d16c60>.response_3

self = <tests.appsec.test_remote_config_rule_changes.Test_BlockingActionChangesWithRemoteConfig object at 0x7f1010d16c60>

    def test_block_405(self):
        # normal block
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
        interfaces.library.assert_waf_attack(self.response_1, rule="ua0-600-56x")
...
tests.appsec.test_suspicious_attacker_blocking.Test_Suspicious_Attacker_Blocking.test_block_suspicious_attacker[php-fpm-8.5] from system_tests_suite   View in Datadog   (Fix with Cursor)
assert 405 == 416
 +  where 405 = HttpResponse(status_code:405, headers:{'Date': 'Tue, 28 Apr 2026 02:24:59 GMT', 'Server': 'Apache/2.4.66 (Ubuntu)', 'T...customer service team. Security provided by Datadog."}],"security_response_id":"66df3184-bae0-4a3d-1b52-d37a6876cf59"}).status_code
 +    where HttpResponse(status_code:405, headers:{'Date': 'Tue, 28 Apr 2026 02:24:59 GMT', 'Server': 'Apache/2.4.66 (Ubuntu)', 'T...customer service team. Security provided by Datadog."}],"security_response_id":"66df3184-bae0-4a3d-1b52-d37a6876cf59"}) = <tests.appsec.test_suspicious_attacker_blocking.Test_Suspicious_Attacker_Blocking object at 0x7f10112b1910>.response_4

self = <tests.appsec.test_suspicious_attacker_blocking.Test_Suspicious_Attacker_Blocking object at 0x7f10112b1910>

    def test_block_suspicious_attacker(self):
        # ASM disabled
        assert self.config_state_1.state == rc.ApplyState.ACKNOWLEDGED
        interfaces.library.assert_no_appsec_event(self.response_1)
...
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c78fad7 | Docs | Datadog PR Page | Give us feedback!

Java tracer supports configurable extended heartbeat interval via
DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL, so enable the parametric
tests.
khanayan123 and others added 3 commits March 30, 2026 22:12
Resolve merge conflicts in manifests/java.yml, manifests/ruby.yml,
and tests/parametric/test_telemetry.py. Keep Test_ExtendedHeartbeat
additions while incorporating main's telemetry key normalization
and stable configuration origin updates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Revert pyproject.toml changes (not needed for this PR) and remove
DD_TELEMETRY_HEARTBEAT_INTERVAL from Test_ExtendedHeartbeat since
we only need to configure the extended heartbeat interval.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use integer interval (1s) for Java compatibility (getLong)
- Assert configs as superset with value matching, order-agnostic
- Build expected config from app-started + config-change overlay
- Collect all config-change events, not just the last one

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/dd-trace-py that referenced this pull request Mar 31, 2026
…eat payload (#17203)

## Summary
Fix the extended heartbeat payload key from `"configurations"` (plural) to `"configuration"` (singular) to match the [telemetry v2 API spec](https://github.com/DataDog/instrumentation-telemetry-api-docs/blob/main/GeneratedDocumentation/ApiDocs/v2/SchemaDocumentation/Schemas/app_extended_heartbeat.md) and align with other SDKs (Java, .NET, Node.js).

## Changes
- **`ddtrace/internal/telemetry/writer.py`**: `payload["configurations"]` → `payload["configuration"]`
- **`tests/telemetry/test_telemetry.py`**: Updated test assertions to match

## Motivation
Cross-SDK system tests validate that `app-extended-heartbeat` payloads use the same schema. The spec and all other SDKs use `"configuration"` (singular). This mismatch would cause system test & dropped telemetry payloads failures for Python.

## Related
- System test PR: DataDog/system-tests#6338
- Original Python implementation: #16628

Co-authored-by: ayan.khan <ayan.khan@datadoghq.com>
khanayan123 and others added 2 commits March 31, 2026 15:23
Resolve merge conflict in manifests/ruby.yml (kept both
Test_ExtendedHeartbeat and updated Test_Stable_Configuration_Origin).
Fix ruff lint issues: D209 docstring closing quotes and SLF001
private member access noqa annotations in test_telemetry.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Collaborator

@cbeauchesne cbeauchesne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already test on heartbeats in DEFAULT scenario.

If you plan to test only one or two set of parameters, then the end-to-end DEFAULT scenario is a best fit from a quality POV as the coverage is by far more complete.

We can chat more about what are the good/bad use cases for parametric/end-to-end if you want, just ping me !

gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Apr 1, 2026
…cheduler (#1824)

## Summary
Wire up the `DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL` config value to the telemetry scheduler. The env var was already parsed into `config.telemetry_extended_heartbeat_interval` but the scheduler hardcoded `Duration::from_secs(60 * 60 * 24)` instead of using it.

Default remains 24h this only enables system tests to use a shorter interval to validate the `app-extended-heartbeat` event fires correctly.

## Changes
- **`libdd-telemetry/src/worker/mod.rs`**: Replace hardcoded `60 * 60 * 24` with `config.telemetry_extended_heartbeat_interval`

## Motivation
Cross-SDK system tests need to set a short extended heartbeat interval (e.g., 2s) to validate parity of the `app-extended-heartbeat` telemetry event across all SDKs. Without this fix, PHP and other libdatadog consumers cannot be system-tested for this feature.

## Related
- System test PR: DataDog/system-tests#6338

Co-authored-by: edmund.kump <edmund.kump@datadoghq.com>
khanayan123 and others added 4 commits April 1, 2026 09:56
Add Test_ExtendedHeartbeat to tests/test_telemetry.py for the DEFAULT
scenario, validating that extended heartbeat config is a superset of
app-started plus any config changes. Set DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL
env var in weblog containers. Add manifest entries for all libraries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL from base container
env to the DEFAULT scenario weblog_env where it belongs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The extended heartbeat validation is better suited as an end-to-end
test in the DEFAULT scenario where it exercises the full tracer stack.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@khanayan123 khanayan123 requested review from a team as code owners April 22, 2026 15:59
doc="Test env var `DD_TELEMETRY_METRICS_ENABLED=false`",
scenario_groups=[scenario_groups.telemetry],
)
telemetry_extended_heartbeat = EndToEndScenario(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to add this scenario in .github/workflows/run-end-to-end.yml

Copy link
Copy Markdown
Contributor Author

@khanayan123 khanayan123 Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 4f7f76e — the TELEMETRY_EXTENDED_HEARTBEAT step is now registered between TELEMETRY_ENHANCED_CONFIG_REPORTING and TELEMETRY_LOG_GENERATION_DISABLED.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 49f6dc1a56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

scenario_groups=[scenario_groups.telemetry],
)
telemetry_extended_heartbeat = EndToEndScenario(
"TELEMETRY_EXTENDED_HEARTBEAT",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wire TELEMETRY_EXTENDED_HEARTBEAT into end-to-end CI

This adds a new TELEMETRY_EXTENDED_HEARTBEAT scenario and binds Test_ExtendedHeartbeat to it, but the end-to-end workflow is not updated to execute that scenario: in .github/workflows/run-end-to-end.yml the telemetry section only runs TELEMETRY_APP_STARTED_PRODUCTS_DISABLED, TELEMETRY_ENHANCED_CONFIG_REPORTING, TELEMETRY_LOG_GENERATION_DISABLED, TELEMETRY_METRIC_GENERATION_DISABLED, and TELEMETRY_DEPENDENCY_LOADED_TEST_FOR_DEPENDENCY_COLLECTION_DISABLED. That means the new test path is effectively untested in CI, so regressions in this feature can merge without detection.

Useful? React with 👍 / 👎.

Addresses review feedback — the new scenario needs a step in
.github/workflows/run-end-to-end.yml so CI can actually run it.
Comment thread manifests/cpp_nginx.yml Outdated
tests/test_standard_tags.py: irrelevant
tests/test_telemetry.py::Test_APMOnboardingInstallID: '>=1.12.0' # Modified by easy win activation script
tests/test_telemetry.py::Test_DependencyEnable: '>=1.12.0' # Modified by easy win activation script
tests/test_telemetry.py::Test_ExtendedHeartbeat: ">1.14.0"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't expect this to work, as nginx-datadog has not received an update of its dd-trace-cpp since the app-extended-heartbeat telemetry event was implemented in DataDog/dd-trace-cpp@1603dce

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I merged in this PR after extended heartbeat got merged in:

DataDog/nginx-datadog#345

would this include it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same with httpd: DataDog/httpd-datadog#54

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — addressed in c78fad7. Marked Test_ExtendedHeartbeat as missing_feature for both cpp_nginx and cpp_httpd until those modules pick up a dd-trace-cpp release that includes the implementation. CI was indeed failing with app-extended-heartbeat event not found on cpp_nginx as you predicted.

Address CI failures across cpp_nginx, cpp_httpd, dotnet, golang, java
(spring-boot-3-native), and php for the new TELEMETRY_EXTENDED_HEARTBEAT
scenario:

- cpp_nginx / cpp_httpd: nginx-datadog and httpd-datadog have not yet
  shipped a release that includes the dd-trace-cpp commit implementing
  app-extended-heartbeat (per zacharycmontoya feedback)
- dotnet: extended-heartbeat payload contains configs missing the
  required `value` field, failing schema validation
- golang: extended-heartbeat only includes app-started configs, not
  app-client-configuration-change configs
- java spring-boot-3-native: native image doesn't capture app-started
  telemetry event
- php: telemetry runs via a sidecar process that does not emit
  app-extended-heartbeat events yet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants