fix(debugger): enable code origin in-product enablement tests by watson · Pull Request #6063 · DataDog/system-tests

watson · 2026-01-19T20:47:20Z

Motivation

The code origin (CO) in-product enablement tests were never actually working - they were skipped by Java, Python, and .NET, and not enabled by Node.js. Investigation revealed a fundamental issue with how the tests were interacting with Remote Config (RC) that made them unreliable.

RC's client_configs field represents the complete list of active configs, not incremental updates. When the tests sent separate APM_TRACING and LIVE_DEBUGGING payloads, previously applied configs would be unapplied if they weren't in the new client_configs list, causing the tests to fail unpredictably.

Changes

Core Fix:

Added send_rc_apm_tracing_and_probes() utility in utils/_remote_config.py to send both APM_TRACING and LIVE_DEBUGGING configs in a single RC command.

Test Improvements (tests/debugger/test_debugger_inproduct_enablement.py):

Fixed CO test to check all traces after a threshold, not just the first trace file
Added language-specific initial state handling (Node.js enables CO by default, others don't)

Manifest Updates:

Node.js: Enabled CO test starting from v5.83.0 (yet to be released to npm)
Java: Simplified CO manifest entry
Python: Simplified CO manifest entry
.NET: Simplified CO and DI manifest entry

Supporting Changes:

Updated scenario config to add faster RC polling and trace flushing intervals
Enhanced wait_for_code_origin_span() to use threshold-based filtering to avoid stale traces

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

watson · 2026-01-19T20:47:46Z

fix(debugger): enable code origin in-product enablement tests #6063 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-01-19T20:47:48Z

CODEOWNERS have been resolved as:

manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/debugger/test_debugger_inproduct_enablement.py                    @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils.py                                                 @DataDog/debugger @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_remote_config.py                                                 @DataDog/system-tests-core

datadog-official · 2026-01-21T10:48:45Z

⚠️ Tests

✨ Fix all issues with Cursor

⚠️ Warnings

❄️ 7 New flaky tests detected

tests.appsec.iast.sink.test_hardcoded_passwords.Test_HardcodedPasswords_ExtendedLocation.test_extended_location_data[express4]

from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: No hardcoded vulnerabilities found
assert []

self = <tests.appsec.iast.sink.test_hardcoded_passwords.Test_HardcodedPasswords_ExtendedLocation object at 0x7ff8888240b0>

    def test_extended_location_data(self):
>       hardcoded_passwords = get_hardcoded_vulnerabilities("HARDCODED_PASSWORD")

tests/appsec/iast/sink/test_hardcoded_passwords.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...

tests.appsec.iast.sink.test_hardcoded_passwords.Test_HardcodedPasswords.test_hardcoded_passwords_exec[express4] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: No hardcoded vulnerabilities found
assert []

self = <tests.appsec.iast.sink.test_hardcoded_passwords.Test_HardcodedPasswords object at 0x7ff8887ebd10>

    def test_hardcoded_passwords_exec(self):
        assert self.r_hardcoded_passwords_exec.status_code == 200
>       hardcoded_passwords = get_hardcoded_vulnerabilities("HARDCODED_PASSWORD")

tests/appsec/iast/sink/test_hardcoded_passwords.py:30: 
...

tests.appsec.iast.sink.test_hardcoded_secrets.Test_HardcodedSecrets_ExtendedLocation.test_extended_location_data[express4]

from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: No hardcoded vulnerabilities found
assert []

self = <tests.appsec.iast.sink.test_hardcoded_secrets.Test_HardcodedSecrets_ExtendedLocation object at 0x7ff8887ebd70>

    def test_extended_location_data(self):
>       hardcode_secrets = get_hardcoded_vulnerabilities("HARDCODED_SECRET")

tests/appsec/iast/sink/test_hardcoded_secrets.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...

View all

ℹ️ Info

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 17fde06 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

watson · 2026-01-21T17:23:52Z

The currently failing tests are unrelated to this PR and actively being fixed in other PRs. The relevant tests to this PR are currently all passing.

cbeauchesne · 2026-01-21T20:16:24Z

Rebasing to fix the PHP issue.

cbeauchesne · 2026-01-23T16:50:58Z

you'll need to rebase again to fix python issue (sorry)

nccatoni

LGTM but you should get a review from someone familiar with the feature

The code origin in-product enablement tests were previously skipped by all languages due to underlying issues with Remote Config interaction. This commit fixes those issues and enables the tests. 1. **Root Cause**: Remote Config's client_configs represents the COMPLETE list of active configs, not incremental updates. Sending separate APM_TRACING and LIVE_DEBUGGING payloads caused previously applied configs to be unapplied when they weren't in the new client_configs list, making the tests unreliable. 2. **Solution**: Introduced send_rc_apm_tracing_and_probes() utility to send both APM_TRACING and LIVE_DEBUGGING configs in a single command, preventing unintended unapplication of configs. 3. **Test Improvements**: - Added timing controls (RC_POLL_INTERVAL, TRACE_FLUSH_WAIT) for reliable config polling and trace flushing - Fixed code origin test to check all traces after threshold, not just one - Added language-specific initial state handling (Node.js enables code origin by default) - Reduced environment setup to minimal required config - Removed decorator-based skip/bug markings in favor of manifest declarations 4. **Manifest Updates**: - Node.js: Enabled code origin test starting from v5.83.0 (first time!) - Java: Code origin remains skipped (missing_feature, simplified manifest) - Python: Code origin remains skipped (missing_feature, simplified manifest) - .NET: Moved bug marking (DEBUG-4637) from test decorators to manifest These changes enable code origin in-product enablement tests for Node.js and establish the correct testing infrastructure for other languages to follow.

tylfin · 2026-01-26T13:35:53Z

-        spring-boot-undertow: v1.48.0
-        spring-boot-wildfly: v1.48.0
-        uds-spring-boot: v1.48.0
+  tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin: missing_feature


When you define it as missing_feature, does it retain the force_skip=True? These tests take quite a bit of time so we'd like to make sure they don't run unless they're passing

I think that is outside the scope of this PR due to new rules. For details of this change, see this discussion: #6063 (comment), and this PR: #6059

cc @nccatoni @cbeauchesne Do the manifests support force_skip yet? I thought we were still supposed to rely on decorators until they did?

@nccatoni is implemeting it. It won't be inside manifest because it's not related to version declaration, but rather a direct property of method classes. But it'll achieve the same

Is it a blocker for this PR though, or can we clean it up later once the new functionality is added? A lot of force-skip was already removed in the PR I mentioned above

I reverted the removal of force_skip in the latest commit

can we clean it up later once the new functionality is added

yes sorry, you can move forward with it, we'll clean everything after.

…-origin-tests

Done

…-origin-tests

watson force-pushed the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch 2 times, most recently from 6304699 to 376a924 Compare January 19, 2026 20:57

watson mentioned this pull request Jan 19, 2026

feat(debugger): add Remote Enablement support DataDog/dd-trace-js#7137

Merged

watson force-pushed the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch 7 times, most recently from 09802dd to 29749db Compare January 21, 2026 10:40

watson force-pushed the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch 2 times, most recently from b0b274e to 14e5970 Compare January 21, 2026 14:25

watson marked this pull request as ready for review January 21, 2026 17:22

watson requested review from a team as code owners January 21, 2026 17:22

watson requested review from ZStriker19, daniel-romano-DD, jandro996 and rachelyangdog and removed request for a team January 21, 2026 17:22

watson requested a review from mhlidd January 21, 2026 17:22

nccatoni previously requested changes Jan 22, 2026

View reviewed changes

Comment thread tests/debugger/test_debugger_inproduct_enablement.py Outdated

watson force-pushed the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch from 2f3dc01 to 499435c Compare January 23, 2026 15:47

nccatoni reviewed Jan 26, 2026

View reviewed changes

nccatoni self-requested a review January 26, 2026 10:30

watson added 3 commits January 26, 2026 13:15

Cleanup

057749b

Address review comments

489fd6c

watson force-pushed the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch from 173745e to 489fd6c Compare January 26, 2026 12:20

tylfin reviewed Jan 26, 2026

View reviewed changes

watson added 2 commits January 26, 2026 17:21

Merge branch 'main' into watson/DEBUG-4402/fix-remote-enablement-code…

13c74bd

…-origin-tests

Merge branch 'main' into watson/DEBUG-4402/fix-remote-enablement-code…

c7bec17

…-origin-tests

watson added 2 commits January 27, 2026 13:41

Merge branch 'main' into watson/DEBUG-4402/fix-remote-enablement-code…

4a34329

…-origin-tests

Address review comments

17fde06

watson force-pushed the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch from a01f5b8 to 17fde06 Compare January 27, 2026 13:01

tylfin approved these changes Jan 27, 2026

View reviewed changes

watson enabled auto-merge (squash) January 27, 2026 14:11

watson merged commit a208834 into main Jan 29, 2026
2553 of 2557 checks passed

watson deleted the watson/DEBUG-4402/fix-remote-enablement-code-origin-tests branch January 29, 2026 16:06

Conversation

watson commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

watson commented Jan 19, 2026

Uh oh!

github-actions Bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Jan 21, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

watson commented Jan 21, 2026

Uh oh!

cbeauchesne commented Jan 21, 2026

Uh oh!

Uh oh!

cbeauchesne commented Jan 23, 2026

Uh oh!

nccatoni left a comment

Choose a reason for hiding this comment

Uh oh!

tylfin Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

watson Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

tylfin Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

cbeauchesne Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

watson Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

watson Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

cbeauchesne Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

watson commented Jan 19, 2026 •

edited

Loading

github-actions Bot commented Jan 19, 2026 •

edited

Loading

datadog-official Bot commented Jan 21, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading