Skip to content

feat(debugger): add Remote Enablement support#7137

Merged
watson merged 11 commits intomasterfrom
watson/DEBUG-4402/remote-enablement
Jan 19, 2026
Merged

feat(debugger): add Remote Enablement support#7137
watson merged 11 commits intomasterfrom
watson/DEBUG-4402/remote-enablement

Conversation

@watson
Copy link
Copy Markdown
Collaborator

@watson watson commented Dec 18, 2025

What does this PR do?

Implements Remote Enablement (RE) for Dynamic Instrumentation/Live Debugger and Code Origin via APM_TRACING_MULTICONFIG.

Key Remote Config changes:

  • Add RCClientLibConfigManager class for priority-based merging of multiple APM_TRACING configs (Service+Env > Service > Env > Cluster > Org)
  • Add APM_TRACING_MULTICONFIG, APM_TRACING_ENABLE_DYNAMIC_INSTRUMENTATION, and APM_TRACING_ENABLE_CODE_ORIGIN capability flags
  • Centralize client library configuration remote config logic in config/remote_config.js

Key Debugger changes:

  • Add lifecycle methods (isStarted(), stop(), configure()) to debugger/index.js for dynamic control

Key Code Origin changes:

  • Add runtime enable/disable support for Code Origin via Remote Config
  • Update Express and Fastify code origin plugins to pre-compute entry tags at route registration time when RC is enabled (allowing runtime enabling to work immediately)
  • Add integration tests for Code Origin Remote Config toggling

Motivation

The Debugger (Dynamic Instrumentation/Live Debugging) needs to support Remote Enablement, allowing operators to dynamically enable, disable, or reconfigure the debugger without requiring service restarts.

This implementation follows the APM_TRACING_MULTICONFIG RFC.

@watson watson self-assigned this Dec 18, 2025
@watson watson added semver-minor debugger Dynamic Instrumentation & Live Debugger labels Dec 18, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 18, 2025

Codecov Report

❌ Patch coverage is 75.00000% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.13%. Comparing base (cb2cd12) to head (8392030).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
packages/dd-trace/src/debugger/index.js 12.00% 22 Missing ⚠️
packages/dd-trace/src/proxy.js 53.33% 7 Missing ⚠️
packages/dd-trace/src/config/remote_config.js 96.00% 3 Missing ⚠️
packages/datadog-plugin-fastify/src/code_origin.js 85.71% 1 Missing ⚠️
packages/dd-trace/src/config.js 94.11% 1 Missing ⚠️
packages/dd-trace/src/llmobs/sdk.js 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7137      +/-   ##
==========================================
- Coverage   85.19%   85.13%   -0.06%     
==========================================
  Files         532      532              
  Lines       22788    22893     +105     
==========================================
+ Hits        19414    19491      +77     
- Misses       3374     3402      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 18, 2025

Overall package size

Self size: 4.41 MB
Deduped: 5.24 MB
No deduping: 5.24 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.0 | 68.46 kB | 797.03 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-datadog-prod-us1

This comment has been minimized.

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Dec 18, 2025

Benchmarks

Benchmark execution time: 2026-01-19 19:32:45

Comparing candidate commit 8392030 in PR branch watson/DEBUG-4402/remote-enablement with baseline commit cb2cd12 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 232 metrics, 28 unstable metrics.

@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 0c5d4ad to 21edda2 Compare December 18, 2025 08:36
@watson watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 84300df to 918975b Compare December 18, 2025 08:36
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 21edda2 to 5d6eabd Compare December 18, 2025 09:01
@watson watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 918975b to 137abe9 Compare December 18, 2025 10:43
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 5d6eabd to a412829 Compare December 18, 2025 10:43
@watson watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 137abe9 to 041ef5b Compare December 18, 2025 11:25
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from a412829 to bc7d261 Compare December 18, 2025 11:25
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from bc7d261 to 91ba2c1 Compare December 18, 2025 15:05
@watson watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 041ef5b to 573563d Compare December 19, 2025 17:38
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch 2 times, most recently from ce9aa71 to c2e520a Compare December 19, 2025 17:44
@watson watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 573563d to 152a63b Compare December 19, 2025 18:05
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from c2e520a to f369517 Compare December 19, 2025 18:05
@watson watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 152a63b to 408e3f5 Compare December 19, 2025 19:02
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch 4 times, most recently from 3d91f59 to f53bcac Compare December 19, 2025 20:18
@watson watson requested a review from shatzi January 8, 2026 14:01
@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 7f8c8d7 to 7195c95 Compare January 8, 2026 20:42
@watson watson marked this pull request as ready for review January 8, 2026 20:58
shatzi
shatzi previously approved these changes Jan 9, 2026
Comment thread packages/dd-trace/src/config/remote_config.js Outdated
@tylfin
Copy link
Copy Markdown
Member

tylfin commented Jan 9, 2026

This generally looks good to me, you might want to run it against this system-test https://github.com/DataDog/system-tests/blob/da17bdc4b54f09001bfd39cf50fd63d8f377cb8d/tests/debugger/test_debugger_inproduct_enablement.py#L55, and try a few manual failure scenarios flipping and deleting the config

@watson watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 9317837 to e5614fe Compare January 13, 2026 13:47
@watson watson requested a review from a team as a code owner January 13, 2026 13:47
@watson
Copy link
Copy Markdown
Collaborator Author

watson commented Jan 13, 2026

@tylfin thanks for reminding me about the system tests. They uncovered a bug where this PR didn't properly enable/disable Code Origin at runtime. The last commit added to this PR now should take care of that. The system tests are now passing locally, and I will update them once this PR lands

tylfin
tylfin previously approved these changes Jan 14, 2026
shatzi
shatzi previously approved these changes Jan 14, 2026
watson and others added 9 commits January 19, 2026 16:02
Implements Remote Enablement (RE) for Dynamic Instrumentation/Live
Debugger via APM_TRACING_MULTICONFIG.

This also fixes a long standing issue, now allowing multiple APM_TRACING
remote configurations to be received and merged with priority-based
rules.

Key Remote Config changes:
- Add ApmTracingManager class for priority-based merging of multiple
  APM_TRACING configs (Service+Env > Service > Env > Cluster > Org)
- Add APM_TRACING_MULTICONFIG,
  APM_TRACING_ENABLE_DYNAMIC_INSTRUMENTATION,
  APM_TRACING_ENABLE_CODE_ORIGIN, and APM_TRACING_ENABLE_LIVE_DEBUGGING
  capability flags
- Implement field guarding in config.js #applyRemote() to adopt highest
  priority non-null value for each field
Co-authored-by: Tyler Finethy <tylfin@gmail.com>
Copy link
Copy Markdown
Member

@BridgeAR BridgeAR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly LGTM, I just left a few minor things to check and some nits

Comment thread packages/dd-trace/test/proxy.spec.js Outdated
Comment thread packages/datadog-plugin-express/src/code_origin.js Outdated
Comment thread packages/datadog-plugin-fastify/src/code_origin.js Outdated
Comment thread packages/dd-trace/src/config.js
Comment thread packages/dd-trace/src/config/remote_config.js Outdated
Comment thread packages/dd-trace/src/proxy.js Outdated
Comment thread packages/dd-trace/src/proxy.js Outdated
@watson
Copy link
Copy Markdown
Collaborator Author

watson commented Jan 19, 2026

@tylfin here's the promised system test update to validate remote enablement works for the Node.js tracer: DataDog/system-tests#6063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debugger Dynamic Instrumentation & Live Debugger semver-minor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants