feat(config): support libdatadog's library_config for config through file by BaptisteFoy · Pull Request #11839 · DataDog/dd-trace-py

BaptisteFoy · 2024-12-27T14:29:47Z

Description of changes

Bumps libdatadog to v15.0.0 & handles the required changes to crashtracking
Starts using the library_config component of libdatadog that allows reading configuration from a file with a higher precedence than environment variables

Supported config entries

As of libdatadog v15.0.0 (source):

DD_TRACE_DEBUG
DD_SERVICE
DD_ENV
DD_VERSION
DD_PROFILING_ENABLED

Testing strategy

Added a unit test.

To test this PR, you can:

Create a Linux VM
Start this tracer version & ensure everything works well (non-regression)
Create a file at the following path: /etc/datadog-agent/managed/datadog-apm-libraries/stable/libraries_config.yaml
Write the following content in the file:

rules:
  - selectors:
      - origin: language
        matches:
          - python
        operator: equals
    configuration:
      DD_TRACE_DEBUG: true

Start this tracer version
Verify that debug logs are properly enabled

Additional concerns

This change does increase the size of the serverless layer by 380KB (14204KB vs 13824 as the limit) even after optimization. I had to go from lto=true to lto=fast, as lto=true somehow prevents release builds on older machines. That accounts for 200KB, while the 200 other KB are mostly coming from the libdatadog bump.

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2024-12-27T14:30:21Z

CODEOWNERS have been resolved as:

src/native/library_config.rs                                            @DataDog/apm-core-python
tests/internal/test_native.py                                           @DataDog/apm-core-python
ddtrace/__init__.py                                                     @DataDog/apm-core-python
ddtrace/internal/native/__init__.py                                     @DataDog/apm-core-python
ddtrace/internal/native/_native.pyi                                     @DataDog/apm-core-python
src/native/Cargo.lock                                                   @DataDog/apm-core-python
src/native/Cargo.toml                                                   @DataDog/apm-core-python
src/native/lib.rs                                                       @DataDog/apm-core-python

…gh file

pr-commenter · 2025-01-17T17:21:28Z

Benchmarks

Benchmark execution time: 2025-02-04 13:17:24

Comparing candidate commit ee3c320 in PR branch baptiste.foy/FA/library-config-libdatadog with baseline commit af9098c in branch main.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 392 metrics, 2 unstable metrics.

scenario:iast_aspects-ospathdirname_aspect

🟩 execution_time [-434.186ns; -373.254ns] or [-10.755%; -9.246%]

scenario:iast_aspects-ospathsplitext_aspect

🟩 execution_time [-372.226ns; -323.480ns] or [-9.303%; -8.085%]

taegyunkim

👋 I'd like to suggest not adding new modules under /profiling as the functionality is not profiling. You should be able to expose the libdatadog API through src/core, or add a new rust/python module under src/

datadog-dd-trace-py-rkomorn · 2025-01-21T14:45:09Z

Datadog Report

Branch report: baptiste.foy/FA/library-config-libdatadog
Commit report: 1575326
Test service: dd-trace-py

✅ 0 Failed, 208 Passed, 1390 Skipped, 5m 3.12s Total duration (35m 0.17s time saved)

## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

brettlangdon · 2025-01-28T16:22:01Z

+    5. Default values
+    """
+    for key, value in get_configuration_from_disk().items():
+        os.environ[key] = str(value).lower()


not sure I love this 🤔

I don't have a good suggestion, but modifying system state in this way just feels weird.

would we expect that a user should be able to access/see the configuration applied by accessing it at os.environ[key] ?

It does really feel weird to me too; and to be fair it's a hack to make it work quickly. There is no global configuration object that is used everywhere that we can use.

Hopefully in the not-so-far future we'll be able to properly apply this configuration and track its origin; but this is a whole other problem to tackle.

brettlangdon · 2025-01-28T16:23:29Z

+    return False  # If one or both modules are not imported, return False
+
+
+def test_native_before_settings():


we should probably have a test that applying a configuration via file gets applied to the application.

do we expect to have system-tests for this stuff as well? (if not, can we?)

I guess we should; would you have any pointers on how I can do so?

Besides, yes we want to add system-tests to test this behaviour on customer-like systems.

Besides, yes we want to add system-tests to test this behaviour on customer-like systems

Can we add those before we merge this? It should be fully tested before we merge, since as soon as it is merged it can be released.

It should be fully tested before we merge, since as soon as it is merged it can be released.

Would you be ok with merging this under an experimental flag that disables calling to this code by default, so we are sure no customer uses it even if we release? It's hard to bootstrap system-tests if we don't have any merged language implementation to run the actual tests against.

We need to make sure that this code doesn't crash stuff obviously but this is something that can be tested in integration tests in ddtrace-py rather than in system-tests.

You should be able to write tests against a dev version of this library pretty easily: https://github.com/DataDog/system-tests/blob/main/docs/execute/binaries.md#python-library

Merging tests in system-tests can be tricky since we need to ensure they are expected to fail on older versions.

I'd prefer the approach of writing at least the first pass of tests against this branch, but otherwise, a private/hidden feature flag would be ok.

I've written a first system test and made it pass locally against this PR: DataDog/system-tests#3982. Happy to get your review @brettlangdon. Let me know if there is anything to change on this PR to run this test.

Depending of the timing, libddwaf loading process could create triggers that would create loops in our instrumentation. From what I investigated: - if loaded too early, it could have bad interactions with gevent. - if loaded too late, it could be self instrumented by the tracer, creating a loop, as ctypes is using Popen and subprocess. while keeping the late loading introduced by 2 previous PRs - #11987 - #12013 this PR introduced a mechanism to bypass tracer instrumentation during ctypes loading, to avoid a possible loop that would prevent the WAF to be loaded. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…t config (#12089) This PR makes a change to our shared distributed tracing header injection method to dispatch signals/events instead of relying on the global config settings, which is only modifiable via env vars. This fixes distributed tracing for users that might rely solely on the `LLMObs.enable()` setup config. Programmatic `LLMObs.enable()/disable()` calls do not set the global `config._llmobs_enabled` boolean setting, which is only controlled by the `DD_LLMOBS_ENABLED` env var. This was problematic for users that relied on manual `LLMObs.enable()` setup (i.e. no env vars) because our distributed tracing injection code only checks the global config to inject llmobs parent IDs into request headers. If users manually enabled LLMObs without any env vars, then this would not be reflected in the global config value and thus LLMObs parent IDs would never be injected into the request headers. We can't check directly if LLMObs is enabled in the http injection module because: 1. This would require us to import significant product-specific LLMObs-code into the shared http injector helper module which would impact non-LLMObs users' app performance 2. Circular imports in LLMObs which imports http injector logic to use in its own helpers Instead of doing our check based on the global `config._llmobs_enabled` setting, we now send a tracing event to our shared product listeners, and register a corresponding `LLMObs._inject_llmobs_context()` hook to be called for all inject() calls if LLMObs is enabled (we check the LLMObs instance, not the global config setting value). ~One risk and why I don't like changing global config settings is because this then implies that it is no longer global or tied to an env var (I want to push for env var configuration where possible over manual overriding/enabling). If a global enabled config can be toggled indiscriminately then this could open a can of worms for enabling/disabling logic in our LLMObs service, which isn't really designed to be toggled on/off multiple times in the app's lifespan. However if some users cannot rely on env vars, then I don't see any other solution that does not couple tracer internal code with LLMObs code which is a no-option.~ (UPDATE: we avoided this issue by using signal dispatching) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

ddtrace v3.0 is set to remove `ddtrace.providers` module from the public API. However we should still allow users to use their own ContextProviders. This PR ensures the BaseContextProvider remains in the public API and can be used in `tracer.configure(...)`. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

The shared pipeline is introducing support for windows OCI packages. This PR ensure dd-trace-py doesn't generate nonsensical packages. Windows support can be added later. This was tested by using the WIP branch of one-pipeline ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>

This PR depends on #12063. Rename src/core to src/native. This is laying ground as a [following PR](#11839) will need to import native code in `ddtrace.__init__.py`. Without this change this would create a dependency loop between `ddtrace` & `ddtrace.core` through `event_hub.py`. This PR only renames files, so a non-regression test is the only thing needed. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

BaptisteFoy · 2025-02-10T08:25:20Z

Closing in favour of #12260

…file (#12260) ## Description of changes Most changes were taken from #11839 - Bumps `libdatadog` to v16.0.2 - Starts using the library_config component of `libdatadog` that allows reading configuration from a file **with a higher precedence than environment variables** ### Supported config entries As of libdatadog v16.0.2: - DD_APM_TRACING_ENABLED - DD_RUNTIME_METRICS_ENABLED - DD_LOGS_INJECTION - DD_PROFILING_ENABLED - DD_DATA_STREAMS_ENABLED - DD_APPSEC_ENABLED - DD_IAST_ENABLED - DD_DYNAMIC_INSTRUMENTATION_ENABLED - DD_DATA_JOBS_ENABLED - DD_APPSEC_SCA_ENABLED - DD_TRACE_DEBUG - DD_SERVICE - DD_ENV - DD_VERSION ## Testing strategy Added a unit test. To test this PR, you can: 1. Create a Linux VM 2. Start this tracer version & ensure everything works well (non-regression) 3. Create a file at the following path: `/etc/datadog-agent/application_monitoring.yaml` 4. Write the following content in the file: ```yaml rules: - selectors: - origin: language matches: - python operator: equals configuration: DD_TRACE_DEBUG: true ``` 5. Start this tracer version 6. Verify that debug logs are properly enabled ## Additional concerns Relevant system test: DataDog/system-tests#3982 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

datadog-datadog-prod-us1 Bot reviewed Dec 27, 2024

View reviewed changes

Comment thread ddtrace/__init__.py Outdated

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch 3 times, most recently from 606b2a5 to cef695a Compare December 27, 2024 14:39

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from cef695a to 3f85c2a Compare January 15, 2025 13:43

upgrade(tracer): Support libdatadog's library_config for config throu…

908ace8

…gh file

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from 3f85c2a to 908ace8 Compare January 15, 2025 13:44

released version of libdatadog

74839d1

taegyunkim reviewed Jan 17, 2025

View reviewed changes

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch 2 times, most recently from 4bd0ef9 to 217fa81 Compare January 21, 2025 13:41

upgrade(python): Use src/config instead of profiling

39af866

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from 217fa81 to 39af866 Compare January 21, 2025 13:58

BaptisteFoy changed the title ~~upgrade(tracer): Support libdatadog's library_config for config through file~~ feat(config): Support libdatadog's library_config for config through file Jan 21, 2025

BaptisteFoy changed the title ~~feat(config): Support libdatadog's library_config for config through file~~ feat(config): support libdatadog's library_config for config through file Jan 21, 2025

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from 55ce7fe to df2f272 Compare January 21, 2025 14:20

fix ci

a1877e2

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from df2f272 to a1877e2 Compare January 21, 2025 14:28

BaptisteFoy added 2 commits January 21, 2025 16:59

Merge branch 'main' into baptiste.foy/FA/library-config-libdatadog

dc5c18b

fix ci

32de0db

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from bb2848b to 32de0db Compare January 21, 2025 16:38

release notes

2f917ed

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch 5 times, most recently from fddf037 to bac1549 Compare January 22, 2025 15:23

BaptisteFoy and others added 2 commits January 28, 2025 15:40

Merge branch 'main' into baptiste.foy/FA/library-config-libdatadog

fa07225

brettlangdon reviewed Jan 28, 2025

View reviewed changes

christophe-papazian and others added 4 commits January 28, 2025 16:28

BaptisteFoy added 2 commits February 3, 2025 16:07

re-enable lto

dde607d

Merge branch 'main' into baptiste.foy/FA/library-config-libdatadog

ee3c320

BaptisteFoy force-pushed the baptiste.foy/FA/library-config-libdatadog branch from e3f4a94 to ee3c320 Compare February 4, 2025 12:35

BaptisteFoy mentioned this pull request Feb 4, 2025

feat(stable-configuration): Add first system test DataDog/system-tests#3982

Merged

7 tasks

erikayasuda force-pushed the main branch from af9098c to 9ba79f4 Compare February 6, 2025 19:00

erikayasuda requested review from a team as code owners February 6, 2025 19:00

erikayasuda requested review from nsrip-dd, rachelyangdog and wantsui February 6, 2025 19:00

erikayasuda force-pushed the main branch from 1247ac2 to 2ccaaef Compare February 6, 2025 20:43

BaptisteFoy closed this Feb 10, 2025

BaptisteFoy mentioned this pull request Feb 10, 2025

feat(config): support libdatadog's library_config for config through file #12260

Merged

2 tasks

		return False # If one or both modules are not imported, return False


		def test_native_before_settings():

Conversation

BaptisteFoy commented Dec 27, 2024 • edited by mabdinur Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Supported config entries

Testing strategy

Additional concerns

Checklist

Reviewer Checklist

Uh oh!

github-actions Bot commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pr-commenter Bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:iast_aspects-ospathdirname_aspect

scenario:iast_aspects-ospathsplitext_aspect

Uh oh!

taegyunkim left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-dd-trace-py-rkomorn Bot commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

Uh oh!

brettlangdon Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

BaptisteFoy Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

brettlangdon Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

BaptisteFoy Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

brettlangdon Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

paullegranddc Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brettlangdon Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

BaptisteFoy Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

BaptisteFoy commented Feb 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

BaptisteFoy commented Dec 27, 2024 •

edited by mabdinur

Loading

github-actions Bot commented Dec 27, 2024 •

edited

Loading

pr-commenter Bot commented Jan 17, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn Bot commented Jan 21, 2025 •

edited

Loading

paullegranddc Feb 3, 2025 •

edited

Loading