Skip to content

feat(profiler): add custom profiling labels API#7879

Merged
szegedi merged 2 commits intomasterfrom
attila.szegedi/custom-profiling-labels
Apr 8, 2026
Merged

feat(profiler): add custom profiling labels API#7879
szegedi merged 2 commits intomasterfrom
attila.szegedi/custom-profiling-labels

Conversation

@szegedi
Copy link
Copy Markdown
Contributor

@szegedi szegedi commented Mar 28, 2026

Summary

  • Adds tracer.profiling.runWithLabels(labels, fn) API that attaches custom key-value labels to all wall profiler samples taken during fn's execution (including async continuations)
  • Adds tracer.profiling.setCustomLabelKeys(keys) to declare label keys for UI indexing and pprof serialization optimization
  • Includes custom_attributes in profile upload event metadata so the Datadog backend knows which keys to index
  • Brings Node.js profiler custom labels to parity with Java and Go (docs)

Usage

const tracer = require('dd-trace').init()

tracer.profiling.setCustomLabelKeys(['customer', 'region'])

tracer.profiling.runWithLabels({ customer: 'acme', region: 'us-east' }, () => {
  handleRequest()
})

Implementation details

Custom labels are stored alongside the span profiling context in pprof's CPED AsyncLocalStorage as a 2-element array [profilingContext, customLabels]. When no custom labels are active, the context remains a plain object; just profilingContext from the array – unchanged behavior, zero cost. A monotonic flag (#customLabelsActive) avoids the getContext() native call overhead when custom labels have never been used, and when they're used then the code uses Array.isArray() to distinguish between the two forms of value in the ALS.

Internal labels (span id, thread name, etc.) are set later than custom labels when constructing samples, so they always take precedence over custom labels with the same key.

setCustomLabelKeys would not even be strictly necessary, but this way we can avoid dynamically constructing a set of labels, and in any case the Java and Go APIs both have this method for explicit declaration of the set of custom label keys.

The feature requires AsyncContextFrame to be enabled; it gracefully falls back to a no-op when ACF is off or profiling is disabled.

Test plan

  • Unit tests for _generateLabels with array contexts (ACF), empty profiling context, non-ACF unchanged
  • Unit tests for runWithLabels: basic operation, nesting/merging, key override, ACF-off passthrough, contexts-off passthrough, internal label precedence
  • Unit tests for #enter optimizations: custom labels preservation, skip-when-unchanged, async continuation correctness
  • Agent exporter tests pass (event JSON unchanged when no custom labels declared)
  • Integration test with real profile export verifying labels in pprof output

🤖 Generated with Claude Code

Add tracer.profiling.runWithLabels() and tracer.profiling.setCustomLabelKeys()
APIs that allow users to attach custom labels to wall profiler samples, enabling
flame graph filtering by business logic dimensions (e.g. customer, region) in
the Datadog UI. This brings Node.js profiler to parity with Java and Go.

The implementation stores custom labels alongside span profiling context in
pprof's CPED AsyncLocalStorage as a 2-element array [profilingContext,
customLabels]. A monotonic flag avoids getContext() overhead when custom labels
have never been used. Labels propagate correctly through async continuations.
Internal labels (span id, thread name, etc.) always take precedence over custom
labels with the same key.

Requires AsyncContextFrame (ACF) to be enabled; gracefully falls back to a
passthrough (just calls fn()) when ACF is off or profiling is disabled.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@szegedi szegedi requested review from a team as code owners March 28, 2026 15:28
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 28, 2026

Overall package size

Self size: 5.48 MB
Deduped: 6.33 MB
No deduping: 6.33 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 28, 2026

Codecov Report

❌ Patch coverage is 52.72727% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.22%. Comparing base (1733e97) to head (36082b5).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
packages/dd-trace/src/profiling/profiler.js 20.00% 12 Missing ⚠️
packages/dd-trace/src/proxy.js 0.00% 5 Missing ⚠️
packages/dd-trace/src/profiling/profilers/wall.js 85.71% 4 Missing ⚠️
packages/dd-trace/src/noop/proxy.js 33.33% 2 Missing ⚠️
packages/dd-trace/src/profiler.js 0.00% 2 Missing ⚠️
...-trace/src/profiling/exporters/event_serializer.js 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7879      +/-   ##
==========================================
- Coverage   74.62%   74.22%   -0.40%     
==========================================
  Files         768      769       +1     
  Lines       35673    36120     +447     
==========================================
+ Hits        26620    26810     +190     
- Misses       9053     9310     +257     
Flag Coverage Δ
aiguard-macos 39.42% <12.50%> (+0.08%) ⬆️
aiguard-ubuntu 39.54% <12.50%> (+0.08%) ⬆️
aiguard-windows 39.20% <12.50%> (+0.08%) ⬆️
apm-capabilities-tracing-macos 49.47% <3.63%> (+0.37%) ⬆️
apm-capabilities-tracing-ubuntu 49.38% <3.63%> (+0.37%) ⬆️
apm-capabilities-tracing-windows 49.24% <3.63%> (+0.37%) ⬆️
apm-integrations-child-process 38.71% <12.50%> (+0.05%) ⬆️
apm-integrations-couchbase-18 37.50% <12.50%> (+0.04%) ⬆️
apm-integrations-couchbase-eol 37.99% <12.50%> (+0.04%) ⬆️
apm-integrations-oracledb 37.85% <12.50%> (+0.06%) ⬆️
appsec-express 55.35% <12.50%> (+0.09%) ⬆️
appsec-fastify 51.69% <12.50%> (+0.09%) ⬆️
appsec-graphql 51.84% <12.50%> (+0.09%) ⬆️
appsec-kafka 44.46% <12.50%> (+0.06%) ⬆️
appsec-ldapjs 44.09% <12.50%> (+0.07%) ⬆️
appsec-lodash 43.68% <12.50%> (+0.06%) ⬆️
appsec-macos 58.06% <12.50%> (-0.19%) ⬇️
appsec-mongodb-core 48.86% <12.50%> (+0.08%) ⬆️
appsec-mongoose 49.53% <12.50%> (+0.09%) ⬆️
appsec-mysql 51.05% <12.50%> (+0.19%) ⬆️
appsec-node-serialize 43.27% <12.50%> (+0.06%) ⬆️
appsec-passport 47.73% <12.50%> (+0.09%) ⬆️
appsec-postgres 50.68% <12.50%> (+0.09%) ⬆️
appsec-sourcing 42.52% <12.50%> (+0.06%) ⬆️
appsec-stripe 44.71% <12.50%> (+0.07%) ⬆️
appsec-template 43.43% <12.50%> (+0.06%) ⬆️
appsec-ubuntu 58.15% <12.50%> (-0.18%) ⬇️
appsec-windows 57.86% <12.50%> (-0.21%) ⬇️
instrumentations-instrumentation-bluebird 32.31% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-body-parser 40.61% <12.50%> (+0.05%) ⬆️
instrumentations-instrumentation-child_process 38.05% <12.50%> (+0.04%) ⬆️
instrumentations-instrumentation-cookie-parser 34.34% <12.50%> (+0.01%) ⬆️
instrumentations-instrumentation-express 34.66% <12.50%> (+0.02%) ⬆️
instrumentations-instrumentation-express-mongo-sanitize 34.47% <12.50%> (+0.01%) ⬆️
instrumentations-instrumentation-express-session 40.24% <12.50%> (+0.05%) ⬆️
instrumentations-instrumentation-fs 31.99% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-generic-pool 29.41% <ø> (-0.05%) ⬇️
instrumentations-instrumentation-http 39.96% <12.50%> (+0.05%) ⬆️
instrumentations-instrumentation-knex 32.38% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-mongoose 33.49% <12.50%> (+0.01%) ⬆️
instrumentations-instrumentation-multer 40.36% <12.50%> (+0.05%) ⬆️
instrumentations-instrumentation-mysql2 38.38% <12.50%> (+0.04%) ⬆️
instrumentations-instrumentation-passport 44.14% <12.50%> (+0.07%) ⬆️
instrumentations-instrumentation-passport-http 43.81% <12.50%> (+0.07%) ⬆️
instrumentations-instrumentation-passport-local 44.35% <12.50%> (+0.07%) ⬆️
instrumentations-instrumentation-pg 37.82% <12.50%> (+0.04%) ⬆️
instrumentations-instrumentation-promise 32.24% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-promise-js 32.24% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-q 32.29% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-url 32.21% <12.50%> (+<0.01%) ⬆️
instrumentations-instrumentation-when 32.26% <12.50%> (+<0.01%) ⬆️
llmobs-ai 41.35% <12.50%> (-1.12%) ⬇️
llmobs-anthropic 40.81% <12.50%> (+0.33%) ⬆️
llmobs-bedrock 39.30% <12.50%> (+0.04%) ⬆️
llmobs-google-genai 39.85% <12.50%> (+0.05%) ⬆️
llmobs-langchain 39.32% <12.50%> (-0.95%) ⬇️
llmobs-openai 44.09% <12.50%> (+0.11%) ⬆️
llmobs-vertex-ai 40.11% <12.50%> (+0.05%) ⬆️
platform-core 31.47% <ø> (ø)
platform-esbuild 34.42% <ø> (ø)
platform-instrumentations-misc 34.11% <ø> (-14.30%) ⬇️
platform-shimmer 37.56% <ø> (ø)
platform-unit-guardrails 32.89% <ø> (ø)
platform-webpack 19.85% <12.50%> (-0.19%) ⬇️
plugins-azure-durable-functions 25.74% <ø> (ø)
plugins-azure-event-hubs 25.90% <ø> (ø)
plugins-azure-service-bus 25.26% <ø> (ø)
plugins-bullmq 43.57% <12.50%> (-0.93%) ⬇️
plugins-cassandra 38.00% <12.50%> (+0.04%) ⬆️
plugins-cookie 26.96% <ø> (ø)
plugins-cookie-parser 26.75% <ø> (ø)
plugins-crypto 26.73% <ø> (ø)
plugins-dd-trace-api 38.40% <12.50%> (+0.04%) ⬆️
plugins-express-mongo-sanitize 26.89% <ø> (ø)
plugins-express-session 26.70% <ø> (ø)
plugins-fastify 42.33% <12.50%> (+0.06%) ⬆️
plugins-fetch 38.49% <12.50%> (+0.04%) ⬆️
plugins-fs 38.73% <12.50%> (+0.04%) ⬆️
plugins-generic-pool 25.94% <ø> (ø)
plugins-google-cloud-pubsub 45.66% <12.50%> (+0.08%) ⬆️
plugins-grpc 40.99% <12.50%> (+0.05%) ⬆️
plugins-handlebars 26.94% <ø> (ø)
plugins-hapi 40.25% <12.50%> (-0.08%) ⬇️
plugins-hono 40.58% <12.50%> (+0.06%) ⬆️
plugins-ioredis 38.57% <12.50%> (+0.07%) ⬆️
plugins-knex 26.57% <ø> (ø)
plugins-langgraph 37.97% <12.50%> (-0.59%) ⬇️
plugins-ldapjs 24.43% <ø> (ø)
plugins-light-my-request 26.30% <ø> (ø)
plugins-limitd-client 32.59% <12.50%> (+<0.01%) ⬆️
plugins-lodash 26.03% <ø> (ø)
plugins-mariadb 39.59% <12.50%> (+0.05%) ⬆️
plugins-memcached 38.31% <12.50%> (+0.07%) ⬆️
plugins-microgateway-core 39.31% <12.50%> (+0.05%) ⬆️
plugins-moleculer 40.61% <12.50%> (+0.05%) ⬆️
plugins-mongodb 39.25% <12.50%> (+0.05%) ⬆️
plugins-mongodb-core 39.09% <12.50%> (+0.05%) ⬆️
plugins-mongoose 38.90% <12.50%> (-0.02%) ⬇️
plugins-multer 26.70% <ø> (ø)
plugins-mysql 39.43% <12.50%> (+0.19%) ⬆️
plugins-mysql2 39.38% <12.50%> (+0.05%) ⬆️
plugins-node-serialize 27.00% <ø> (ø)
plugins-opensearch 37.72% <12.50%> (+0.04%) ⬆️
plugins-passport-http 26.76% <ø> (ø)
plugins-postgres 35.52% <12.50%> (-0.09%) ⬇️
plugins-process 26.73% <ø> (ø)
plugins-pug 26.96% <ø> (ø)
plugins-redis 39.01% <12.50%> (+0.05%) ⬆️
plugins-router 43.34% <12.50%> (+0.07%) ⬆️
plugins-sequelize 25.55% <ø> (ø)
plugins-test-and-upstream-amqp10 38.59% <12.50%> (+0.04%) ⬆️
plugins-test-and-upstream-amqplib 44.34% <12.50%> (+0.08%) ⬆️
plugins-test-and-upstream-apollo 39.21% <12.50%> (+0.04%) ⬆️
plugins-test-and-upstream-avsc 38.67% <12.50%> (+0.05%) ⬆️
plugins-test-and-upstream-bunyan 33.92% <12.50%> (+0.01%) ⬆️
plugins-test-and-upstream-connect 40.91% <12.50%> (+0.06%) ⬆️
plugins-test-and-upstream-graphql 40.25% <12.50%> (+0.06%) ⬆️
plugins-test-and-upstream-koa 40.49% <12.50%> (+0.05%) ⬆️
plugins-test-and-upstream-protobufjs 38.89% <12.50%> (+0.05%) ⬆️
plugins-test-and-upstream-rhea 44.37% <12.50%> (+0.08%) ⬆️
plugins-undici 39.34% <12.50%> (+0.17%) ⬆️
plugins-url 26.73% <ø> (ø)
plugins-valkey 38.28% <12.50%> (+0.06%) ⬆️
plugins-vm 26.73% <ø> (ø)
plugins-winston 34.24% <12.50%> (+0.15%) ⬆️
plugins-ws 42.09% <12.50%> (+0.07%) ⬆️
profiling-macos 40.76% <68.42%> (+0.19%) ⬆️
profiling-ubuntu 40.88% <68.42%> (+0.19%) ⬆️
profiling-windows 42.36% <54.71%> (+0.11%) ⬆️
serverless-azure-functions-client 25.62% <ø> (ø)
serverless-azure-functions-eventhubs 25.62% <ø> (ø)
serverless-azure-functions-servicebus 25.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-datadog-prod-us1
Copy link
Copy Markdown

datadog-datadog-prod-us1 Bot commented Mar 28, 2026

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 44.64%
Overall Coverage: 68.67% (-4.87%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 36082b5 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Mar 28, 2026

Benchmarks

Benchmark execution time: 2026-04-08 09:33:09

Comparing candidate commit 36082b5 in PR branch attila.szegedi/custom-profiling-labels with baseline commit 1733e97 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 229 metrics, 31 unstable metrics.

IlyasShabi
IlyasShabi previously approved these changes Apr 8, 2026
Copy link
Copy Markdown
Contributor

@IlyasShabi IlyasShabi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one NIT comment

Comment thread index.d.ts
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add test for the API part in docs/test.ts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@szegedi szegedi merged commit 635402e into master Apr 8, 2026
1035 of 1041 checks passed
@szegedi szegedi deleted the attila.szegedi/custom-profiling-labels branch April 8, 2026 16:05
dd-octo-sts Bot pushed a commit that referenced this pull request Apr 9, 2026
* feat(profiler): add custom profiling labels API

Add tracer.profiling.runWithLabels() and tracer.profiling.setCustomLabelKeys()
APIs that allow users to attach custom labels to wall profiler samples, enabling
flame graph filtering by business logic dimensions (e.g. customer, region) in
the Datadog UI. This brings Node.js profiler to parity with Java and Go.

The implementation stores custom labels alongside span profiling context in
pprof's CPED AsyncLocalStorage as a 2-element array [profilingContext,
customLabels]. A monotonic flag avoids getContext() overhead when custom labels
have never been used. Labels propagate correctly through async continuations.
Internal labels (span id, thread name, etc.) always take precedence over custom
labels with the same key.

Requires AsyncContextFrame (ACF) to be enabled; gracefully falls back to a
passthrough (just calls fn()) when ACF is off or profiling is disabled.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Apr 9, 2026
juan-fernandez pushed a commit that referenced this pull request Apr 10, 2026
* feat(profiler): add custom profiling labels API

Add tracer.profiling.runWithLabels() and tracer.profiling.setCustomLabelKeys()
APIs that allow users to attach custom labels to wall profiler samples, enabling
flame graph filtering by business logic dimensions (e.g. customer, region) in
the Datadog UI. This brings Node.js profiler to parity with Java and Go.

The implementation stores custom labels alongside span profiling context in
pprof's CPED AsyncLocalStorage as a 2-element array [profilingContext,
customLabels]. A monotonic flag avoids getContext() overhead when custom labels
have never been used. Labels propagate correctly through async continuations.
Internal labels (span id, thread name, etc.) always take precedence over custom
labels with the same key.

Requires AsyncContextFrame (ACF) to be enabled; gracefully falls back to a
passthrough (just calls fn()) when ACF is off or profiling is disabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants