Skip to content

test(dsm): regression tests for DSM context leak between concurrent consumers#7580

Draft
robcarlan-datadog wants to merge 7 commits intomasterfrom
rcarlan/dsm-context-leak-regression-test
Draft

test(dsm): regression tests for DSM context leak between concurrent consumers#7580
robcarlan-datadog wants to merge 7 commits intomasterfrom
rcarlan/dsm-context-leak-regression-test

Conversation

@robcarlan-datadog
Copy link
Copy Markdown
Contributor

@robcarlan-datadog robcarlan-datadog commented Feb 19, 2026

Summary

  • Adds regression tests that reproduce a DSM context leak when consumers process messages and produce to output topics
  • Covers kafkajs, amqplib, bullmq, rhea (new dsm.spec.js), and google-cloud-pubsub
  • Each test forces two consume-produce flows and asserts that each produce's DSM parent context chains from its own consume, not the other's
  • All tests fail deterministically (verified 10/10 runs per plugin)

Root Cause

ctx.currentStore is set by startSpan() which snapshots the ALS store before decodeDataStreamsContext/setCheckpoint call enterWith() to set the DSM context. After store.run() restores the prior store (with DSM context) to the outer context, the next consumer's startSpan captures that leaked DSM context into its own ctx.currentStore, while the first consumer's callback has no DSM context at all.

Test plan

  • kafkajs: 10/10 deterministic failures (1.4.0, 2.2.4)
  • amqplib: 10/10 deterministic failures (0.5.3, 0.10.9)
  • bullmq: 10/10 deterministic failures (5.66.0, 5.66.5)
  • rhea: 10/10 deterministic failures (1.0.0, 3.0.4)
  • google-cloud-pubsub: 10/10 deterministic failures (1.2.0, 5.2.0)
  • CI should show the same failures

🤖 Generated with Claude Code

robcarlan-datadog and others added 2 commits February 19, 2026 12:26
…rrent consumers

When two KafkaJS consumers process messages concurrently and each
produces to a different topic, the DSM (Data Streams Monitoring) context
leaks between them. The first consumer to process loses its DSM context
entirely (null parent), while the second consumer picks up the first's
context instead of its own.

This test forces the interleaving by using promise gates to ensure both
eachMessage handlers have fired before either produces, reliably
reproducing the bug.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 19, 2026

Overall package size

Self size: 4.69 MB
Deduped: 5.53 MB
No deduping: 5.53 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.6 | 81.92 kB | 816.75 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Feb 19, 2026

Benchmarks

Benchmark execution time: 2026-02-19 19:48:16

Comparing candidate commit 7f52edd in PR branch rcarlan/dsm-context-leak-regression-test with baseline commit 495b56a in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 230 metrics, 30 unstable metrics.

@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.96%. Comparing base (495b56a) to head (7f52edd).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7580      +/-   ##
==========================================
- Coverage   80.29%   78.96%   -1.33%     
==========================================
  Files         732      727       -5     
  Lines       31468    31207     -261     
==========================================
- Hits        25267    24643     -624     
- Misses       6201     6564     +363     
Flag Coverage Δ
aiguard-macos 38.93% <ø> (-0.18%) ⬇️
aiguard-ubuntu 39.05% <ø> (-0.18%) ⬇️
aiguard-windows 38.79% <ø> (-0.18%) ⬇️
apm-capabilities-tracing-macos 48.63% <ø> (+0.24%) ⬆️
apm-capabilities-tracing-ubuntu 48.66% <ø> (+0.24%) ⬆️
apm-capabilities-tracing-windows 48.32% <ø> (+0.19%) ⬆️
apm-integrations-child-process 38.50% <ø> (-0.18%) ⬇️
apm-integrations-couchbase-18 37.27% <ø> (-0.31%) ⬇️
apm-integrations-couchbase-eol 37.90% <ø> (-0.02%) ⬇️
apm-integrations-oracledb 37.73% <ø> (-0.18%) ⬇️
appsec-express 55.53% <ø> (-0.21%) ⬇️
appsec-fastify 51.84% <ø> (-0.21%) ⬇️
appsec-graphql 52.02% <ø> (-0.19%) ⬇️
appsec-kafka 44.48% <ø> (-0.18%) ⬇️
appsec-ldapjs 44.09% <ø> (-0.18%) ⬇️
appsec-lodash 43.77% <ø> (-0.17%) ⬇️
appsec-macos ?
appsec-mongodb-core 48.84% <ø> (-0.21%) ⬇️
appsec-mongoose 49.64% <ø> (-0.21%) ⬇️
appsec-mysql 51.01% <ø> (-0.21%) ⬇️
appsec-node-serialize 43.29% <ø> (-0.17%) ⬇️
appsec-passport 47.78% <ø> (-0.21%) ⬇️
appsec-postgres 50.77% <ø> (-0.21%) ⬇️
appsec-sourcing 42.64% <ø> (-0.17%) ⬇️
appsec-template 43.46% <ø> (-0.17%) ⬇️
appsec-ubuntu 58.68% <ø> (-0.21%) ⬇️
appsec-windows 58.47% <ø> (-0.21%) ⬇️
instrumentations-instrumentation-bluebird 32.20% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-body-parser 40.51% <ø> (-0.18%) ⬇️
instrumentations-instrumentation-child_process 37.82% <ø> (-0.17%) ⬇️
instrumentations-instrumentation-cookie-parser 34.24% <ø> (-0.18%) ⬇️
instrumentations-instrumentation-express 34.58% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-express-mongo-sanitize 34.37% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-express-session 40.13% <ø> (-0.18%) ⬇️
instrumentations-instrumentation-fs 31.80% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-generic-pool 29.76% <ø> (ø)
instrumentations-instrumentation-http 39.85% <ø> (-0.18%) ⬇️
instrumentations-instrumentation-knex 32.20% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-mongoose 33.37% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-multer 40.25% <ø> (-0.18%) ⬇️
instrumentations-instrumentation-mysql2 38.29% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-passport 44.09% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-passport-http 43.76% <ø> (-0.18%) ⬇️
instrumentations-instrumentation-passport-local 44.30% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-pg 37.71% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-promise 32.13% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-promise-js 32.13% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-q 32.18% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-url 32.10% <ø> (-0.19%) ⬇️
instrumentations-instrumentation-when 32.15% <ø> (-0.19%) ⬇️
llmobs-ai 41.33% <ø> (-0.18%) ⬇️
llmobs-anthropic 40.32% <ø> (-0.18%) ⬇️
llmobs-bedrock 39.25% <ø> (-0.15%) ⬇️
llmobs-google-genai 39.84% <ø> (-0.16%) ⬇️
llmobs-langchain 39.43% <ø> (-0.14%) ⬇️
llmobs-openai 44.14% <ø> (-0.19%) ⬇️
llmobs-vertex-ai 40.04% <ø> (-0.17%) ⬇️
platform-core 29.71% <ø> (ø)
platform-esbuild 32.89% <ø> (ø)
platform-instrumentations-misc 40.53% <ø> (ø)
platform-shimmer 36.14% <ø> (ø)
platform-unit-guardrails 31.27% <ø> (ø)
plugins-azure-event-hubs 24.02% <ø> (ø)
plugins-azure-service-bus 23.42% <ø> (ø)
plugins-bullmq ?
plugins-cassandra 37.77% <ø> (-0.18%) ⬇️
plugins-cookie 25.08% <ø> (ø)
plugins-cookie-parser 24.87% <ø> (ø)
plugins-crypto 24.72% <ø> (ø)
plugins-dd-trace-api 38.36% <ø> (-0.18%) ⬇️
plugins-express-mongo-sanitize 25.04% <ø> (ø)
plugins-express-session 24.83% <ø> (ø)
plugins-fastify 42.27% <ø> (-0.19%) ⬇️
plugins-fetch 38.32% <ø> (-0.17%) ⬇️
plugins-fs 38.61% <ø> (-0.18%) ⬇️
plugins-generic-pool 24.06% <ø> (ø)
plugins-google-cloud-pubsub ?
plugins-grpc 40.97% <ø> (-0.18%) ⬇️
plugins-handlebars 25.08% <ø> (ø)
plugins-hapi 40.14% <ø> (-0.18%) ⬇️
plugins-hono 40.41% <ø> (-0.18%) ⬇️
plugins-ioredis 38.41% <ø> (-0.18%) ⬇️
plugins-knex ?
plugins-ldapjs 22.61% <ø> (ø)
plugins-light-my-request 24.48% <ø> (ø)
plugins-limitd-client 32.50% <ø> (-0.19%) ⬇️
plugins-lodash 24.13% <ø> (ø)
plugins-mariadb 39.49% <ø> (-0.20%) ⬇️
plugins-memcached 38.15% <ø> (-0.18%) ⬇️
plugins-microgateway-core 39.17% <ø> (-0.18%) ⬇️
plugins-moleculer 40.53% <ø> (-0.18%) ⬇️
plugins-mongodb 39.20% <ø> (-0.16%) ⬇️
plugins-mongodb-core 39.03% <ø> (-0.17%) ⬇️
plugins-mongoose 38.85% <ø> (-0.19%) ⬇️
plugins-multer 24.83% <ø> (ø)
plugins-mysql 39.17% <ø> (-0.14%) ⬇️
plugins-mysql2 39.27% <ø> (-0.17%) ⬇️
plugins-node-serialize 25.12% <ø> (ø)
plugins-opensearch 37.60% <ø> (-0.18%) ⬇️
plugins-passport-http 24.91% <ø> (ø)
plugins-postgres 35.69% <ø> (-0.12%) ⬇️
plugins-process 24.72% <ø> (ø)
plugins-pug 25.08% <ø> (ø)
plugins-redis 38.88% <ø> (-0.19%) ⬇️
plugins-router 43.03% <ø> (-0.20%) ⬇️
plugins-sequelize 23.66% <ø> (ø)
plugins-test-and-upstream-amqp10 38.48% <ø> (-0.18%) ⬇️
plugins-test-and-upstream-amqplib ?
plugins-test-and-upstream-apollo 39.03% <ø> (-0.16%) ⬇️
plugins-test-and-upstream-avsc 38.70% <ø> (-0.23%) ⬇️
plugins-test-and-upstream-bunyan 33.79% <ø> (-0.20%) ⬇️
plugins-test-and-upstream-connect 40.81% <ø> (-0.19%) ⬇️
plugins-test-and-upstream-graphql 40.15% <ø> (-0.18%) ⬇️
plugins-test-and-upstream-koa 40.39% <ø> (-0.18%) ⬇️
plugins-test-and-upstream-protobufjs 38.93% <ø> (-0.23%) ⬇️
plugins-test-and-upstream-rhea ?
plugins-undici 39.11% <ø> (-0.17%) ⬇️
plugins-url 24.72% <ø> (ø)
plugins-valkey 38.07% <ø> (-0.18%) ⬇️
plugins-vm 24.72% <ø> (ø)
plugins-winston 33.99% <ø> (-0.19%) ⬇️
plugins-ws 41.91% <ø> (-0.19%) ⬇️
profiling-macos 39.84% <ø> (-0.24%) ⬇️
profiling-ubuntu 39.97% <ø> (-0.24%) ⬇️
profiling-windows 41.20% <ø> (-0.24%) ⬇️
serverless-azure-functions-client 23.75% <ø> (ø)
serverless-azure-functions-eventhubs 23.75% <ø> (ø)
serverless-azure-functions-servicebus 23.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…mq, rhea, google-cloud-pubsub

Same root cause as the kafkajs test: ctx.currentStore is set by
startSpan before decodeDataStreamsContext/setCheckpoint call enterWith,
so the DSM context is never included in the bound store for async
continuations.

Verified deterministic: 10/10 failures across all plugins and versions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@datadog-datadog-prod-us1

This comment has been minimized.

@robcarlan-datadog robcarlan-datadog changed the title test(kafkajs): regression test for DSM context leak between concurrent consumers test(dsm): regression tests for DSM context leak between concurrent consumers Feb 19, 2026
robcarlan-datadog and others added 2 commits February 19, 2026 13:33
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@robcarlan-datadog robcarlan-datadog force-pushed the rcarlan/dsm-context-leak-regression-test branch from c75ac65 to ee7424d Compare February 19, 2026 18:41
robcarlan-datadog and others added 2 commits February 19, 2026 14:33
Moved to #7582 to fix independently.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant