[SVLS-7168]Support GCP PubSub Batch Publish#6782
Conversation
Overall package sizeSelf size: 13.67 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.7.0 | 35.02 MB | 35.02 MB | | @datadog/native-appsec | 10.3.0 | 20.73 MB | 20.74 MB | | @datadog/pprof | 5.12.0 | 11.19 MB | 11.57 MB | | @datadog/native-iast-taint-tracking | 4.1.0 | 9.01 MB | 9.02 MB | | @opentelemetry/resources | 1.30.1 | 557.67 kB | 7.71 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.5.4 | 2.95 MB | 5.83 MB | | @datadog/wasm-js-rewriter | 5.0.1 | 2.82 MB | 3.53 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api-logs | 0.208.0 | 199.48 kB | 1.42 MB | | @opentelemetry/api | 1.9.0 | 1.22 MB | 1.22 MB | | jsonpath-plus | 10.3.0 | 617.18 kB | 1.08 MB | | import-in-the-middle | 1.15.0 | 127.66 kB | 856.24 kB | | lru-cache | 10.4.3 | 804.3 kB | 804.3 kB | | @datadog/openfeature-node-server | 0.2.0 | 118.51 kB | 437.19 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | source-map | 0.7.6 | 185.63 kB | 185.63 kB | | pprof-format | 2.2.1 | 163.06 kB | 163.06 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | @isaacs/ttlcache | 2.1.3 | 90.79 kB | 90.79 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 7.0.5 | 63.38 kB | 63.38 kB | | istanbul-lib-coverage | 3.2.2 | 34.37 kB | 34.37 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.3 | 23.74 kB | 23.74 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | mutexify | 1.4.0 | 5.71 kB | 8.74 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | module-details-from-path | 1.0.4 | 3.96 kB | 3.96 kB | | escape-string-regexp | 5.0.0 | 3.66 kB | 3.66 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
BenchmarksBenchmark execution time: 2025-12-11 14:03:18 Comparing candidate commit 75dd311 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 287 metrics, 33 unstable metrics. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## nina.rei/SVLS-7168/gcp-push-pubsub-plugin #6782 +/- ##
=============================================================================
- Coverage 84.44% 83.96% -0.48%
=============================================================================
Files 517 515 -2
Lines 22162 22359 +197
=============================================================================
+ Hits 18714 18774 +60
- Misses 3448 3585 +137 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
d7f0d5e to
3fdb180
Compare
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected * Fix with Cursor requires Datadog plugin ≥v2.17.0 🔗 Commit SHA: 87ecea4 | Docs | Datadog PR Page | Was this helpful? Give us feedback! |
d76ed3b to
f638388
Compare
8e3014c to
10eb97c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
7862105 to
505588b
Compare
a67bc15 to
ed3cc6d
Compare
- Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests
- Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1
Ensures pubsub.delivery spans are properly tagged with the correct integration name for test validation and proper categorization.
ccc9ae5 to
f7af73c
Compare
…68/gcp-pubsub-batch-plugin
…i/SVLS-7168/gcp-pubsub-batch-plugin
…i/SVLS-7168/gcp-pubsub-batch-plugin
…i/SVLS-7168/gcp-pubsub-batch-plugin
* feat: add producer-side batch message handling with span linking
- Collect span links from messages 2-N (first becomes parent)
- Extract parent context from first message trace context
- Create pubsub.request span with span links metadata
- Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*)
- Add 128-bit trace ID support (_dd.p.tid)
- Add operation tag for batched vs single requests
* feat: add ack context map and producer improvements for batching
- Add ack context map to preserve trace context across batched acknowledges
- Update producer to use batchSpan._startTime for accurate publish time
- Add explicit parent span support in client plugin
- Wrap Message.ack() to store context before batched gRPC acknowledge
- Update Subscription.emit to properly handle storage context
- Sync auto-load improvements from Branch 1
* fix: resolve linting errors in google-cloud-pubsub.js
* fix: remove trailing whitespace in client.js
* fix comments
* feat: add span linking from delivery span to pubsub.request
- Add _reconstructPubSubContext to extract pubsub.request span ID from headers
- Add span link to original trace context if different from pubsub.request
- Supports same-trace parenting for better trace continuity
* feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers
- HTTP plugin now checks if a delivery span is active in storage
- If found, uses delivery span as parent for http.request
- Ensures proper span hierarchy for push subscriptions
* feat: add span linking and batch metadata to pull-based consumer
- Extract pubsub.request span ID from message attributes
- Add span link correlation tags
- Calculate delivery duration from publish start time
- Add batch size and index tags for batched messages
* feat: add comprehensive span linking for consumer and push subscriptions
- Add _reconstructPubSubRequestContext in consumer for proper span reparenting
- Reparent first message in batch to pubsub.request span
- Add comprehensive batch metadata and correlation tags
- Improve resource naming and service separation
- Add delivery duration calculation
- Update push subscription logging to warn level with better messages
- Add missing headers warning for troubleshooting
* Remove comments
* run npm lint
* fix test
* fix test
* fix test
* fix index.js test
* fix index.js test
* fix context error
* fix lint
* fix lint
* fix lint
* fix index.js test
* fix plugin
* add ctx
* test the index
* update consumer subscription bindstart
* update addhook subscription
* add debug logs
* Fix consumer span context loss with WeakMap and add comprehensive debug logging
* Fix double finish
* Fix consumer span type: explicitly set _type='worker' after startSpan
* Fix: use span.setTag('span.type', 'worker') instead of span._type
* Debug: Add type in startSpan options and via setTag, improve logging
* Fix: Use runStores for _dispense and publish for remove with improved logging
* add debug logs
* Fix: Guarantee test harness loads instrumentation before requiring @google-cloud/pubsub
* update subscription
* update subscription
* test fix for index.spec.js timeout
* fix test logic
* fix linter
* fix linter
* remove comments
* feat: add producer-side batch message handling with span linking
- Collect span links from messages 2-N (first becomes parent)
- Extract parent context from first message trace context
- Create pubsub.request span with span links metadata
- Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*)
- Add 128-bit trace ID support (_dd.p.tid)
- Add operation tag for batched vs single requests
* feat: add ack context map and producer improvements for batching
- Add ack context map to preserve trace context across batched acknowledges
- Update producer to use batchSpan._startTime for accurate publish time
- Add explicit parent span support in client plugin
- Wrap Message.ack() to store context before batched gRPC acknowledge
- Update Subscription.emit to properly handle storage context
- Sync auto-load improvements from Branch 1
* fix: resolve linting errors in google-cloud-pubsub.js
* fix: remove trailing whitespace in client.js
* fix comments
* update from review
* addional cleanup
* [SVLS-7168] Create tests for GCP PubSub Push Subscriptions Plugin (#6414)
* feat: add producer-side batch message handling with span linking
- Collect span links from messages 2-N (first becomes parent)
- Extract parent context from first message trace context
- Create pubsub.request span with span links metadata
- Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*)
- Add 128-bit trace ID support (_dd.p.tid)
- Add operation tag for batched vs single requests
* feat: add span linking from delivery span to pubsub.request
- Add _reconstructPubSubContext to extract pubsub.request span ID from headers
- Add span link to original trace context if different from pubsub.request
- Supports same-trace parenting for better trace continuity
* feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers
- HTTP plugin now checks if a delivery span is active in storage
- If found, uses delivery span as parent for http.request
- Ensures proper span hierarchy for push subscriptions
* feat: add span linking and batch metadata to pull-based consumer
- Extract pubsub.request span ID from message attributes
- Add span link correlation tags
- Calculate delivery duration from publish start time
- Add batch size and index tags for batched messages
* remove comments
* new test file
* Fix push subscription test module resolution for CI
* Implement full HTTP+Pub/Sub integration tests
* run linter
* test other frameworks
* Fix push subscription tests
fa47166
into
nina.rei/SVLS-7168/gcp-push-pubsub-plugin
* feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * small cleanup from merge * Add integrationName to fix _dd.integration tag in push subscription Ensures pubsub.delivery spans are properly tagged with the correct integration name for test validation and proper categorization. * update from review * [SVLS-7168] Create inferred Span and Span links for GCP PubSub (#6415) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * feat: add comprehensive span linking for consumer and push subscriptions - Add _reconstructPubSubRequestContext in consumer for proper span reparenting - Reparent first message in batch to pubsub.request span - Add comprehensive batch metadata and correlation tags - Improve resource naming and service separation - Add delivery duration calculation - Update push subscription logging to warn level with better messages - Add missing headers warning for troubleshooting * Remove comments * run npm lint * fix test * fix test * fix test * fix index.js test * fix index.js test * fix context error * fix lint * fix lint * fix lint * fix index.js test * fix plugin * add ctx * test the index * update consumer subscription bindstart * update addhook subscription * add debug logs * Fix consumer span context loss with WeakMap and add comprehensive debug logging * Fix double finish * Fix consumer span type: explicitly set _type='worker' after startSpan * Fix: use span.setTag('span.type', 'worker') instead of span._type * Debug: Add type in startSpan options and via setTag, improve logging * Fix: Use runStores for _dispense and publish for remove with improved logging * add debug logs * Fix: Guarantee test harness loads instrumentation before requiring @google-cloud/pubsub * update subscription * update subscription * test fix for index.spec.js timeout * fix test logic * fix linter * fix linter * remove comments * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * update from review * addional cleanup * [SVLS-7168] Create tests for GCP PubSub Push Subscriptions Plugin (#6414) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * remove comments * new test file * Fix push subscription test module resolution for CI * Implement full HTTP+Pub/Sub integration tests * run linter * test other frameworks * Fix push subscription tests
* feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * small cleanup from merge * Add integrationName to fix _dd.integration tag in push subscription Ensures pubsub.delivery spans are properly tagged with the correct integration name for test validation and proper categorization. * update from review * [SVLS-7168] Create inferred Span and Span links for GCP PubSub (#6415) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * feat: add comprehensive span linking for consumer and push subscriptions - Add _reconstructPubSubRequestContext in consumer for proper span reparenting - Reparent first message in batch to pubsub.request span - Add comprehensive batch metadata and correlation tags - Improve resource naming and service separation - Add delivery duration calculation - Update push subscription logging to warn level with better messages - Add missing headers warning for troubleshooting * Remove comments * run npm lint * fix test * fix test * fix test * fix index.js test * fix index.js test * fix context error * fix lint * fix lint * fix lint * fix index.js test * fix plugin * add ctx * test the index * update consumer subscription bindstart * update addhook subscription * add debug logs * Fix consumer span context loss with WeakMap and add comprehensive debug logging * Fix double finish * Fix consumer span type: explicitly set _type='worker' after startSpan * Fix: use span.setTag('span.type', 'worker') instead of span._type * Debug: Add type in startSpan options and via setTag, improve logging * Fix: Use runStores for _dispense and publish for remove with improved logging * add debug logs * Fix: Guarantee test harness loads instrumentation before requiring @google-cloud/pubsub * update subscription * update subscription * test fix for index.spec.js timeout * fix test logic * fix linter * fix linter * remove comments * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * update from review * addional cleanup * [SVLS-7168] Create tests for GCP PubSub Push Subscriptions Plugin (#6414) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * remove comments * new test file * Fix push subscription test module resolution for CI * Implement full HTTP+Pub/Sub integration tests * run linter * test other frameworks * Fix push subscription tests
* feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * small cleanup from merge * Add integrationName to fix _dd.integration tag in push subscription Ensures pubsub.delivery spans are properly tagged with the correct integration name for test validation and proper categorization. * update from review * [SVLS-7168] Create inferred Span and Span links for GCP PubSub (#6415) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * feat: add comprehensive span linking for consumer and push subscriptions - Add _reconstructPubSubRequestContext in consumer for proper span reparenting - Reparent first message in batch to pubsub.request span - Add comprehensive batch metadata and correlation tags - Improve resource naming and service separation - Add delivery duration calculation - Update push subscription logging to warn level with better messages - Add missing headers warning for troubleshooting * Remove comments * run npm lint * fix test * fix test * fix test * fix index.js test * fix index.js test * fix context error * fix lint * fix lint * fix lint * fix index.js test * fix plugin * add ctx * test the index * update consumer subscription bindstart * update addhook subscription * add debug logs * Fix consumer span context loss with WeakMap and add comprehensive debug logging * Fix double finish * Fix consumer span type: explicitly set _type='worker' after startSpan * Fix: use span.setTag('span.type', 'worker') instead of span._type * Debug: Add type in startSpan options and via setTag, improve logging * Fix: Use runStores for _dispense and publish for remove with improved logging * add debug logs * Fix: Guarantee test harness loads instrumentation before requiring @google-cloud/pubsub * update subscription * update subscription * test fix for index.spec.js timeout * fix test logic * fix linter * fix linter * remove comments * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * update from review * addional cleanup * [SVLS-7168] Create tests for GCP PubSub Push Subscriptions Plugin (#6414) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * remove comments * new test file * Fix push subscription test module resolution for CI * Implement full HTTP+Pub/Sub integration tests * run linter * test other frameworks * Fix push subscription tests
* feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * small cleanup from merge * Add integrationName to fix _dd.integration tag in push subscription Ensures pubsub.delivery spans are properly tagged with the correct integration name for test validation and proper categorization. * update from review * [SVLS-7168] Create inferred Span and Span links for GCP PubSub (#6415) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * feat: add comprehensive span linking for consumer and push subscriptions - Add _reconstructPubSubRequestContext in consumer for proper span reparenting - Reparent first message in batch to pubsub.request span - Add comprehensive batch metadata and correlation tags - Improve resource naming and service separation - Add delivery duration calculation - Update push subscription logging to warn level with better messages - Add missing headers warning for troubleshooting * Remove comments * run npm lint * fix test * fix test * fix test * fix index.js test * fix index.js test * fix context error * fix lint * fix lint * fix lint * fix index.js test * fix plugin * add ctx * test the index * update consumer subscription bindstart * update addhook subscription * add debug logs * Fix consumer span context loss with WeakMap and add comprehensive debug logging * Fix double finish * Fix consumer span type: explicitly set _type='worker' after startSpan * Fix: use span.setTag('span.type', 'worker') instead of span._type * Debug: Add type in startSpan options and via setTag, improve logging * Fix: Use runStores for _dispense and publish for remove with improved logging * add debug logs * Fix: Guarantee test harness loads instrumentation before requiring @google-cloud/pubsub * update subscription * update subscription * test fix for index.spec.js timeout * fix test logic * fix linter * fix linter * remove comments * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * update from review * addional cleanup * [SVLS-7168] Create tests for GCP PubSub Push Subscriptions Plugin (#6414) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * remove comments * new test file * Fix push subscription test module resolution for CI * Implement full HTTP+Pub/Sub integration tests * run linter * test other frameworks * Fix push subscription tests
* feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * small cleanup from merge * Add integrationName to fix _dd.integration tag in push subscription Ensures pubsub.delivery spans are properly tagged with the correct integration name for test validation and proper categorization. * update from review * [SVLS-7168] Create inferred Span and Span links for GCP PubSub (#6415) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * feat: add comprehensive span linking for consumer and push subscriptions - Add _reconstructPubSubRequestContext in consumer for proper span reparenting - Reparent first message in batch to pubsub.request span - Add comprehensive batch metadata and correlation tags - Improve resource naming and service separation - Add delivery duration calculation - Update push subscription logging to warn level with better messages - Add missing headers warning for troubleshooting * Remove comments * run npm lint * fix test * fix test * fix test * fix index.js test * fix index.js test * fix context error * fix lint * fix lint * fix lint * fix index.js test * fix plugin * add ctx * test the index * update consumer subscription bindstart * update addhook subscription * add debug logs * Fix consumer span context loss with WeakMap and add comprehensive debug logging * Fix double finish * Fix consumer span type: explicitly set _type='worker' after startSpan * Fix: use span.setTag('span.type', 'worker') instead of span._type * Debug: Add type in startSpan options and via setTag, improve logging * Fix: Use runStores for _dispense and publish for remove with improved logging * add debug logs * Fix: Guarantee test harness loads instrumentation before requiring @google-cloud/pubsub * update subscription * update subscription * test fix for index.spec.js timeout * fix test logic * fix linter * fix linter * remove comments * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add ack context map and producer improvements for batching - Add ack context map to preserve trace context across batched acknowledges - Update producer to use batchSpan._startTime for accurate publish time - Add explicit parent span support in client plugin - Wrap Message.ack() to store context before batched gRPC acknowledge - Update Subscription.emit to properly handle storage context - Sync auto-load improvements from Branch 1 * fix: resolve linting errors in google-cloud-pubsub.js * fix: remove trailing whitespace in client.js * fix comments * update from review * addional cleanup * [SVLS-7168] Create tests for GCP PubSub Push Subscriptions Plugin (#6414) * feat: add producer-side batch message handling with span linking - Collect span links from messages 2-N (first becomes parent) - Extract parent context from first message trace context - Create pubsub.request span with span links metadata - Inject batch metadata into all messages (_dd.pubsub_request.*, _dd.batch.*) - Add 128-bit trace ID support (_dd.p.tid) - Add operation tag for batched vs single requests * feat: add span linking from delivery span to pubsub.request - Add _reconstructPubSubContext to extract pubsub.request span ID from headers - Add span link to original trace context if different from pubsub.request - Supports same-trace parenting for better trace continuity * feat: check for pubsub.delivery span in AsyncLocalStorage before extracting from headers - HTTP plugin now checks if a delivery span is active in storage - If found, uses delivery span as parent for http.request - Ensures proper span hierarchy for push subscriptions * feat: add span linking and batch metadata to pull-based consumer - Extract pubsub.request span ID from message attributes - Add span link correlation tags - Calculate delivery duration from publish start time - Add batch size and index tags for batched messages * remove comments * new test file * Fix push subscription test module resolution for CI * Implement full HTTP+Pub/Sub integration tests * run linter * test other frameworks * Fix push subscription tests
What does this PR do?
This PR implements comprehensive batch message handling for GCP Pub/Sub, addressing a critical challenge: the Pub/Sub client library automatically batches messages from multiple independent requests, causing async context loss. GCP Pub/Sub has transparent internal batching that happens on the publisher side and internally inside the topic.
When multiple HTTP requests each call
topic.publishMessage(), they return immediately to the user. But behind the scenes, the Pub/Sub client queues these messages and N seconds later makes ONE batched gRPC call with ALL the messages together. So by the time the batched gRPC call happens, the original HTTP requests have completed and their async contexts are gone. We're left with either no active span or the wrong span active, leading to miss-parented spans. If we only instrumented at the gRPC level, we'd have no way to determine which trace each message came from.To handle this we introduced dual-wrapping:
High-Level Wrapping (
Topic.publishMessage / Topic.publish):Low-Level Wrapping (
PublisherClient.publish):Producer Plugin Enhancements
- Creates span links to all other messages in the batch (messages 2-N)
Next PR in the batch is #6415
Motivation
Plugin Checklist
Additional Notes
An inferred span for the push subscription HTTP POST request to the Cloud Run service from a pub/sub topic
Example full Push Distributed Trace of a cloud run service triggering another service using a push subscription