Skip to content

feat(debugger): implement intelligent snapshot pruning for oversized payloads#7006

Merged
watson merged 4 commits intomasterfrom
watson/DEBUG-2624/implement-pruning-algo
Dec 17, 2025
Merged

feat(debugger): implement intelligent snapshot pruning for oversized payloads#7006
watson merged 4 commits intomasterfrom
watson/DEBUG-2624/implement-pruning-algo

Conversation

@watson
Copy link
Copy Markdown
Collaborator

@watson watson commented Nov 27, 2025

What does this PR do?

Implements an intelligent snapshot pruning algorithm for Dynamic Instrumentation/Live Debugger that selectively removes the largest and deepest leaf nodes from oversized payloads while preserving the overall schema structure. This replaces the previous approach which simply deleted all captured variables when a snapshot exceeded the 1MB size limit.

Motivation

Align with how the other tracers prunes large snapshots.

Previously, when a snapshot payload exceeded 1MB, we would delete all captures entirely and show an error message to users. This was a poor user experience because users would lose all captured variable data, even though most of the snapshot was likely still valuable.

With this new pruning algorithm, we can intelligently reduce the size of oversized snapshots by removing only the least valuable data (deepest, largest nodes, and nodes that were already truncated due to depth limits), allowing users to still see most of their captured variables even when dealing with large data structures.

Additional Notes

The pruning algorithm:

  • Parses snapshots into a tree structure tracking JSON object positions
  • Uses a priority queue to select nodes for pruning based on:
    1. Presence of notCapturedReason: 'depth' (highest priority - already truncated data)
    2. Depth level (deeper nodes pruned first)
    3. Presence of any notCapturedReason (any truncated data)
    4. Size (larger nodes pruned first)
  • Only prunes nodes at level 6 or deeper (locals and below)
  • Promotes parent nodes when all children are pruned to reduce overhead
  • Iteratively prunes if needed to reach target size
  • Falls back to prune all everything if pruning fails or errors out

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Nov 27, 2025

Overall package size

Self size: 4.35 MB
Deduped: 5.23 MB
No deduping: 5.23 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 1.15.0 | 127.66 kB | 856.24 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

Copy link
Copy Markdown
Collaborator Author

watson commented Nov 27, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 27, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.76%. Comparing base (86b4f71) to head (2a18882).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7006      +/-   ##
==========================================
- Coverage   84.77%   84.76%   -0.01%     
==========================================
  Files         521      521              
  Lines       22149    22151       +2     
==========================================
  Hits        18776    18776              
- Misses       3373     3375       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Nov 27, 2025

Benchmarks

Benchmark execution time: 2025-12-17 12:38:44

Comparing candidate commit 2a18882 in PR branch watson/DEBUG-2624/implement-pruning-algo with baseline commit 86b4f71 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 291 metrics, 29 unstable metrics.

@watson watson self-assigned this Nov 28, 2025
@watson watson added semver-minor debugger Dynamic Instrumentation & Live Debugger labels Nov 28, 2025
@watson watson force-pushed the watson/DEBUG-2624/implement-pruning-algo branch 2 times, most recently from ecf4d0c to 2c58667 Compare November 28, 2025 10:05
@datadog-datadog-prod-us1

This comment has been minimized.

@watson watson force-pushed the watson/DEBUG-2624/implement-pruning-algo branch 2 times, most recently from 90f07f4 to a217410 Compare November 30, 2025 07:13
@watson watson marked this pull request as ready for review November 30, 2025 07:16
@watson watson requested review from a team as code owners November 30, 2025 07:16
@watson watson force-pushed the watson/DEBUG-2624/implement-pruning-algo branch 2 times, most recently from 7d33a6c to 693c32a Compare December 1, 2025 10:41
Comment thread packages/dd-trace/src/debugger/devtools_client/snapshot-pruner.js Outdated
Comment thread packages/dd-trace/src/debugger/devtools_client/snapshot-pruner.js Outdated
@watson watson force-pushed the watson/DEBUG-2624/implement-pruning-algo branch from 693c32a to 4f0478a Compare December 16, 2025 12:58
@watson watson requested a review from shatzi December 16, 2025 12:58
shatzi
shatzi previously approved these changes Dec 16, 2025
Copy link
Copy Markdown

@shatzi shatzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

Comment thread packages/dd-trace/src/debugger/devtools_client/snapshot-pruner.js Outdated
…payloads

Replace the crude "delete all captures" approach with an intelligent pruning
algorithm that selectively removes the largest and deepest leaf nodes while
preserving the schema structure.

The algorithm prunes like so:

- Parses snapshots into a tree structure tracking JSON object positions
- Uses a priority queue to select nodes for pruning based on:
  1. Presence of `notCapturedReason: 'depth'`
  2. Depth level (deeper nodes pruned first)
  3. Presence of any `notCapturedReason`
  4. Size (larger nodes pruned first)
- Only prunes nodes at level 6 or deeper (`locals`)
- Promotes parent nodes when all children are pruned to reduce overhead
- Iteratively prunes if needed to reach target size
@watson watson force-pushed the watson/DEBUG-2624/implement-pruning-algo branch from 4f0478a to 8e9d8c7 Compare December 17, 2025 12:25
@watson watson requested a review from shatzi December 17, 2025 12:27
@watson watson force-pushed the watson/DEBUG-2624/implement-pruning-algo branch from 8e9d8c7 to 2a18882 Compare December 17, 2025 12:30
@watson watson merged commit 48cbba9 into master Dec 17, 2025
793 checks passed
@watson watson deleted the watson/DEBUG-2624/implement-pruning-algo branch December 17, 2025 13:41
dd-octo-sts Bot pushed a commit that referenced this pull request Dec 17, 2025
…payloads (#7006)

Replace the crude "delete all captures" approach with an intelligent pruning
algorithm that selectively removes the largest and deepest leaf nodes while
preserving the schema structure.

The algorithm prunes like so:

- Parses snapshots into a tree structure tracking JSON object positions
- Uses a priority queue to select nodes for pruning based on:
  1. Presence of `notCapturedReason: 'depth'`
  2. Depth level (deeper nodes pruned first)
  3. Presence of any `notCapturedReason`
  4. Size (larger nodes pruned first)
- Only prunes nodes at level 6 or deeper (`locals`)
- Promotes parent nodes when all children are pruned to reduce overhead
- Iteratively prunes if needed to reach target size
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Dec 17, 2025
BridgeAR pushed a commit that referenced this pull request Dec 17, 2025
…payloads (#7006)

Replace the crude "delete all captures" approach with an intelligent pruning
algorithm that selectively removes the largest and deepest leaf nodes while
preserving the schema structure.

The algorithm prunes like so:

- Parses snapshots into a tree structure tracking JSON object positions
- Uses a priority queue to select nodes for pruning based on:
  1. Presence of `notCapturedReason: 'depth'`
  2. Depth level (deeper nodes pruned first)
  3. Presence of any `notCapturedReason`
  4. Size (larger nodes pruned first)
- Only prunes nodes at level 6 or deeper (`locals`)
- Promotes parent nodes when all children are pruned to reduce overhead
- Iteratively prunes if needed to reach target size
nina9753 pushed a commit that referenced this pull request Dec 20, 2025
…payloads (#7006)

Replace the crude "delete all captures" approach with an intelligent pruning
algorithm that selectively removes the largest and deepest leaf nodes while
preserving the schema structure.

The algorithm prunes like so:

- Parses snapshots into a tree structure tracking JSON object positions
- Uses a priority queue to select nodes for pruning based on:
  1. Presence of `notCapturedReason: 'depth'`
  2. Depth level (deeper nodes pruned first)
  3. Presence of any `notCapturedReason`
  4. Size (larger nodes pruned first)
- Only prunes nodes at level 6 or deeper (`locals`)
- Promotes parent nodes when all children are pruned to reduce overhead
- Iteratively prunes if needed to reach target size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debugger Dynamic Instrumentation & Live Debugger semver-minor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants