Skip to content

test(debugger): fix flaky time budget integration test#7103

Merged
BridgeAR merged 2 commits intomasterfrom
watson/fix-flaky
Dec 18, 2025
Merged

test(debugger): fix flaky time budget integration test#7103
BridgeAR merged 2 commits intomasterfrom
watson/fix-flaky

Conversation

@watson
Copy link
Copy Markdown
Collaborator

@watson watson commented Dec 15, 2025

What does this PR do?

Fixes a flaky integration test for the debugger's time budget mechanism by adding dedicated unit tests with mocked time and removing strict timing assertions from the integration test.

The changes add a new unit test file (collector-deadline.spec.js) that uses mocked process.hrtime.bigint() to deterministically test the deadline checking logic in the snapshot collector. The problematic 1ms budget integration test now only validates behavioral outcomes (that timeout markers appear in snapshots) without asserting on exact execution time, which was causing CI failures.

Motivation

The snapshot-time-budget.spec.js integration test was flaky in GitHub Actions CI environments. The test set a 1ms capture timeout and expected the thread to be paused for ≤16ms (1ms + 15ms tolerance). This assertion frequently failed in CI due to:

  • Resource-constrained or throttled GitHub Actions VMs
  • Unpredictable execution times in shared infrastructure
  • The artificially tight timing constraints (16ms total tolerance)

The flakiness was affecting CI reliability while not providing meaningful test coverage - the important behavior is that timeout markers appear, not the exact pause duration. By adding unit tests with mocked time for the actual deadline logic and relaxing the integration test to only check behavioral outcomes, we maintain full test coverage while eliminating CI flakiness.

Adds unit tests with mocked time for the debugger snapshot collector
deadline mechanism and removes strict timing assertions from the flaky
1ms budget integration test.

The integration test was failing in CI due to unpredictable execution
times in GitHub Actions environments. The test set a 1ms capture timeout
and expected thread pause time ≤16ms, which was too tight for
resource-constrained CI environments.
@watson watson requested review from a team as code owners December 15, 2025 09:11
Copy link
Copy Markdown
Collaborator Author

watson commented Dec 15, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@watson watson self-assigned this Dec 15, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 15, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.77%. Comparing base (a8ed0d8) to head (e662378).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7103      +/-   ##
==========================================
- Coverage   84.78%   84.77%   -0.01%     
==========================================
  Files         521      521              
  Lines       22149    22149              
==========================================
- Hits        18778    18776       -2     
- Misses       3371     3373       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 15, 2025

Overall package size

Self size: 3.59 MB
Deduped: 4.47 MB
No deduping: 4.47 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 1.15.0 | 127.66 kB | 856.24 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Dec 15, 2025

Benchmarks

Benchmark execution time: 2025-12-16 10:41:14

Comparing candidate commit e662378 in PR branch watson/fix-flaky with baseline commit a8ed0d8 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 291 metrics, 29 unstable metrics.

@watson watson added the debugger Dynamic Instrumentation & Live Debugger label Dec 15, 2025
BridgeAR
BridgeAR previously approved these changes Dec 15, 2025
Copy link
Copy Markdown
Member

@BridgeAR BridgeAR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, while I would use the other mock

})

it('should not mark properties with timeout when deadline is not exceeded', async function () {
process.hrtime = /** @type {any} */ ({ bigint: () => 0n })
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible to also mock this with sinon: sinon.useFakeTimers()

Copy link
Copy Markdown
Collaborator Author

@watson watson Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did consider it, but didn't think it was worth while. I just pushed a commit with your suggestion, but I'm not sure the code is that much better. Maybe it's slightly worse depending on how you look at it. Let me know if you prefer it or if I should revert the last commit

@datadog-datadog-prod-us1
Copy link
Copy Markdown

datadog-datadog-prod-us1 Bot commented Dec 16, 2025

⚠️ Tests

Fix all issues with Cursor

⚠️ Warnings

❄️ 1 New flaky test detected

tests.test_resource_renaming.Test_Resource_Renaming_Stats_Aggregation_Keys.test_stats_aggregation_with_method_and_endpoint[fastify] from system_tests_suite (Datadog) (Fix with Cursor)
AssertionError: Expected 5 hits for GET /resource_renaming/api/users/{param:int}, got 4
assert 4 == 5

self = <tests.test_resource_renaming.Test_Resource_Renaming_Stats_Aggregation_Keys object at 0x7feaa8e61a30>

    @bug(library="python", reason="APMSP-2359")  # trace exporter uses a wrong fieldname
    def test_stats_aggregation_with_method_and_endpoint(self):
        """Test that stats are aggregated by method and endpoint"""
        stats_points = []
        for stats_data in interfaces.library.get_data("/v0.6/stats"):
...

ℹ️ Info

🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e662378 | Docs | Datadog PR Page | Was this helpful? Give us feedback!

@BridgeAR BridgeAR merged commit a37816b into master Dec 18, 2025
1064 of 1066 checks passed
@BridgeAR BridgeAR deleted the watson/fix-flaky branch December 18, 2025 19:25
dd-octo-sts Bot pushed a commit that referenced this pull request Dec 19, 2025
Adds unit tests with mocked time for the debugger snapshot collector
deadline mechanism and removes strict timing assertions from the flaky
1ms budget integration test.

The integration test was failing in CI due to unpredictable execution
times in GitHub Actions environments. The test set a 1ms capture timeout
and expected thread pause time ≤16ms, which was too tight for
resource-constrained CI environments.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Dec 19, 2025
nina9753 pushed a commit that referenced this pull request Dec 20, 2025
Adds unit tests with mocked time for the debugger snapshot collector
deadline mechanism and removes strict timing assertions from the flaky
1ms budget integration test.

The integration test was failing in CI due to unpredictable execution
times in GitHub Actions environments. The test set a 1ms capture timeout
and expected thread pause time ≤16ms, which was too tight for
resource-constrained CI environments.
nina9753 pushed a commit that referenced this pull request Jan 15, 2026
Adds unit tests with mocked time for the debugger snapshot collector
deadline mechanism and removes strict timing assertions from the flaky
1ms budget integration test.

The integration test was failing in CI due to unpredictable execution
times in GitHub Actions environments. The test set a 1ms capture timeout
and expected thread pause time ≤16ms, which was too tight for
resource-constrained CI environments.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debugger Dynamic Instrumentation & Live Debugger semver-patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants