[test optimization] Prevent ATF tests from being incorrectly tagged as new in jest#7960
Conversation
…n jest
The `retriedTestsToNumAttempts` Map is shared across all retry features
(ATF, EFD, impacted tests). When a known test was marked as attempt-to-fix,
it was added to this map, causing `test_start` and `test_done` to
incorrectly identify it as a new test. This made EFD's lenient pass
criteria ("any pass = pass") override ATF's strict criteria
("all must pass"), silently suppressing real failures for flaky ATF tests.
Introduces a dedicated `newTests` Set populated only when a test is
genuinely absent from the known tests list, and uses it for the
`isNewTest` check in both `test_start` and `test_done`.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Overall package sizeSelf size: 5.47 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 3c889a3 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
BenchmarksBenchmark execution time: 2026-04-09 10:10:42 Comparing candidate commit 3c889a3 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 229 metrics, 31 unstable metrics. |
juan-fernandez
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7960 +/- ##
==========================================
- Coverage 73.87% 73.87% -0.01%
==========================================
Files 773 773
Lines 35946 35952 +6
==========================================
+ Hits 26556 26559 +3
- Misses 9390 9393 +3 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2 participants
What does this PR do?
Fixes a bug where attempt-to-fix (ATF) tests were incorrectly tagged as
test.is_new: truewhen the known tests feature was enabled. This happened becauseretriedTestsToNumAttemptsis shared across all retry features (ATF, EFD, impacted tests), andtest_start/test_doneusedretriedTestsToNumAttempts.has()to determine if a test was new.Introduces a dedicated
newTestsSet that is populated only when a test is genuinely absent from the known tests list, decoupled from the shared retry tracking map.Motivation
A customer reported that a test marked as attempt-to-fix was also being tagged as
test.is_new: true, even though the test was present in the known tests list returned by the backend. Beyond incorrect tagging, this caused a behavioral bug: EFD's lenient pass criteria ("any pass = pass") would override ATF's strict criteria ("all must pass") ingetFinalStatus, silently suppressing real failures for flaky ATF tests.Additional Notes
newTests.add()call is placed outside the!retriedTestsToNumAttempts.has()guard so that tests which are both genuinely new AND modified (impacted) are still correctly tagged as new.TEST_IS_NEWis never set. Verified it fails without the fix.