Skip to content

[test-optimization] [SDTEST-1830] Add @test.retry_reason to the retries of different frameworks#5550

Merged
Mariovido merged 11 commits intomasterfrom
mario.vidal/add_test_retry_reason
Apr 10, 2025
Merged

[test-optimization] [SDTEST-1830] Add @test.retry_reason to the retries of different frameworks#5550
Mariovido merged 11 commits intomasterfrom
mario.vidal/add_test_retry_reason

Conversation

@Mariovido
Copy link
Copy Markdown
Contributor

@Mariovido Mariovido commented Apr 8, 2025

What does this PR do?

This PR adds @test.retry_reason tag to all retries. The new type of retries are the following ones:

  • early_flake_detection previously (efd): The retry is from Early Flake Detection
  • auto_test_retry: The retry is from Auto Test Retries
  • attempt_to_fix: The retry is from Test Management (Attempt to Fix)
  • external: The retry is from an unknown source

The frameworks which tags are changed are the following ones:

  • Jest
  • Mocha
  • Cucumber
  • Playwright
  • Cypress
  • Vitest

Motivation

Previously we were missing the tags in the retries and it was hard to identify which type of retry it was. Specially for the cases where the retry was done by the testing framework.

Plugin Checklist

Additional Notes

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2025

Overall package size

Self size: 9.28 MB
Deduped: 101.68 MB
No deduping: 102.2 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.5.0 | 29.83 MB | 29.83 MB | | @datadog/native-appsec | 8.5.1 | 19.26 MB | 19.27 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.7.0 | 9.89 MB | 10.26 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.4.0 | 2.77 MB | 5.42 MB | | @datadog/wasm-js-rewriter | 3.1.0 | 2.37 MB | 2.52 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.13.1 | 117.64 kB | 839.26 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | dc-polyfill | 0.1.6 | 24.56 kB | 24.56 kB | | shell-quote | 1.8.2 | 23.54 kB | 23.54 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-datadog-prod-us1
Copy link
Copy Markdown

datadog-datadog-prod-us1 Bot commented Apr 8, 2025

Datadog Report

Branch report: mario.vidal/add_test_retry_reason
Commit report: 0132d2b
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 930 Passed, 0 Skipped, 17m 25.07s Total Time

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 8, 2025

Codecov Report

Attention: Patch coverage is 4.34783% with 22 lines in your changes missing coverage. Please review.

Project coverage is 79.23%. Comparing base (c2b55f4) to head (e6034a4).
Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
packages/datadog-plugin-mocha/src/index.js 0.00% 9 Missing ⚠️
packages/datadog-plugin-cucumber/src/index.js 0.00% 7 Missing ⚠️
...ckages/datadog-instrumentations/src/mocha/utils.js 0.00% 3 Missing ⚠️
packages/datadog-instrumentations/src/cucumber.js 0.00% 2 Missing ⚠️
...ackages/datadog-instrumentations/src/mocha/main.js 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5550      +/-   ##
==========================================
- Coverage   79.26%   79.23%   -0.03%     
==========================================
  Files         512      512              
  Lines       23195    23232      +37     
==========================================
+ Hits        18386    18409      +23     
- Misses       4809     4823      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Apr 8, 2025

Benchmarks

Benchmark execution time: 2025-04-10 08:34:41

Comparing candidate commit e6034a4 in PR branch mario.vidal/add_test_retry_reason with baseline commit c2b55f4 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 947 metrics, 16 unstable metrics.

@Mariovido Mariovido changed the title [WIP] [test-optimization] [SDTEST-1821] Add @test.retry_reason to the retries of different frameworks [test-optimization] [SDTEST-1821] Add @test.retry_reason to the retries of different frameworks Apr 9, 2025
@Mariovido Mariovido changed the title [test-optimization] [SDTEST-1821] Add @test.retry_reason to the retries of different frameworks [test-optimization] [SDTEST-1830] Add @test.retry_reason to the retries of different frameworks Apr 9, 2025
@Mariovido Mariovido marked this pull request as ready for review April 9, 2025 15:47
@Mariovido Mariovido requested a review from a team as a code owner April 9, 2025 15:47
Comment thread packages/datadog-plugin-cucumber/src/index.js Outdated
Comment thread packages/datadog-plugin-cypress/src/cypress-plugin.js Outdated
@Mariovido Mariovido requested a review from a team as a code owner April 10, 2025 08:23
@Mariovido Mariovido requested a review from anmarchenko April 10, 2025 08:31
@Mariovido Mariovido merged commit 38f62e9 into master Apr 10, 2025
427 checks passed
@Mariovido Mariovido deleted the mario.vidal/add_test_retry_reason branch April 10, 2025 08:44
github-actions Bot pushed a commit that referenced this pull request Apr 10, 2025
This was referenced Apr 10, 2025
github-actions Bot pushed a commit that referenced this pull request Apr 10, 2025
@github-actions github-actions Bot mentioned this pull request Apr 10, 2025
bengl pushed a commit that referenced this pull request Apr 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants