Skip to content

test: make assertions order-insensitive to prevent failing tests based on Playwright version#7129

Merged
calvinbayer merged 1 commit intomasterfrom
calvin.bayer/test/fix-failing-playwright-tests
Dec 17, 2025
Merged

test: make assertions order-insensitive to prevent failing tests based on Playwright version#7129
calvinbayer merged 1 commit intomasterfrom
calvin.bayer/test/fix-failing-playwright-tests

Conversation

@calvinbayer
Copy link
Copy Markdown
Contributor

@calvinbayer calvinbayer commented Dec 17, 2025

What does this PR do?

  • Playwright tests are failing in the 5.81.0 release proposal PR
  • During the migration to node.js asserts, the Playwright test assertions were changed from order-insensitive array comparison to order-sensitive comparison. The tests run against the oldest supported Playwright version, which depends on the dd-trace major version: on master this is v6 with Playwright 1.38.0, while on the v5.x branch it is v5 with Playwright 1.18.0. It looks like Playwright orders test execution differently in 1.18.0 than in 1.38.0, which is why the tests pass on master but fail on the v5.81.0-proposal branch
  • Sorting the array before comparison allows to use the node.js asserts while restoring the old order-insensitive assertions

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Dec 17, 2025

Overall package size

Self size: 4.35 MB
Deduped: 5.23 MB
No deduping: 5.23 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 1.15.0 | 127.66 kB | 856.24 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 17, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.76%. Comparing base (48cbba9) to head (82fd8a4).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7129      +/-   ##
==========================================
- Coverage   84.77%   84.76%   -0.01%     
==========================================
  Files         521      521              
  Lines       22151    22151              
==========================================
- Hits        18778    18776       -2     
- Misses       3373     3375       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-official

This comment has been minimized.

@calvinbayer calvinbayer force-pushed the calvin.bayer/test/fix-failing-playwright-tests branch from 530d6a7 to beee200 Compare December 17, 2025 16:00
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Dec 17, 2025

Benchmarks

Benchmark execution time: 2025-12-17 16:08:29

Comparing candidate commit beee200 in PR branch calvin.bayer/test/fix-failing-playwright-tests with baseline commit 48cbba9 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 290 metrics, 30 unstable metrics.

@calvinbayer calvinbayer force-pushed the calvin.bayer/test/fix-failing-playwright-tests branch from beee200 to 9988aaa Compare December 17, 2025 16:10
@calvinbayer calvinbayer force-pushed the calvin.bayer/test/fix-failing-playwright-tests branch from 9988aaa to 82fd8a4 Compare December 17, 2025 16:16
@calvinbayer calvinbayer marked this pull request as ready for review December 17, 2025 16:35
@calvinbayer calvinbayer requested a review from a team as a code owner December 17, 2025 16:35
@calvinbayer calvinbayer merged commit f6e1219 into master Dec 17, 2025
793 checks passed
@calvinbayer calvinbayer deleted the calvin.bayer/test/fix-failing-playwright-tests branch December 17, 2025 16:40
dd-octo-sts Bot pushed a commit that referenced this pull request Dec 17, 2025
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Dec 17, 2025
BridgeAR pushed a commit that referenced this pull request Dec 17, 2025
nina9753 pushed a commit that referenced this pull request Dec 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants