perf(browser): improve browser parallelisation

[sheremet-va]

Description

Fixes #7616
Fixes #6834

This made tests run 2 as fast as before in PW provider for me

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Name	Link
🔨 Latest commit	c58a06e
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/67eab87debd281000854d2b0
😎 Deploy Preview	https://deploy-preview-7665--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[pkg-pr-new]

@vitest/browser

npm i https://pkg.pr.new/@vitest/browser@7665

@vitest/coverage-istanbul

npm i https://pkg.pr.new/@vitest/coverage-istanbul@7665

@vitest/coverage-v8

npm i https://pkg.pr.new/@vitest/coverage-v8@7665

@vitest/expect

npm i https://pkg.pr.new/@vitest/expect@7665

@vitest/mocker

npm i https://pkg.pr.new/@vitest/mocker@7665

@vitest/pretty-format

npm i https://pkg.pr.new/@vitest/pretty-format@7665

@vitest/runner

npm i https://pkg.pr.new/@vitest/runner@7665

@vitest/spy

npm i https://pkg.pr.new/@vitest/spy@7665

@vitest/snapshot

npm i https://pkg.pr.new/@vitest/snapshot@7665

@vitest/ui

npm i https://pkg.pr.new/@vitest/ui@7665

@vitest/utils

npm i https://pkg.pr.new/@vitest/utils@7665

vite-node

npm i https://pkg.pr.new/vite-node@7665

vitest

npm i https://pkg.pr.new/vitest@7665

@vitest/web-worker

npm i https://pkg.pr.new/@vitest/web-worker@7665

@vitest/ws-client

npm i https://pkg.pr.new/@vitest/ws-client@7665

commit: c58a06e

[sheremet-va]

@nstepien can you check if this improves things for you? watch mode, no-isolation and debugging don't work properly now, but I think just running with --run should work

[nstepien]

With our unchanged config:

npx vitest run

 RUN  v3.0.8 D:/repo

4:04:53 PM [vite] (client) Re-optimizing dependencies because lockfile has changed
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: [vitest-api]: Timeout calling "createTesters"
 ❯ Object.onTimeoutError node_modules/@vitest/browser/dist/index.js:3010:17
 ❯ Timeout._onTimeout node_modules/@vitest/browser/dist/index.js:2715:41
 ❯ listOnTimeout node:internal/timers:614:17
 ❯ process.processTimers node:internal/timers:549:7

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯


 Test Files   (232)
      Tests  53 passed (65)
     Errors  1 error
   Start at  16:04:53
   Duration  74.49s (transform 0ms, setup 1.57s, collect 2.46s, tests 0ms, environment 0ms, prepare 1.07s)

At first I though it was breaking the default reporter, but it's actually only running one test suite at a time.
Not sure if I need to configure anything differently, but this doesn't work well.

[nstepien]

Another run in a subset of our codebase:

npx vitest run src/camp

 RUN  v3.0.8 D:/repo

 ✓  chromium  src/....test.tsx (13 tests) 13214ms
 ✓  chromium  src/....test.tsx (6 tests) 8096ms
 ✓  chromium  src/....test.tsx (11 tests) 7908ms
 ✓  chromium  src/....test.tsx (3 tests) 2447ms
 ✓  chromium  src/....test.tsx (3 tests) 1469ms
                                                                                              
 Test Files  5 passed (5)                                                                     
      Tests  36 passed (36)                                                                   
   Start at  16:16:19                                                                         
   Duration  46.19s (transform 0ms, setup 2.45s, collect 1.50s, tests 33.13s, environment 0ms, prepare 528ms)      

it runs to completion, but it still only runs one test suite at a time.

[sheremet-va]

it runs to completion, but it still only runs one test suite at a time.

This was a mistake on my part. Last minute if broke the parallelisation because of isolate: false

At first I though it was breaking the default reporter, but it's actually only running one test suite at a time.

Does it print any tests running? It seems like there is a timeout for createTesters - there shouldn't be one

[nstepien]

Does it print any tests running?

It does, seems like the default reporter behaves as expected.

[nstepien]

The latest build works better, but I still get the Timeout calling "createTesters" error before I can confirm whether parallelization has been fixed.

[nstepien]

The latest builds work great now 👍

[sheremet-va]

The latest builds work great now 👍

Is it any faster than before?

[nstepien]

Absolutely, yes

[sheremet-va]

Absolutely, yes

Can you share the numbers?

[nstepien]

In our CI, before:

Duration  800.08s (transform 0ms, setup 679.36s, collect 877.16s, tests 1486.22s, environment 0ms, prepare 207.40s)

After, with maxWorkers: 8 because tests somehow become flaky with increased parallelization, we need to fix our tests:

Duration  360.87s (transform 0ms, setup 364.73s, collect 483.01s, tests 1519.42s, environment 0ms, prepare 64.00s)

Once we fix our flaky tests it'll be faster still.

[sheremet-va]

With this change, firefox now fails consistently on Windows 😞 I am also getting weird time outs in different tests

I think we should wait for #7377 to be fixed first before releasing this

[mleimer]

I understand that this fix shall run multiple test files in parallel in browser-mode. However, I am unable to achieve that in a non-headless mode, i.e. only one test file is running at a time.

Repository
https://github.com/mleimer/vitest-browser-parallelism

Do I misunderstand anything or what am I missing out?

When setting browser.headless: true, the test files indeed run in parallel. Is there anyway to achieve the same without using headless?

[sheremet-va]

I understand that this fix shall run multiple test files in parallel in browser-mode. However, I am unable to achieve that in a non-headless mode, i.e. only one test file is running at a time.

Repository https://github.com/mleimer/vitest-browser-parallelism

Do I misunderstand anything or what am I missing out?

When setting browser.headless: true, the test files indeed run in parallel. Is there anyway to achieve the same without using headless?

Tests run in parallel only in headless mode.