[BUG] test.use of storage state regression in 1.24

[daddyman]

Context:

• Playwright Version: 1.24.1
• Operating System: Windows and Linux
• Node.js version: 16.16.0
• Browser: Chromium
• Extra:

System:

• OS: Windows 10 10.0.22000
• Memory: 15.67 GB / 31.64 GB

Binaries:

• Node: 16.16.0 - ~\AppData\Local\nvs\node\16.16.0\x64\node.EXE
• Yarn: 1.22.19 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
• npm: 8.11.0 - ~\AppData\Local\nvs\node\16.16.0\x64\npm.CMD

Languages:

• Bash: 5.1.16 - C:\Windows\system32\bash.EXE

Code Snippet

There is a test app to reproduce the problem here: https://github.com/daddyman/playwrightstate

Describe the bug

We have tests where there is an top-level test.describe and the page in the tests use the storage state to setup a test. This is an inner test.describe to run some tests without the storage state using test.use({ storageState: undefined });. The tests inside this test.describe should not use the storage state.

In Playwright 1.23 and older when we use a test.use({ storageState: undefined }); in the inner test.describe the tests under that describe to not use the storage state setup that was setup globally - as expected.

In Playwright 1.24 the behavior has changed and the tests inside the inner test.describe are using the global storage state which is causing some of our tests to fail.

Output from Playwright 1.23.4 (good)

The outer tests have the storage state and the inner tests do not. This is expected.

Running 4 tests using 1 worker

  ✓  [chromium] › example.spec.ts:15:3 › parent test › test 1 with state (3s)
test 1 {"myname1":"myvalue1"}
  ✓  [chromium] › example.spec.ts:33:5 › parent test › inner test › inner test 1 (2s)
inner test 1 {}
  ✓  [chromium] › example.spec.ts:38:5 › parent test › inner test › inner test 2 (105ms)
inner test 2 {}
  ✓  [chromium] › example.spec.ts:44:3 › parent test › test 2 with state (114ms)
test 2 {"myname1":"myvalue1"}

Output from Playwright 1.24.1 (bad)

The outer tests have the storage state and the inner tests also have the storage state. This is a regression from 1.23.

Running 4 tests using 1 worker

  ✓  [chromium] › example.spec.ts:15:3 › parent test › test 1 with state (660ms)
test 1 {"myname1":"myvalue1"}
  ✓  [chromium] › example.spec.ts:33:5 › parent test › inner test › inner test 1 (614ms)
inner test 1 {"myname1":"myvalue1"}
  ✓  [chromium] › example.spec.ts:38:5 › parent test › inner test › inner test 2 (95ms)
inner test 2 {"myname1":"myvalue1"}
  ✓  [chromium] › example.spec.ts:44:3 › parent test › test 2 with state (110ms)
test 2 {"myname1":"myvalue1"}

[miguelofbc]

@daddyman @dgozman @pavelfeldman I am suspecting that this regression is also in 1.25? At least I am seeing the same failure for 1.25.2.
When I am using an empty json file as follows, the test is passing and working as expected.
test.use({ storageState: "emptyStorageState.json" });
If I pass undefined as follows, it fails and the storageState.json is being used.
test.use({ storageState: undefined });

[daddyman]

@miguelofbc Yes it looks like it regressed again. I changed my code to do the following instead of using undefined.

test.use({ storageState: { cookies: [], origins: [] } });