[BUG]:[Tests running on Microsoft Playwright service are inconsistent and failing]

[RF2023]

Describe the bug
Tests running on Microsoft Playwright service are inconsistent. Same tests have better pass rate via localhost and our non prod env vs running on playwright service.

To Reproduce
Steps to reproduce the behavior:
N/A

Expected behavior
Less errors and pass rate should match our local env runs with less time

Screenshots

Setup information (please complete the following information):

• Azure Subscription Id:798f112d-0c47-431d-9b57-eeae586eb023
• Workspace Name: Playwright
• Service trace Id (if available):
• Log snippet (if available):
• Browser on which tests are executed (if applicable): chromium
• Time range (in PST) when the issue was experienced: Multiple times
• Playwright Version: 1.42.1
• Client machine OS: MAC
• OS requested on Service: linux

Additional context
We have implemented playwright service for few tests, especially when a Pull request is open it runs on 10 workers for 25 tests and once the Pull request is merged it runs on 10 workers for 40 tests and the pass rate is not 100% and throws errors as shown above. We have observed some discrepancies on the results running on the service vs the tests without the service and only on 4 workers (100% pass rate). Couple of tests we have identified that backend was sending data but it was not able to read the data being sent and the UI verification fails.

We tried running 25 tests across 10 workers across 5 instances in parallel and only 3 of them passed with 40-0 and the other two never finished.

Another thing to note is we have few tests which need to run in serial mode, due to the nature of file upload and other scenarios and one test fails in that all of them get skipped.

Call log:
  - <ws connecting> wss://eastus.api.playwright.microsoft.com/api/authorize/connectSession
  - <ws error> wss://eastus.api.playwright.microsoft.com/api/authorize/connectSession error WebSocket was closed before the connection was established
  - <ws connect error> wss://eastus.api.playwright.microsoft.com/api/authorize/connectSession WebSocket was closed before the connection was established
  - <ws disconnected> wss://eastus.api.playwright.microsoft.com/api/authorize/connectSession code=1006 reason=
  - <ws connecting> wss://browser.playwright.microsoft.com/ws

AB#2017175

[puagarwa]

Hi @RF2023
thanks for reaching out to us and explaining your issue in detail. I looked into your issue in our telemetry and i found that the failure and latency issue you are facing is due to browser quota getting exhausted for your account.

We provide 50 parallel worker as default quota which means at any point of time your 50 request would immediately get browser but beyond that they will get queued at our side until your timeout happen or any existing browser closes.

I could see multiple instances in last few days where your account has gone beyond the quota limit and hence either request got delayed or they got timedout and failed.

Two such instances
At: 01 April 11:20 GMT
109 request -> out of those 50 request ran success, 23 failed

At: 01 April 11:33 IST
78 request -> 55 ran success, 13 failed

There are two things we can do about it

• Raise request for quota increase via Github ticket
• Increase your clientTimeout to high value say 2-3min (this would save it from failing in case if it still hit quota)

The probable reason for your quota going beyond 50 is you have multiple jobs running in parallel. Even though individually your job uses only 10 worker but it get multiplied with number of jobs in parallel.

[RF2023]

Thanks for responding back to us, so if i understood correctly and lets take this example. When we open a PR we have 25 tests that will run on playwright service on 10 workers. So the consumption is only 10 correct?
Similarly when we merge the PR its 10 workers against 40 tests . The data you shared with us is true for those instances because we did 5 parallel runs and that was total of 50 workers for 125 tests. Did i understand that correctly?

[RF2023]

Timeout is set at default 30000

[puagarwa]

HI @RF2023
Yes, PR with 10 worker would consume 10 parallel workers.
Yes, build with 5*10 worker would consume 50 workers

I think whats probably happening here is sometime your multiple worklow might have ran in parallel either multiple PR build or multiple post PR test workflow. which would increase your total parallel browser count to more than 50.

we could increase it to 2-3mins for the case when your workflow consume more than quota and browser latency would increase.

[RF2023]

so the 50 workers was a test, we can ignore that we want to go in depth for the one with 10 workers with 30 tests and 40 tests and why there is a discrepancy with our local runs.

[puagarwa]

Sure, can you please do a fresh run. please make sure you dont run other workspace/jobs in parallel to avoid quota issue and share us with time stamp and result from your side.

[RF2023]

sounds good.

[RF2023]

Triggered one just now.

[RF2023]

8541740441-2-b4bed07479969177b44f6cf6b5f7227c1d19e986
The test run history doesnt allow us to expand a particular run, is there any other place we can see the logs or results?

[puagarwa]

8541740441-2-b4bed07479969177b44f6cf6b5f7227c1d19e986 The test run history doesnt allow us to expand a particular run, is there any other place we can see the logs or results?

The test run history is not clickable as it only contain few metadata information around run duration and max worker. You would already have all logs at your client side itself. We dont store any logs at server side.

[puagarwa]

HI @RF2023
I checked our telemetry and i can confirm that in last 4 runs, i dont see any failures as load is within quota. Let me know if you see any issue at your side.

[miguelofbc]

Hello @puagarwa !

Thank you for your previous assistance. We're following up on our discussion about worker quotas.

We recently ran a test suite with the following configuration:

• 5 runs in parallel
• 30 tests per run
• 10 workers per run (total of 50 workers)

We ensured nothing else was running concurrently. Based on our understanding, this setup would require a quota of 50 workers, is that correct?

For reference, here are the run IDs:

1. 8550393480-1-32a7cf53994942134c02b0dfa95005756dc13110
2. 8550393116-1-32a7cf53994942134c02b0dfa95005756dc13110
3. 8550394120-1-32a7cf53994942134c02b0dfa95005756dc13110
4. 8550393820-1-32a7cf53994942134c02b0dfa95005756dc13110
5. 8550394373-1-32a7cf53994942134c02b0dfa95005756dc13110

However, we encountered failures in two of the runs:

1. Error: locator.fill: Target page, context or browser has been closed (SessionID: 3f3601c0-6aa9-4ccd-aa57-b8f3afc07ebd)
2. Error: locator.fill: Test timeout of 180000ms exceeded (SessionID: 9110dfc8-f36a-4532-9344-d0513a2daef9)

Interestingly, when we run the same tests with this configuration on our own infrastructure, everything works as expected. This suggests the tests themselves might not be flaky.

Would you be able to share your insights on these failures and confirm the required worker quota for our scenario?

[puagarwa]

Hi @miguelofbc ,
First thing to confirm that the above failures are not due to quota issue, as you stated you were well under quota limit hence you didnt face any issue during this run.

Coming to the above two failures, both of these happened during the test steps.

Error: locator.fill: Test timeout of 180000ms exceeded (SessionID: 9110dfc8-f36a-4532-9344-d0513a2daef9)

This one seems like test timeout, please check trace or video to understand why this happened

Error: locator.fill: Target page, context or browser has been closed (SessionID: 3f3601c0-6aa9-4ccd-aa57-b8f3afc07ebd)

This one does seems like browser crash at our end, it should be one off, let me know if you see it again.

[puagarwa]

Also, one thing you mentioned above around local vs service testing. You mentioned during local testing, you use local endpoint whereas with service you are using public endpoint, it would be better to compare local with public endpoint.