[Feature] implement `await expect(page).toHaveScreenshot()`

[aslushnikov]

introduce a web-first assertion for Visual regression testing.

Today:

expect(await page.screenshot()).toMatchSnapshot('foo.png');

After this:

await expect(page).toHaveScreenshot('foo.png');

toHaveScreenshot should auto-retry screenshot until golden is reached.

toHaveScreenshot should also have:

• hide field cursor
• support locators
• disalbe css animations — [Feature] disable CSS animations when making screenshots  #11912
• mask out certain parts of screen? e.g. with selector — [Feature] mask certain elements while making screenshots #11913

[aslushnikov]

A summary of the VC with Dmitry regarding toHaveScreenshot() retries.

Main thought

We would love toHaveScreenshot() to do retires and wait for the page to have the screenshot that's saved.

Challenge: how to generate initial screenshots?

• Manually: it would be handy to have a 📸 button in codegen. When I see with my eyes that things are nice on the page, I take a snapshot, and it's saved to the right place!
• Automatically: auto-update 1000s of screenshots when website changes. How to wait for proper signal?
  • good idea: fighting progress animations: wait until screenshots stop changing
  • controversial idea: wait for network idle
  • not-so-good idea: wait for 1 additional second?!

---

Convenient code pointer for implementing retries:

playwright/src/test/matchers/toMatchText.ts

Lines 65 to 77 in d0a4480

	await pollUntilDeadline(testInfo, async remainingTime => {
	received = await query(remainingTime);
	if (options.normalizeWhiteSpace && isString(expected))
	received = normalizeWhiteSpace(received);
	if (options.matchSubstring)
	pass = received.includes(expected as string);
	else if (typeof expected === 'string')
	pass = received === expected;
	else
	pass = expected.test(received);
	return pass === !matcherOptions.isNot;
	}, options.timeout, testInfo._testFinished);

[p01]

The method name, toHaveScreenshot(path), is pretty confusing. What does is it mean for a page to have a screenshot ?
The previous syntax is clear:

expect(await page.screenshot()).toMatchSnapshot('foo.png');` 

If you want to sprinkle some sugar on top, make the .screenshot() call implicit using this syntax:

await expect(page).toMatchSnapshot('foo.png');

Where the toMatchSnapshot(path) method would do all the heavy lifting of waiting for the page to be stable, taking the screenshot and matching it against the snapshot.

As for how to automatically wait to take the screenshots, what I have done with great result is to force CSS animation and transition duration to a 0.001ms, wait for networkIdle, and check that there are no pending Promises and setTimeout or setInterval ( triggered less than 2-3 seconds in the future, as anything longer is usually related to telemetry push ).

[aslushnikov]

If you want to sprinkle some sugar on top, make the .screenshot() call implicit using this syntax:

@p01 changing toMatchSnapshot semantics would be a breaking change for the existing clients that we cannot afford.
So it'll stay unchanged for some time as a legacy solution.

The method name, toHaveScreenshot(path), is pretty confusing.

We're open to suggestions!