[Feature] Trace viewer - more highlights, show assertions

[A-ZC-Lau]

TLDR

1. There should be more highlights on more actions, i.e. getAttribute
2. Trace viewer should show assertions

It would be nice to have some visual display when getting attributes and making assertions, similarly to Cypress. Currently, the highlighting is limited to certain actions, such as clicking, and there is no information on any assertions (passed or failed).

Highlighting

1. Lack of highlighting example

Does highlight

page.locator("button").click()

Does not highlight

page.locator("button").getAttribute("title")

Assertions

Would be nice to assertions as part of the actions tab or just somewhere to have a better understanding of what and where it exactly failed

Cypress

With Cypress's trace viewer, shows what element was located, and then what assertion was also made on it

e.g. cy.get("#username").invoke("attr", "placeholder").should("equal", "please enter your username")

[pavelfeldman]

Is that for your local workflow or for CI/CD? We have all these details about the failure in the test report, you don't even need to open trace for that. But maybe you have a different workflow in mind, could you share more?

[A-ZC-Lau]

@pavelfeldman

This is more for local testing, where we are just beginning our testing and may be getting some targeting or something else wrong.

Take for example, cypress

Cypress

it("should find the right a", () => {
	cy.visit("https://getbootstrap.com/");

	cy.get("a").first().should("have.text", "Download");
})

when you run the above code, I know exactly why my test failed, because it was getting some element I wasn't expecting (should be getting the second element)

Playwright

However, with similar code in playwright, I am not certain why it is failing. (code uses @playwright/test with typescript

import { test } from "@playwright/test";

test("should find the right a", async ({ page, context }) => {
	await context.tracing.start({
		name: "test",
		screenshots: true,
		snapshots: true,
	})

	await page.goto("https://getbootstrap.com/");
	
	const text = await page.locator("a").first().innerText();

	await context.tracing.stop({
		path: "trace.zip"
	});
	
	test.expect(text).toBe("Download");
})

On a somewhat related note, if test.expect was placed BEFORE context.tracing.stop, and the assertion fails, no trace is actually created.

[pavelfeldman]

You would typically use trace viewer for post-mortem where your tests ran remotely and you need trace as a self-contained recording of your test.

---

Here is how you would typically enable tracing for your tests:
https://playwright.dev/docs/trace-viewer#recording-a-trace (use TypeScript tab).

---

Here is what I get w/o Trace viewer:

---

Here is how you could use locator to have full auto-wait and auto-retry:

import { test, expect } from "@playwright/test";

test("should find the right a", async ({ page }) => {
	await page.goto("https://getbootstrap.com/");
	await expect(page.locator("a").first()).toHaveText("Download");
})

In that case, Playwright will be retrying for expect timeout before it gives up, 5 sec. Similarly to how Cypress does. It will the give the same result.

So we seem to be giving similar amount of information w/o trace viewer, or would you like to only use trace, if that is the case, why? We typically use Playwright Inspector while debugging tests.

[A-ZC-Lau]

You would typically use trace viewer for post-mortem where your tests ran remotely and you need trace as a self-contained recording of your test.

Yes, but a report on something that failed is different to knowing why it got those details in the first place.

With the cypress example, you can see why the locator got text that it got, because it was retrieving a hidden/off screen "a" element through the highlight. As opposed to the intended element with the "Download" text.

But if you run it with playwright, you have no idea of what element it's getting, because the trace doesn't highlight any elements that you're acting upon, i.e. .innerText()

[pavelfeldman]

Got it, I think I understand what you mean. We are working on --debug that would do pretty much PWDEBUG=1 npx playwright test, but assertion-oriented. Trace viewer will also get assertions at some point, but we keep it for the post-mortem story mostly.

[A-ZC-Lau]

Got it, I think I understand what you mean. We are working on --debug that would do pretty much PWDEBUG=1 npx playwright test, but assertion-oriented. Trace viewer will also get assertions at some point, but we keep it for the post-mortem story mostly.

Sounds great. I think at minimum, showing highlights in the trace when a locator performs any sort of action, e.g..innerText() would be a great minor release, since that functionality already exists for .click().

[pavelfeldman]

Some work has been done here, please report on what is missing after 1.16!