Skip to content

Conversation

@castastrophe
Copy link
Contributor

@castastrophe castastrophe commented Jun 11, 2021

Visual regression test reporting for pull requests

This PR adds a json output to the wdio config which can be used by the Test reporter action to the pull request in a summarized view that is easier for users to parse for meaning.

Related issues

  • #1350 Visual regression testing challenges

Testing instructions

  • Validate pull request opened, in draft state, no labels
    1. Open a new pull request against chore-report-results with a small, non-breaking change
    2. All tests should be skipped
  • Validate pull request opened, not draft, no labels | passing tests
    1. Use the previous PR, change state to ready for review
    2. Should skip e2e tests, no results reported
  • Validate pull request opened, not draft, label run e2e | passing tests
    1. Use the previous PR but add the run e2e label
    2. Should run e2e tests, report back results, pass the new action step
  • Validate pull request opened, not draft, no labels | failing tests
    1. In the previous PR, delete a chunk of one of the demo pages to cause a failing test for e2e
    2. Should run e2e tests, report back failure, fail the new action step
  • Validate pull request opened, not draft, label skip vrt
    1. Use the previous PR but add the skip vrt label
    2. Should skip e2e tests, no results reported

Ready-for-merge Checklist

  • Expected files: all files in this pull request are related to one request or issue (no stragglers or scope-creep).
  • Documentation (README.md, WHY.md, etc.) updated or added.

Merging

Please squash when merging and ensure your commit message uses conventional commit formatting.

Be sure to share your updates with the patternfly-elements-contribute@redhat.com mailing list!

Add json reporting
@github-actions github-actions bot added the work in progress POC / Not ready for review label Jun 11, 2021
@castastrophe castastrophe changed the title Update wdio.conf.js chore: Update result reporting Jun 11, 2021
@github-actions github-actions bot added the tools Development and build tools label Jun 11, 2021
@github-actions github-actions bot removed the tools Development and build tools label Jun 11, 2021
@github-actions github-actions bot added the tools Development and build tools label Jun 11, 2021
@castastrophe castastrophe linked an issue Jun 11, 2021 that may be closed by this pull request
@castastrophe castastrophe marked this pull request as ready for review June 11, 2021 15:37
@castastrophe castastrophe added priority: low Severity level: 3 run e2e Trigger automated visual regression tests and removed work in progress POC / Not ready for review labels Jun 11, 2021
@castastrophe castastrophe marked this pull request as draft June 16, 2021 15:06
@castastrophe castastrophe linked an issue Jun 21, 2021 that may be closed by this pull request
@castastrophe castastrophe added the work in progress POC / Not ready for review label Jul 19, 2021
@bennypowers bennypowers deleted the chore-report-results branch June 25, 2024 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: low Severity level: 3 run e2e Trigger automated visual regression tests tools Development and build tools work in progress POC / Not ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Visual regression testing challenges [ci] Visual regression result reporting

2 participants