[HOLD for payment 2024-11-07] [HOLD for payment 2024-11-05] E2E: Don't fail whole e2e test suite when one test fails

[hannojg]

Currently in the e2e testing pipeline if one test fails the whole test suite is failed. Instead the test suite should always complete, and failed test cases should be reported individually.

This has been discussed in here (in the #e2e-announce channel).

Notes:

• for each completely failed test case we should send a individual slack notification
• failed tests should be marked in the report as failed
• if a test failed in the test suite we should comment the report to the PR against which the test suite was executed, noting that a certain test is broken
  • If a test is broken on main and merged PRs after that would have this test as failed. we should add a note in the github comment that its possible that the test has been broken in a previous PR and that the author can check the commits on main to see if this PR's commit was the first failed one. This can help avoid confusion for PR authors

Issue Owner

Current Issue Owner: @kirillzyusko

[hannojg]

cc @mountiny and @kirillzyusko and @chrispader (i remember one of the two was available to work on this ticket)

[melvin-bot]

Triggered auto assignment to @MitchExpensify (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details. Please add this Feature request to a GH project, as outlined in the SO.

[melvin-bot]

⚠️ It looks like this issue is labelled as a New Feature but not tied to any GitHub Project. Keep in mind that all new features should be tied to GitHub Projects in order to properly track external CAP software time ⚠️

[mountiny]

Thanks! Please comment on this issue whoever will work on this

[melvin-bot]

Triggered auto assignment to Design team member for new feature review - @dubielzyk-expensify (NewFeature)

[dubielzyk-expensify]

Assuming you don't need design help here?

[kirillzyusko]

I can handle this @mountiny 🙌