Contributor Experience Improvements

[perdasilva]

Epic Description

It's been a bit hard to get some PRs through lately. The main culprit seems to be the flaky tests although some other areas of improvement have also been identified. This epic collects the different improvements we'd like to target with the goal of delivering a better experience to our contributors: the information they get back from the PR checks is accurate and actionable, reduce PR lead time, reduce waste (retesting the whole suite, when one test fails), improve communication back to the contributor.

Here are the current suggestions for sub-stories. Once we agree on them, we can turn these into issues.

• Measure contributor experience #2588
• Remove flakes from the critical path #2589
• Define a process for eliminating flakes #2590
• chore(e2e): naively parallelize CI jobs by chunking alphabetically #2520
• Investigate ways to reduce the need to rebase after each merge #2591
• Add notifications when unit/e2e-tests fail #2592
• Clean up makefile #2598
• Improve developer documentation #2597
• Provide docker container containing whole environment #2596
• Make build work with podman #2594
• Improve e2e debugging experience #2593

[timflannagan]

Referencing #2520 which is another iteration for speeding up the e2e suite's execution by parallelizing test case chunks into their own cluster. Previous work on this topic that we didn't pursue further is #1476.

[timflannagan]

@perdasilva Should we track utilizing #2527 more throughout the testing suite as well? Or try migrating any grpc-based CatalogSources that reference remote container images, to housing that index image as FBC in the repository? Whoops, I was commenting on the wrong issue.

[timflannagan]

Another thing we can pursue to improve contributing experience is through improving the debugging experience of failed e2e runs such that an individual test case failure triggers the collection of OLM-related resources in a namespace (or collecting logs of the catalog/olm operators). #2519 is one instance that can be used to collect test fixtures, container logs, etc. We can also continue to utilize the SetupGeneratedTestNamespace and TeardownNamespace which help with this workflow.

[fgiloux]

Additional suggestions:

• improving the developer documentation.
• cleaning Makefile
• Making build work with podman

some background here: #2504 (comment)

[akihikokuroda]

My 2 cents,

Improve the developer documentation

• pre-req tools
• for using MAC
• build, unit test and e2e test steps

Provide the container image for the build and unit test so that the developers don't need to install any tools on their system except the container runtime environment.

[perdasilva]

Hey @akihikokuroda @fgiloux @timflannagan thank you for your suggestions. I've tried to capture them in the stories/tasks above. Could I ask you guys to help me fill out the tickets:

@akihikokuroda #2596 #2587
@timflannagan #2593
@fgiloux #2594 #2598

Could I also please ask you to follow the format, e.g. #2588, i.e. description + acceptance criteria (try to keep it tightly scoped) + suggested implementation (optional)

If you have time to review the ones I've created, I'd also appreciate some feedback