[WIP] 🌱 cleanup/e2e-fixtures

[perdasilva]

Description of the change:
Our e2e tests cover the case of serving a catalog whose content was extracted (i.e. not using the bundled opm + being able to ship just content). The problem is that we were using the official operatorhub io image. This image is not versioned (it only has the latest tag). Because this registry's cache is built with a newer version of opm, it is breaking tests in older versions of the product.

In this PR:

• Refactors the never used communityCatalogImage (or something like that) into a opmVersion and refactors the tests to reflect the changes
• Refactors the e2e job to detect changes to either the operator-registry dependency version of changes to the fixture files and rebuild the fixtures and kind load them for the e2e tests
• OLM is tested with the newly built fixture images
• If on master and e2e succeeds, the new fixture images are published by a new "fixture" GHA
• Updates tests to use the opm version compatible fixture
• I also had to update the catalog operator to treat test images (images coming from the quay.io/olmtest) as a digest image (basically PullIfNotPresent) - this is needed to avoid having to push temporary images but rather just kind load them instead

The new e2e flow is:

build:

• rebuild fixture images if necessary
• build controller image
• upload artifacts

e2e:

• download artifacts
• load images
• deploy test clusters with images (kind load docker-image)
• run e2e

publish-e2e-fixtures:

• only executes if on master and there have been changes
• pushes new fixtures up to the olmtest registry

[m1kola]

Trying to understand this, so posted a bunch of questions.

[m1kola]

/lgtm

[openshift-ci]

New changes are detected. LGTM label has been removed.

[m1kola]

Overall all looks good to me, but this made me wondering - can we do something so we are not dependent on quay.io for tests? Maybe something like we do with OLMv1 where we have a local registry? Just to clarify: not asking to do this as part of this PR.

[m1kola]

I'm not a big fan of having a special cases for tests. Is there a way to avoid this?

[perdasilva]

Same - the only other way I could think of is by threading another command line arg down the stack - idwtdt

I figured this isn't so bad, since the pattern should only ever match our registry. So, there shouldn't be any hidden surprises.

But, yeah, I don't like it either. But anything else that I could think of felt like too many changes T_T

🤔 let me see if I can programmatically grab the sha from the image:tag - I think we already have a dependency on the docker client anyway

[perdasilva]

Overall all looks good to me, but this made me wondering - can we do something so we are not dependent on quay.io for tests? Maybe something like we do with OLMv1 where we have a local registry? Just to clarify: not asking to do this as part of this PR.

Yeah, that'd be heaps better! The current approach relies on kind loading. Maybe the right way is to just as v1. Get rid of quay.io/olmtest and just rebuild against a local registry for the tests. I've created an issue to track this: #3315

[perdasilva]

No luck. Doesn't seem to find the image on the node by its sha - so kind load didn't work. Maybe we'll have to look at a local registry...=S

[perdasilva]

/hold exploring registry option

[openshift-merge-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.