build RHCOS as an OCI image

[cheesesashimi]

This PR enables us to build RHCOS as a container image and take advantage of OpenShift CI's container-based workflow.

There are two key things which make this possible:

1. The OpenShift Image Builder allows one to place a file from a previously built container into the build context for another container using inputs.
2. Buildah supports multiple image transports, including oci-archive which allows one to write a Containerfile which points to an OCI archive on disk, e.g., FROM oci-archive:/path/to/oci/archive.

How this works is:

1. COSA (coreos-assembler) runs within a container build context and creates an OCI archive, which is baked into the built image (cosa-build). Naturally, this OCI archive baked into another image is not very useful.
2. The OCI archive is extracted from the cosa-build image and placed someplace where we can easily import it (cosa-oci-archive)
3. We use the cosa-oci-archive image as an input to machine-os-oci-context. This reads the OCI image from the build context and uses it to build a new image. Since all we do is FROM oci-archive:/path/to/oci/archive, this performs a very basic image build and pushes the final image to the ephemeral CI ImageStream for further consumption.
4. The tests make use of the cosa-build image because it has the full COSA build context contained within it. However, we can now shard the tests, which allows the suite to run concurrently and offers additional control over which tests should be executed / retried / etc.

For now, the periodic tests are duplicates of the PR builds and is configured to push images to the rhcos-devel namespace, whereas PR builds will not produce an artifact. Additionally, it was discovered that we were unintentionally mirroring the build-test-qemu-img via the core-services/image-mirroring/openshift/mapping_origin_4_11 file. A future TODO is to enable the PR builds to layer OS content on top of the nightly-built images which should dramatically reduce the total build / test time for PRs. This would likely require additional work within COSA to facilitate.

A more complete braindump of how this works as well as caveats / pitfalls encountered along the way is in progress here: openshift/os#780

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[sohankunkerkar]

/retest

[cheesesashimi]

/assign cgwalters

[cgwalters]

I think since we know we have passwordless sudo enabled int the cosa container this could also just be a && sudo chgrp -Rf root ... or so attached to the previous RUN invocation?

But good to know we're squashing anyways.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[cheesesashimi]

/test build05-dry

[cgwalters]

/approve

[cheesesashimi]

/assign @bparees

Context for approval: While setting up the PR builds, I was somewhat surprised to see that we were promoting and mirroring the build-test-qemu-img. This is an intermediate image used for this build and not very useful outside of that context, so I'm not sure why it was being mirrored to Quay, especially since the config used the .promotion.excluded_images[0] = '*' convention. For the time-being, I've explicitly disabled image promotion for PR builds (.promotion.disabled = true), which removed the entry from the core-services/image-mirroring/openshift/mapping_origin_4_11 file, hence the need for approval. I'm happy to answer any further questions!

[bparees]

approving removal of the mirroring config for this image, thanks for the explanation!

/approve

[openshift-ci]

@cheesesashimi: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/os/master/validate-image-build	2e70ac270b8ecadfb121046ceb3c6747fca9d4b7	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/test-qemu	66d1b4e8434224faca999724ca56392831f16574	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/validate-user-id	b217755e24f62857a84e6bace9c7d1f64e0f6c41	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/test-qemu-kola-upgrade	1b85152c634b5da2dd82f2edb536cac30d748c62	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/test-qemu-nvme	1b85152c634b5da2dd82f2edb536cac30d748c62	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/test-qemu-kola	1b85152c634b5da2dd82f2edb536cac30d748c62	link	unknown	/test pj-rehearse
ci/rehearse/periodic-ci-openshift-os-master-periodic-test-qemu-kola-upgrade	83f07d2	link	unknown	/test pj-rehearse
ci/prow/pj-rehearse	83f07d2	link	false	/test pj-rehearse
ci/rehearse/periodic-ci-openshift-os-master-periodic-test-qemu-metal	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/validate	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/periodic-ci-openshift-os-master-periodic-test-qemu-firmware-uefi	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/periodic-images	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/periodic-ci-openshift-os-master-periodic-test-qemu-nvme	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/periodic-ci-openshift-os-master-periodic-test-in-cluster	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/images	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/periodic-ci-openshift-os-master-periodic-test-qemu-kola	83f07d2	link	unknown	/test pj-rehearse
ci/rehearse/openshift/os/master/test-in-cluster	83f07d2	link	unknown	/test pj-rehearse

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[miabbott]

@cheesesashimi Thank you for doing this and wiring it all together! The comments are immensely useful for understanding how this is pieced together. Let's see it in action!

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, cgwalters, cheesesashimi, miabbott

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/os/OWNERS [bparees,cgwalters,miabbott]
• ci-operator/jobs/openshift/os/OWNERS [bparees,cgwalters,miabbott]
• core-services/image-mirroring/openshift/OWNERS [bparees]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@cheesesashimi: Updated the following 5 configmaps:

• job-config-master-postsubmits configmap in namespace ci at cluster app.ci using the following files:
  • key openshift-os-master-postsubmits.yaml using file ci-operator/jobs/openshift/os/openshift-os-master-postsubmits.yaml
• job-config-master-presubmits configmap in namespace ci at cluster app.ci using the following files:
  • key openshift-os-master-presubmits.yaml using file ci-operator/jobs/openshift/os/openshift-os-master-presubmits.yaml
• image-mirror-mappings configmap in namespace ci at cluster app.ci using the following files:
  • key mapping_origin_4_11 using file core-services/image-mirroring/openshift/mapping_origin_4_11
• ci-operator-master-configs configmap in namespace ci at cluster app.ci using the following files:
  • key openshift-os-master.yaml using file ci-operator/config/openshift/os/openshift-os-master.yaml
  • key openshift-os-master__periodic.yaml using file ci-operator/config/openshift/os/openshift-os-master__periodic.yaml
• job-config-master-periodics configmap in namespace ci at cluster app.ci using the following files:
  • key openshift-os-master-periodics.yaml using file ci-operator/jobs/openshift/os/openshift-os-master-periodics.yaml

Details

In response to this:

This PR enables us to build RHCOS as a container image and take advantage of OpenShift CI's container-based workflow.

There are two key things which make this possible:

1. The OpenShift Image Builder allows one to place a file from a previously built container into the build context for another container using inputs.
2. Buildah supports multiple image transports, including oci-archive which allows one to write a Containerfile which points to an OCI archive on disk, e.g., FROM oci-archive:/path/to/oci/archive.

How this works is:

1. COSA (coreos-assembler) runs within a container build context and creates an OCI archive, which is baked into the built image (cosa-build). Naturally, this OCI archive baked into another image is not very useful.
2. The OCI archive is extracted from the cosa-build image and placed someplace where we can easily import it (cosa-oci-archive)
3. We use the cosa-oci-archive image as an input to machine-os-oci-context. This reads the OCI image from the build context and uses it to build a new image. Since all we do is FROM oci-archive:/path/to/oci/archive, this performs a very basic image build and pushes the final image to the ephemeral CI ImageStream for further consumption.
4. The tests make use of the cosa-build image because it has the full COSA build context contained within it. However, we can now shard the tests, which allows the suite to run concurrently and offers additional control over which tests should be executed / retried / etc.

For now, the periodic tests are duplicates of the PR builds and is configured to push images to the rhcos-devel namespace, whereas PR builds will not produce an artifact. Additionally, it was discovered that we were unintentionally mirroring the build-test-qemu-img via the core-services/image-mirroring/openshift/mapping_origin_4_11 file. A future TODO is to enable the PR builds to layer OS content on top of the nightly-built images which should dramatically reduce the total build / test time for PRs. This would likely require additional work within COSA to facilitate.

A more complete braindump of how this works as well as caveats / pitfalls encountered along the way is in progress here: openshift/os#780

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.