providing more useful metadata about what is in the machine-os-content image

[miabbott]

This conversation started as a way to provide useful metadata about the available/supported extensions that are shipped alongside the OS update in the machine-os-content. But it grew into providing more information like kernel version or glibc version.

We will likely pursue the idea of encoding the kernel version as an annotation to the machine-os-content, but let's continue to think about other useful metadata to machine-os-content consumers/users.

Pulling out some conversation from openshift/machine-config-operator#1850

@miabbott says:

Maybe we should be including a JSON file or the like in machine-os-content that provides the list of supported extensions? Since we are responsible for sticking the right RPMs and deps into the container, seems like we are in the best position to enumerate the supported extensions per oscontainer/release.

Could even go as far as providing a supported and unsupported list, if we were to ever decide to drop support for an extension.

@sinnykumari replies:

agree, if we want to extend this to supported/unsupported list then machine-os-content seems to be the right place to provide that detail.

For now I think it should be fine for MCO to consume extensions as it is because we ship only those extra packages in machine-os-content which we want users to install. #1766 makes things really flexible to update MCD behavior whenever neededd

@cgwalters replies:

The new v2 layout https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/952 aims to have "enumerate extensions" simply use the filesystem. Basically:

List all sub-directories of extensions/
Skip dependencies/
If the directory has any .rpm files in it, it's an extension

Not opposed to a JSON list, but keep in mind the rpm-md repodata is already a "list of packages" (just in XML form). We'd be inventing "rpm metadata in JSON form"?

@jlebon replies:

I think this is fine to start, though IMO there's definitely an argument for having the flexibility that some metadata layer affords us. For example, if we want extension "foobar" to actually result in the installation of two related, but not necessarily dependent packages. Or if we want to change how extension "foobar" is implemented at the RPM level across an update.

I think actually this might help with the Recommends issue as well. If we solve that and turn it off by default both server-side and client-side, then we'll need a way to pull in recommended packages which no longer get pulled in when it makes sense (and without globally turning on weak deps again client-side).

Anything we change inside the container though isn't solving the issue that validating a MachineConfig fragment requires fetching and unpacking the container.

@miabbott revives the idea:

In a real-time meeting with @zvonkok @mrunalp @darkmuggle (re: openshift/enhancements#357) we discussed adding additional annotations to the machine-os-content about the kernel version (and possibly the glibc version) used by RHCOS. This would help the users doing out-of-tree module builds by providing the information in the metadata, rather than having to pull down the entire container image.

It was brought up that we could go a step further and also enumerate the available extensions in an annotation on the machine-os-content. This could provide similar benefits to allowing users (or tooling) to inspect the metadata of machine-os-content to determine which extensions are supported/available in a particular image.

[jlebon]

Image labels sound fine, though worth mentioning that since we already do pull down the oscontainer during pivot/update, we could also just copy any embedded metadata over from the container while it's mounted.

Heck, we could even just copy all the RPMs onto the host because relatively speaking they're small. And it means we won't have to fetch the oscontainer at all even when we do need to install extensions.

[cgwalters]

Heck, we could even just copy all the RPMs onto the host because relatively speaking they're small. And it means we won't have to fetch the oscontainer at all even when we do need to install extensions.

Yeah, I think @ericavonb wanted to do that. I'm not opposed but it feels like it's going to scale badly if more extensions get created. I'd rather go in the "run a service in the cluster" path being proposed in https://hackmd.io/WeqiDWMAQP2sNtuPRul9QA

[jlebon]

I'm not opposed but it feels like it's going to scale badly if more extensions get created. I'd rather go in the "run a service in the cluster" path being proposed in hackmd.io/WeqiDWMAQP2sNtuPRul9QA

Oh yeah definitely agree this is the more sustainable approach long-term for both the OSTree content and RPM extensions. Though that's clearly a larger chunk of work which is harder to make 4.6. Can the extensions work wait until 4.7?

[miabbott]

Oh yeah definitely agree this is the more sustainable approach long-term for both the OSTree content and RPM extensions. Though that's clearly a larger chunk of work which is harder to make 4.6. Can the extensions work wait until 4.7?

"Don't let perfection be the enemy of good."

We can't keep punting the extensions work until we find the most optimal solution. The proposal that has been worked in the MCO PR and related enhancement is good enough for us at this point. We should follow through on that to completion and iterate on the design for 4.7

[miabbott]

Image labels sound fine, though worth mentioning that since we already do pull down the oscontainer during pivot/update, we could also just copy any embedded metadata over from the container while it's mounted.

I believe the thought was that having information available in the image labels would allow cluster admins (or actual k8s operators) to make informed decisions if the pending upgrade was safe to take. We want to avoid the case where an out of tree driver cannot be recompiled because of a kernel incompatibility and thus the cluster is degraded.

[jlebon]

"Don't let perfection be the enemy of good."

+1. The suggestion I had was more of a minor iteration on top of the current design. But regardless, as long as we maintain the same user-facing API, there should be no problem reworking how it's all implemented down the road in a way that better aligns with openshift/machine-config-operator#1897.

[sinnykumari]

We can't keep punting the extensions work until we find the most optimal solution. The proposal that has been worked in the MCO PR and related enhancement is good enough for us at this point. We should follow through on that to completion and iterate in the design for 4.7

+1,
If this can go into enhancement, it will be nice to have a 4.7 enhancement to identify and discuss at broader level all the possible benefits we can have.

Also, let's keep this upcoming work in mind while reviewing current extension PR openshift/machine-config-operator#1850.

[zvonkok]

One of the intentions from my side was to have an interface that I (SRO) can use to examine if an update will cause troubles with out-of-tree drivers, other kernel modules or other artifacts (e.g. sysdig agents) that rely on kernel versions, glibc or the OS version (openshift/enhancements#357).

With some vendors, we developed a compatibility or support matrix (similar to what we have for ubi7,8 containers running on rhel7,8) which explains which driver works with which kernel or os version.

This vendor-specific business logic can then be used as a preflight check during an upgrade.

Having a minimally invasive way to check for kernel, glibc, RHEL version in the update is preferable, rather than pulling the image, mounting and inspecting which rpms are in the image, getting the RHEL OS version (not the RHCOS version, drivers are developed on specific RHEL versions 8.0, 8.1 ... ) etc.

How is this anticipated to work? Should operators e.g. run skopeo inspect updae-os-image-url?

Can we make this work in such a way so I do not need an authfile?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[miabbott]

/remove-lifecycle stale

This is still tracked on our Jira board, though at a low priority