[rhel-9.4 variant] `ext.config.rpm-ostree.replace-rt-kernel` fails

[marmijo]

Oct 16 21:33:00 qemu0 systemd[1]: Started kola-runext.service.
Oct 16 21:33:00 qemu0 kola-runext-replace-rt-kernel[2046]: + rm -rf /etc/yum.repos.d/centos-addons.repo /etc/yum.repos.d/centos.repo
Oct 16 21:33:00 qemu0 kola-runext-replace-rt-kernel[2048]: + curl -sSLf https://raw.githubusercontent.com/openshift/os/master/c9s.repo -o /etc/yum.repos.d/cs.repo
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2051]: + curl -sSLf https://centos.org/keys/RPM-GPG-KEY-CentOS-Official -o /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2054]: + sed -i 's|^gpgkey.*|gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official|' /etc/yum.repos.d/cs.repo
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2040]: Testing overriding with CentOS Stream kernel
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2056]: + sed -i s/enabled=1/enabled=0/g /etc/yum.repos.d/cs.repo
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2058]: + sed -i '/\[baseos\]/,/^ *\[/ s/enabled=0/enabled=1/' /etc/yum.repos.d/cs.repo
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2060]: + sed -i '/\[appstream\]/,/^ *\[/ s/enabled=0/enabled=1/' /etc/yum.repos.d/cs.repo
Oct 16 21:33:01 qemu0 kola-runext-replace-rt-kernel[2062]: + rpm-ostree override replace --experimental --from repo=baseos kernel kernel-core kernel-modules kernel-modules-extra kernel-modules-core
Oct 16 21:33:02 qemu0 kola-runext-replace-rt-kernel[2063]: Checking out tree 3b41b5b...done
Oct 16 21:33:02 qemu0 kola-runext-replace-rt-kernel[2063]: Enabled rpm-md repositories: baseos appstream
Oct 16 21:33:03 qemu0 kola-runext-replace-rt-kernel[2063]: Updating metadata for 'baseos'...done
Oct 16 21:33:04 qemu0 kola-runext-replace-rt-kernel[2063]: Updating metadata for 'appstream'...done
Oct 16 21:33:05 qemu0 kola-runext-replace-rt-kernel[2063]: Importing rpm-md...done
Oct 16 21:33:05 qemu0 kola-runext-replace-rt-kernel[2063]: rpm-md repo 'baseos'; generated: 2023-10-16T04:03:11Z solvables: 1158
Oct 16 21:33:05 qemu0 kola-runext-replace-rt-kernel[2063]: rpm-md repo 'appstream'; generated: 2023-10-16T04:02:52Z solvables: 5630
Oct 16 21:33:05 qemu0 kola-runext-replace-rt-kernel[2063]: Resolving dependencies...done
Oct 16 21:33:05 qemu0 kola-runext-replace-rt-kernel[2063]: error: Request to reinstall exact base package versions: kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra
Oct 16 21:33:05 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 21:33:05 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.rpm-ostree.replace-rt-kernel (31.43s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2023-10-16T21:33:05Z cli: Unit kola-runext.service exited with code 1
        harness.go:1262: kolet failed: : kolet run-test-unit failed: Process exited with status 1
FAIL, output in tmp/kola

The journal output of the test shows:

Oct 16 22:36:05.041749 rpm-ostreed.service[2062]: Reading config file '/etc/rpm-ostreed.conf'
Oct 16 22:36:05.046789 rpm-ostreed.service[2062]: In idle state; will auto-exit in 60 seconds
Oct 16 22:36:05.047012 init.scope[1]: Started rpm-ostree System Management Daemon.
Oct 16 22:36:05.050871 rpm-ostreed.service[2062]: client(id:cli dbus:1.58 unit:kola-runext.service uid:0) added; new total=1
Oct 16 22:36:05.054962 rpm-ostreed.service[2062]: Loaded sysroot
Oct 16 22:36:05.056242 rpm-ostreed.service[2062]: Locked sysroot
Oct 16 22:36:05.056350 rpm-ostreed.service[2062]: Initiated txn UpdateDeployment for client(id:cli dbus:1.58 unit:kola-runext.service uid:0): /org/projectatomic/rpmostree1/rhcos
Oct 16 22:36:05.056875 kernel: EXT4-fs (vda3): re-mounted. Quota mode: none.
Oct 16 22:36:05.059728 rpm-ostreed.service[2062]: Process [pid: 2058 uid: 0 unit: kola-runext.service] connected to transaction progress
Oct 16 22:36:05.061301 rpm-ostreed.service[2062]: Librepo version: 1.14.5 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.76.1 OpenSSL/3.0.7 zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.10.4/openssl/zlib nghttp2/1.43.0)
Oct 16 22:36:06.042476 kola-runext.service[2058]: Checking out tree 3b41b5b...done
Oct 16 22:36:06.055549 rpm-ostreed.service[2062]: Ignoring pkgcache-only request in presence of module requests and/or remote overrides
Oct 16 22:36:06.056119 kola-runext.service[2058]: Enabled rpm-md repositories: baseos appstream
Oct 16 22:36:06.056687 rpm-ostreed.service[2062]: Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
Oct 16 22:36:06.080721 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/repodata/repomd.xml
Oct 16 22:36:06.382246 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/repodata/81f57e1f40d46e3143f2e73dde2de23fc889c32ec62ca97e6157b28927e128ea-primary.xml.gz
Oct 16 22:36:06.382846 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/repodata/f53861660a341db814c216f759b1364299b124f4353aa3a87c65e70f059f09ce-filelists.xml.gz
Oct 16 22:36:06.382995 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/repodata/d52d95f0f2d1a56b197a698aa328813a7e45e2d5b2883e04f4a3f604638b7a4a-comps-BaseOS.x86_64.xml
Oct 16 22:36:06.749963 kola-runext.service[2058]: Updating metadata for 'baseos'...done
Oct 16 22:36:06.750599 rpm-ostreed.service[2062]: Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
Oct 16 22:36:06.762417 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/repodata/repomd.xml
Oct 16 22:36:06.936546 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/repodata/280eaa4724c3982e01fa845294b17b28a94edfb24a103c70e124fa021a55bb98-primary.xml.gz
Oct 16 22:36:06.936744 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/repodata/33eb35fbae0b61a6742918621d71d33fa7b42c6fc92748b53dd9d2a118961cdb-filelists.xml.gz
Oct 16 22:36:06.936836 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/repodata/5bf6d702e7792539d4136f33fa06a5449dc416a543738cdaa5e68da2a1f0e5b6-comps-AppStream.x86_64.xml
Oct 16 22:36:07.196143 rpm-ostreed.service[2062]: Downloading: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/repodata/7f2ab4e03e642bc5eedbd42c6dfb60a35c082001c345a6c76df89864676c49a9-modules.yaml.gz
Oct 16 22:36:07.450216 kola-runext.service[2058]: Updating metadata for 'appstream'...done
Oct 16 22:36:08.694777 kola-runext.service[2058]: Importing rpm-md...done
Oct 16 22:36:08.695198 kola-runext.service[2058]: rpm-md repo 'baseos'; generated: 2023-10-16T04:03:11Z solvables: 1158
Oct 16 22:36:08.695315 kola-runext.service[2058]: rpm-md repo 'appstream'; generated: 2023-10-16T04:02:52Z solvables: 5630
Oct 16 22:36:08.695532 rpm-ostreed.service[2062]: Preparing pkg txn; enabled repos: ['baseos', 'appstream'] solvables: 6788
Oct 16 22:36:08.701215 kola-runext.service[2058]: Resolving dependencies...done
Oct 16 22:36:08.704490 rpm-ostreed.service[2062]: Txn UpdateDeployment on /org/projectatomic/rpmostree1/rhcos failed: Request to reinstall exact base package versions: kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra
Oct 16 22:36:08.704737 rpm-ostreed.service[2062]: Unlocked sysroot
Oct 16 22:36:08.705916 kola-runext.service[2058]: error: Request to reinstall exact base package versions: kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra
Oct 16 22:36:08.707093 init.scope[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 22:36:08.707160 init.scope[1]: kola-runext.service: Failed with result 'exit-code'.
Oct 16 22:36:08.708390 rpm-ostreed.service[2062]: Process [pid: 2058 uid: 0 unit: (unknown)] disconnected from transaction progress
Oct 16 22:36:08.708976 rpm-ostreed.service[2062]: client(id:cli dbus:1.58 unit:kola-runext.service uid:0) vanished; remaining=0
Oct 16 22:36:08.709103 rpm-ostreed.service[2062]: In idle state; will auto-exit in 61 seconds

Note the identical error message in both outputs:

Oct 16 22:36:08.704490 rpm-ostreed.service[2062]: Txn UpdateDeployment on /org/projectatomic/rpmostree1/rhcos failed: Request to reinstall exact base package versions: kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra
Oct 16 22:36:08.704737 rpm-ostreed.service[2062]: Unlocked sysroot
Oct 16 22:36:08.705916 kola-runext.service[2058]: error: Request to reinstall exact base package versions: kernel kernel-core kernel-modules kernel-modules-core kernel-modules-extra

This failed in the rhcos-devel pipeline run #65

journal.txt
console.txt

[jlebon]

The test tries to first replace with the CentOS Stream kernel:

os/tests/kola/rpm-ostree/replace-rt-kernel

Lines 39 to 47 in 1a68ae4

	"")
	echo "Testing overriding with CentOS Stream kernel"
	# Disable all repos except baseos and appstream as not all of them have support for all RHCOS supported architectures
	runv sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/cs.repo
	runv sed -i '/\[baseos\]/,/^ *\[/ s/enabled=0/enabled=1/' /etc/yum.repos.d/cs.repo
	runv sed -i '/\[appstream\]/,/^ *\[/ s/enabled=0/enabled=1/' /etc/yum.repos.d/cs.repo
	runv rpm-ostree override replace --experimental --from repo=baseos kernel{,-core,-modules,-modules-extra,-modules-core}
	runv /tmp/autopkgtest-reboot 1
	;;

But currently we already are on CentOS Stream. Probably we need to adapt it to just skip that bit of the test in that case and just go straight to the kernel-rt test. (Well, we still need the yum repo sed fiddling bits probably.)

That said, that "Request to reinstall" doesn't smell right; rpm-ostree should've made it an inactive override I think, but possibly we're not handling this path correctly in the --experimental --from=repo=... case. (That part should probably be tracked in a separate rpm-ostree issue.)

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[travier]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale