[BUG] RPM database issue building RHEL 9.4 images

[treydock]

Describe the bug
Creating RHEL 9.4 image layers results in a corrupt RPM database. Would likely be an issue with Rocky 9.x too.

To Reproduce
Generate RHEL 9.4 base image and attempt to run RPM commands outside of packages or package_groups.

$ podman run --rm -it OMIT/osc/cluster-images/rhel-base:9.4 /bin/bash
[root@5e3974217d90 /]# rpm -q kernel
warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.

This causes issues when I try to run rpm or dnf commands outside of packages such as in cmd:

  - cmd: >
      cd /opt/osc/src && git clone https://github.com/Mellanox/nv_peer_memory.git &&
      cd /opt/osc/src/nv_peer_memory &&
      git reset --hard 1.3-0 &&
      ./build_module.sh &&
      rpmbuild --rebuild --define 'KVERSION 5.14.0-427.76.1.el9_4.x86_64' /tmp/nvidia_peer_memory-1.3-0.src.rpm &&
      rpm --force --nodeps -ivh /root/rpmbuild/RPMS/x86_64/nvidia_peer_memory-1.3-0.x86_64.rpm

Which produces this:

ERROR - warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
ERROR - warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
ERROR - error: cannot open Packages index using bdb_ro - Operation not permitted (1)
ERROR - error: cannot open Packages database in /var/lib/rpm
ERROR - subprocess exited with status 1

When I attempt to rebuild the RPM database I see this:

[root@5e3974217d90 /]# rpmdb --rebuilddb
warning: Converting database from bdb_ro to sqlite backend
error: failed to replace old database with new database!
error: replace files in /var/lib/rpm with files from /var/lib/rpmrebuilddb.29 to recover

Something I noticed is the files inside the RPM database on RHEL 9 hosts looks like this:

From failed image rebuild built using image-builder:

[root@9592a8c749db /]# ls -la /var/lib/rpmrebuilddb.23/
total 59444
drwxr-xr-x 2 root root     4096 Aug  4 18:51 .
drwxr-xr-x 1 root root     4096 Aug  4 18:51 ..
-rw-r--r-- 1 root root 60633088 Aug  4 18:51 rpmdb.sqlite
-rw-r--r-- 1 root root   229376 Aug  4 18:51 rpmdb.sqlite-shm
-rw-r--r-- 1 root root        0 Aug  4 18:51 rpmdb.sqlite-wal

From regular RHEL 9.4 host:

[tdockendorf@cardinal-rw02 container-images]$ ls -la /var/lib/rpm
total 432140
drwxr-xr-x  2 root root      4096 Dec 13  2023 .
drwxr-xr-x 84 root root      8192 Jan 18  2024 ..
-rw-r--r--  1 root root 440725504 Apr 30 10:19 rpmdb.sqlite
-rw-r--r--  1 root root     32768 Aug  4 14:14 rpmdb.sqlite-shm
-rw-r--r--  1 root root         0 Apr 30 10:19 rpmdb.sqlite-wal
-rw-r--r--  1 root root         0 Jan 10  2024 .rpm.lock

This is what our RHEL 8 hosts look like:

[root@build-el8 ~]# ls -la /var/lib/rpm
total 859944
drwxr-xr-x.  2 root root      4096 Aug  4 12:00 .
drwxr-xr-x. 54 root root      4096 Aug  6  2024 ..
-rw-r--r--.  1 root root  11599872 Aug  4 10:40 Basenames
-rw-r--r--.  1 root root     16384 Aug  4 10:39 Conflictname
-rw-r--r--   1 root root    311296 Aug  4 14:51 __db.001
-rw-r--r--   1 root root     90112 Aug  4 14:51 __db.002
-rw-r--r--   1 root root   1318912 Aug  4 14:51 __db.003
-rw-r--r--.  1 root root         0 Jan 31  2022 .dbenv.lock
-rw-r--r--.  1 root root  20668416 Aug  4 10:40 Dirnames
-rw-r--r--.  1 root root      8192 Jan 31  2022 Enhancename
-rw-r--r--.  1 root root      8192 May 21 09:12 Filetriggername
-rw-r--r--.  1 root root     28672 Aug  4 10:40 Group
-rw-r--r--.  1 root root     32768 Aug  4 10:40 Installtid
-rw-r--r--.  1 root root     57344 Aug  4 10:40 Name
-rw-r--r--.  1 root root     24576 Aug  4 10:40 Obsoletename
-rw-r--r--.  1 root root 839446528 Aug  4 10:40 Packages
-rw-r--r--.  1 root root   6324224 Aug  4 10:40 Providename
-rw-r--r--.  1 root root     16384 Aug  4 10:39 Recommendname
-rw-r--r--.  1 root root    466944 Aug  4 10:39 Requirename
-rw-r--r--.  1 root root         0 Jan 31  2022 .rpm.lock
-rw-r--r--.  1 root root    143360 Aug  4 10:40 Sha1header
-rw-r--r--.  1 root root     81920 Aug  4 10:40 Sigmd5
-rw-r--r--.  1 root root      8192 May 21 09:12 Suggestname
-rw-r--r--.  1 root root      8192 May 21 09:12 Supplementname
-rw-r--r--.  1 root root      8192 May 21 09:12 Transfiletriggername
-rw-r--r--.  1 root root      8192 May 21 09:12 Triggername

Expected behavior

I'd expect the RPM database generated for an EL9 image to be compatible with EL9 but instead it seems to inherit from the image-builder image's OS which is EL8.

This may require OS-specific versions of the image-builder image, ie EL9 image for EL9 image builds as I'm pretty sure if I replaced the RPM database to use sqlite for EL9 it would break the image-builder image's ability to run dnf commands to install packages into new layers.

I'd be happy to contribute a PR to add Rocky 9 based image-builder image , maybe ghcr.io/openchami/image-build-el9:latest?