RHCOS 4.11 ppc64le test failures due to `libguestfs` errors?

[miabbott]

There's a lot going on in the ppc64le RHCOS 4.11 build:

libguestfs failure messages:

Details

libguestfs: error: appliance closed the connection unexpectedly.
This usually means the libguestfs appliance crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
libguestfs: error: /usr/bin/qemu-system-ppc64 exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
libguestfs: error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
2022-02-16T14:23:39Z kola: retryloop: failed to bring up machines: ignition injection with guestfs failed: guestfish launch failed: exit status 1
libguestfs: error: appliance closed the connection unexpectedly.
This usually means the libguestfs appliance crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
libguestfs: error: /usr/bin/qemu-system-ppc64 exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
libguestfs: error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
2022-02-16T14:23:41Z kola: retryloop: failed to bring up machines: ignition injection with guestfs failed: guestfish launch failed: exit status 1
--- FAIL: ext.config.shared.networking.prefer-ignition-networking (5.59s)
        harness.go:1381: Cluster failed starting machines: ignition injection with guestfs failed: guestfish launch failed: exit status 1

List of failed tests:

• FAIL: ext.config.shared.networking.prefer-ignition-networking (5.59s)
• FAIL: ext.config.shared.networking.mtu-on-bond-kargs (1.97s)
• FAIL: coreos.ignition.mount.partitions (682.80s)
• FAIL: ext.config.shared.networking.force-persist-ip
• FAIL: coreos.ignition.mount.disks
• FAIL: coreos.boot-mirror
• FAIL: coreos.boot-mirror.luks
• FAIL: ext.config.shared.reboot
• FAIL: ext.config.shared.kdump.crash

Build log - https://url.corp.redhat.com/f0f692f

kola_artifacts.zip

[miabbott]

cc: @Prashanth684 @ravanelli @deepsm007

[ravanelli]

Running the libguest test it seems to end as expected ( we pass in the CI vsmt here).

We probably will need to check if the wrapped is being used/working in these tests + issues with F35 and the cluster host.

I will try to run kola tests with the libguest debug.

LIBGUESTFS_BACKEND=direct
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SELinux: Disabled
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: direct
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/bin/qemu-system-ppc64
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: powerpc64le
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.46.2fedora=35,release=1.fc35,libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: unix
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsqp50c1
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu powerpc64le
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.3.1
supermin: rpm: detected RPM version 4.17
supermin: rpm: detected RPM architecture ppc64le
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-packages-xfs type uncompressed packages
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 188 packages, including dependencies
supermin: build: 35419 files
supermin: build: 8467 files, after matching excludefiles
supermin: build: 8478 files, after adding hostfiles
supermin: build: 8439 files, after removing unreadable files
supermin: build: 8461 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/5.15.18-200.fc35.ppc64le/vmlinuz
supermin: kernel: kernel_version 5.15.18-200.fc35.ppc64le
supermin: kernel: modpath /lib/modules/5.15.18-200.fc35.ppc64le
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.ntsf1ugk/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.ntsf1ugk/initrd'
supermin: ext2: wrote 37 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.ntsf1ugk to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: begin testing qemu features
libguestfs: checking for previously cached test results of /usr/bin/qemu-system-ppc64, in /var/tmp/.guestfs-0
libguestfs: command: run: /usr/bin/qemu-system-ppc64
libguestfs: command: run: \ -display none
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/bin/qemu-system-ppc64
libguestfs: command: run: \ -display none
libguestfs: command: run: \ -machine pseries,accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: command: run: echo '{ "execute": "qmp_capabilities" }' '{ "execute": "query-qmp-schema" }' '{ "execute": "quit" }'  | QEMU_AUDIO_DRV=none "/usr/bin/qemu-system-ppc64" -display none -machine "pseries,accel=kvm:tcg" -qmp stdio
libguestfs: did not understand QMP monitor output from /usr/bin/qemu-system-ppc64 (ignored)
libguestfs: command: run: echo '{ "execute": "qmp_capabilities" }' '{ "execute": "query-kvm" }' '{ "execute": "quit" }'  | QEMU_AUDIO_DRV=none "/usr/bin/qemu-system-ppc64" -display none -machine "pseries,accel=kvm:tcg" -qmp stdio
libguestfs: did not understand QMP monitor output from /usr/bin/qemu-system-ppc64 (ignored)
libguestfs: saving test results
libguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)
libguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)
libguestfs: qemu version: 6.1
libguestfs: qemu mandatory locking: yes
libguestfs: qemu KVM: enabled
libguestfs: finished testing qemu features
/usr/bin/qemu-system-ppc64 \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -nodefaults \
    -display none \
    -machine pseries,accel=kvm:tcg \
    -m 1280 \
    -no-reboot \
    -rtc driftfix=slew \
    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/tmp/libguestfsqp50c1/scratch1.img,cache=unsafe,format=raw,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -chardev socket,path=/tmp/libguestfsd6gFaR/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -append "panic=1 console=hvc0 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=16c82726-639b-4e93-a394-dd8faf328dc5 selinux=0 guestfs_verbose=1 TERM=xterm"
qemu-system-ppc64: Failed to set KVM's VSMT mode to 1 (errno -22)
On PPC, a VM with 1 threads/core on a host with 8 threads/core requires the use of VSMT mode 1.
This KVM seems to be too old to support VSMT.
libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages
libguestfs: child_cleanup: 0x10034a91320: child process died
libguestfs: sending SIGTERM to process 829
libguestfs: error: /usr/bin/qemu-system-ppc64 exited with error status 1, see debug messages above
libguestfs: error: guestfs_launch failed, see earlier error messages
libguestfs: closing guestfs handle 0x10034a91320 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsqp50c1
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsd6gFaR

[mrcxavier]

log-libguest2.txt
This is a full log that I extracted. As Renata posted earlier, it reports "libguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)" and could be related to libguestfish.sh placed in the line 19 from coreos-assembler repo
Here it is:

14 if [ "$arch" = "ppc64le" ] ; then
15 tmp_qemu_wrapper=$(mktemp -tdp /tmp gf-vsmt.XXXXXX)
16 qemu_wrapper=${tmp_qemu_wrapper}/qemu-wrapper.sh
17 cat <<-'EOF' > "${qemu_wrapper}"
18 #!/bin/bash -
19 exec qemu-system-ppc64 "$@" -machine pseries,accel=kvm:tcg,vsmt=8,cap-fwnmi=off
20 EOF
21 chmod +x "${qemu_wrapper}"

[miabbott]

Thanks for the log and extra info, Marcio

Looking at the libguestfish.sh code, it hasn't changed since early 2021, so I'm not convinced the error lies in there.

The error even says ignored:

libguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)
libguestfs: QMP parse error: '[' or '{' expected near end of file (ignored)

I suspect the real error is related to this:

qemu-system-ppc64: Failed to set KVM's VSMT mode to 1 (errno -22)
On PPC, a VM with 1 threads/core on a host with 8 threads/core requires the use of VSMT mode 1.
This KVM seems to be too old to support VSMT.

Since this error seems to have cropped up with the use of the F35 cosa, I wonder if something changed in QEMU that is causing this problem.

An interesting test might be to build a cosa container from the rhcos-4.10 branch using F34 and see if it is reproducible.

[cgwalters]

This KVM seems to be too old to support VSMT.

Probably related to being on a RHEL7 host. We may need to accelerate our move to the new cluster.

Bigger picture, we need to invert all of this and run the host system faster while keeping workloads i.e. more stable as a general rule. (With cherry picked fast paths for e.g. ostree etc.)

Downgrading qemu would probably help indeed, but using F34 for everything would break the coreos layering work and not be sustainable. (We can stay on F34 for 4.10 cosa)

I'd try this instead:

$ podman run --rm -ti quay.io/coreos-assembler/coreos-assembler:latest shell
[coreos-assembler]$ sudo yum --setopt=releasever=34 distro-sync --allowerasing qemu-kvm
Repository 'f35-coreos-continuous' is missing name in configuration, using id.
yLast metadata expiration check: 0:00:32 ago on Thu Feb 17 15:37:53 2022.
Dependencies resolved.
====================================================================================================================================================================================
 Package                                                        Architecture                    Version                                     Repository                         Size
====================================================================================================================================================================================
Upgrading:
 glusterfs-client-xlators                                       x86_64                          9.5-1.fc34                                  updates                           903 k
 libgfapi0                                                      x86_64                          9.5-1.fc34                                  updates                            93 k
 libgfrpc0                                                      x86_64                          9.5-1.fc34                                  updates                            57 k
 libgfxdr0                                                      x86_64                          9.5-1.fc34                                  updates                            30 k
 libglusterfs0                                                  x86_64                          9.5-1.fc34                                  updates                           302 k
Installing dependencies:
 device-mapper-multipath-libs                                   x86_64                          0.8.5-4.fc34                                fedora                            268 k
Removing dependent packages:
 qemu-audio-jack                                                x86_64                          2:6.1.0-10.fc35                             @updates                           24 k
 qemu-device-display-vhost-user-gpu                             x86_64                          2:6.1.0-10.fc35                             @updates                          255 k
 qemu-device-display-virtio-gpu-gl                              x86_64                          2:6.1.0-10.fc35                             @updates                           38 k
 qemu-device-display-virtio-gpu-pci-gl                          x86_64                          2:6.1.0-10.fc35                             @updates                           16 k
 qemu-device-display-virtio-vga-gl                              x86_64                          2:6.1.0-10.fc35                             @updates                           15 k
 qemu-device-usb-host                                           x86_64                          2:6.1.0-10.fc35                             @updates                           51 k
Downgrading:
 boost-iostreams                                                x86_64                          1.75.0-4.fc34                               fedora                             39 k
 boost-system                                                   x86_64                          1.75.0-4.fc34                               fedora                             14 k
 boost-thread                                                   x86_64                          1.75.0-4.fc34                               fedora                             57 k
 fmt                                                            x86_64                          7.1.3-3.fc34                                updates                           116 k
 librados2                                                      x86_64                          2:16.2.7-2.fc34                             updates                           3.5 M
 librbd1                                                        x86_64                          2:16.2.7-2.fc34                             updates                           3.7 M
 liburing                                                       x86_64                          0.7-4.fc34                                  fedora                             26 k
 lttng-ust                                                      x86_64                          2.12.0-4.fc34                               fedora                            289 k
 qemu-audio-alsa                                                x86_64                          2:5.2.0-8.fc34                              updates                            21 k
 qemu-audio-oss                                                 x86_64                          2:5.2.0-8.fc34                              updates                            19 k
 qemu-audio-pa                                                  x86_64                          2:5.2.0-8.fc34                              updates                            20 k
 qemu-audio-sdl                                                 x86_64                          2:5.2.0-8.fc34                              updates                            17 k
 qemu-audio-spice                                               x86_64                          2:5.2.0-8.fc34                              updates                            17 k
 qemu-block-curl                                                x86_64                          2:5.2.0-8.fc34                              updates                            22 k
 qemu-block-dmg                                                 x86_64                          2:5.2.0-8.fc34                              updates                            15 k
 qemu-block-gluster                                             x86_64                          2:5.2.0-8.fc34                              updates                            24 k
 qemu-block-iscsi                                               x86_64                          2:5.2.0-8.fc34                              updates                            30 k
 qemu-block-nfs                                                 x86_64                          2:5.2.0-8.fc34                              updates                            23 k
 qemu-block-rbd                                                 x86_64                          2:5.2.0-8.fc34                              updates                            24 k
 qemu-block-ssh                                                 x86_64                          2:5.2.0-8.fc34                              updates                            25 k
 qemu-char-baum                                                 x86_64                          2:5.2.0-8.fc34                              updates                            19 k
 qemu-char-spice                                                x86_64                          2:5.2.0-8.fc34                              updates                            18 k
 qemu-common                                                    x86_64                          2:5.2.0-8.fc34                              updates                           2.1 M
 qemu-device-display-qxl                                        x86_64                          2:5.2.0-8.fc34                              updates                            40 k
 qemu-device-display-virtio-gpu                                 x86_64                          2:5.2.0-8.fc34                              updates                            35 k
 qemu-device-display-virtio-gpu-pci                             x86_64                          2:5.2.0-8.fc34                              updates                            16 k
 qemu-device-display-virtio-vga                                 x86_64                          2:5.2.0-8.fc34                              updates                            17 k
 qemu-device-usb-redirect                                       x86_64                          2:5.2.0-8.fc34                              updates                            35 k
 qemu-device-usb-smartcard                                      x86_64                          2:5.2.0-8.fc34                              updates                            24 k
 qemu-img                                                       x86_64                          2:5.2.0-8.fc34                              updates                           1.2 M
 qemu-kvm                                                       x86_64                          2:5.2.0-8.fc34                              updates                            11 k
 qemu-kvm-core                                                  x86_64                          2:5.2.0-8.fc34                              updates                            11 k
 qemu-system-x86                                                x86_64                          2:5.2.0-8.fc34                              updates                            12 k
 qemu-system-x86-core                                           x86_64                          2:5.2.0-8.fc34                              updates                           6.8 M
 qemu-ui-curses                                                 x86_64                          2:5.2.0-8.fc34                              updates                            24 k
 qemu-ui-egl-headless                                           x86_64                          2:5.2.0-8.fc34                              updates                            17 k
 qemu-ui-gtk                                                    x86_64                          2:5.2.0-8.fc34                              updates                            36 k
 qemu-ui-opengl                                                 x86_64                          2:5.2.0-8.fc34                              updates                            22 k
 qemu-ui-sdl                                                    x86_64                          2:5.2.0-8.fc34                              updates                            26 k
 qemu-ui-spice-app                                              x86_64                          2:5.2.0-8.fc34                              updates                            17 k
 qemu-ui-spice-core                                             x86_64                          2:5.2.0-8.fc34                              updates                            34 k
 userspace-rcu                                                  x86_64                          0.12.1-3.fc34                               fedora                            105 k
 xen-libs                                                       x86_64                          4.14.3-3.fc34                               updates                           620 k

Transaction Summary
====================================================================================================================================================================================
Install     1 Package
Upgrade     5 Packages
Remove      6 Packages
Downgrade  43 Packages

Total download size: 21 M
Is this ok [y/N]: y
...
Complete!
[coreos-assembler]$ 

[cgwalters]

If that works we can do that in the main cosa builds for now.

[miabbott]

@mrcxavier @ravanelli @Prashanth684 could one of you try rebuilding the cosa:main container with the change Colin suggested and see if it allows for the tests to pass?

[ravanelli]

@miabbott I was already doing @cgwalters suggestion to downgrade qemu, I will post the results here soon.

[ravanelli]

Downgrading libguestfs, qemu and libvirt even to F34 versions didn't work. Looks like the libguestfs is not quite able to get the wrapped done via file, we will need to spend more time debugging it. If we do the full wrapped manually and export it, that works.
It is kind of ugly, but to unlock the power builds we may force it via build.sh or some place else.

Versions tested:
libguestfs-1.46.2-1.fc34.ppc64le
qemu-kvm-5.2.0-8.fc34.ppc64le
qemu-kvm-core-5.2.0-8.fc34.ppc64le
libvirt-daemon-7.0.0-8.fc34.ppc64le

Do you think @cgwalters and @miabbott ?

sh-5.1# export LIBGUESTFS_HV="${qemu_wrapper}"
sh-5.1# cosa kola run ext.config.shared.networking.mtu-on-bond-kargs 
kola --output-dir tmp/kola run ext.config.shared.networking.mtu-on-bond-kargs
⚠️  Skipping kola test pattern "coreos.ignition.journald-log":
  👉 https://github.com/coreos/coreos-assembler/issues/1173
=== RUN   ext.config.shared.networking.mtu-on-bond-kargs
--- PASS: ext.config.shared.networking.mtu-on-bond-kargs (63.12s)
PASS, output in tmp/kola

[cgwalters]

Let's put up a PR with whatever changes you had to make? I don't quite understand what's going on with the LIBGUESTFS_HV bits...

Maybe we could do a screen share session? I'll be around for a bit longer today.

My suspicion offhand is that libguestfs is trying to speak QMP over stdout or something, and qemu it outputting a warning there that's conflicting.

[ravanelli]

Probably related to being on a RHEL7 host. We may need to accelerate our move to the new cluster.

I think it will be also a pain, RHEL8 host probably won't work with F35 (https://gitlab.com/qemu-project/qemu/-/issues/860). due a missing Kernel patch. We probably will need to have a strong reason/person to ask RHEL8 folks to allow this backport for us, or we will need to use a F35 host.