PMU causes guestfish to fail on VMware hypervisors

[dcode]

I'm trying to just get through a successful build. My apologies if this is another oops.

The host I'm running on is a Fedora 29 (upgraded from 28 today) VM on ESXi, with virtualization passthrough turned on (/dev/kvm is available). After doing a clean init, I run coreos-assembler build per the docs.

The build produces a qcow2 image from virt-install and proceeds to run gf-oemid, where it fails.

I've attached the full build log.

+ guestfish --remote -- run
libguestfs: error: appliance closed the connection unexpectedly.
This usually means the libguestfs appliance crashed.

coreos-build.log

[dustymabe]

are you building locally? I know @miabbott was hitting some issues with the VMs, but I think his were way earlier than this. Any chance your machine ran out of disk or memory ?

[cgwalters]

Can you do a coreos-assembler shell, and inside there do e.g. env LIBGUESTFS_DEBUG=1 coreos-assembler gf-oemid /path/to/qemu.qcow2.

[dcode]

@dustymabe the VM has 32 GB of memory, I've got more in the hypervisor, but I'd be surprised if I'm hitting that. As for disk, I'm got 17 GB free after the qcow2 image is already built.

@cgwalters that command failed.

[coreos-assembler]$ export LIBGUESTFS_DEBUG=1
[coreos-assembler]$ coreos-assembler gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
Unknown command: gf-oemid

[cgwalters]

Sorry, it should be env LIBGUESTFS_DEBUG=1 /usr/lib/coreos-assembler/gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /srv/tmp/fcos-qemu.qcow2 qemu

[dcode]

Okay, so it does appear to be a VMware hypervisor issue with nested VMs. gotoz/runq#2 looks like possibly the same problem. They have a fix by passing some additional CPU args to qemu in gotoz/runq@ee66e5a.

It's not clear to me if this has to be tried in coreos-assembler, supermin, or guestfish. I'll give it some effort and see if I can make it work though.

coreos-assembler_gf-oemid.log

[dustymabe]

If you see your original log the anaconda install (which does run in a VM) did work, but that is run via virt-install and thus probably calls qemu-kvm with different options than guestfish.

From your last log these are interesting statements:

qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /builddir/build/BUILD/qemu-2.11.2/target/i386/kvm.c:1807: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

Try running the virt-install command from your original log in the container and add --dry-run --print-xml and see if there is anything interesting in the xml for cpu information. The guestfish command that is failing is using -cpu host.

[cgwalters]

http://libguestfs.org/guestfs.3.html#qemu-wrappers

[dcode]

I ran

[coreos-assembler]$ qemu-img create -f qcow2 /srv/tmp/build/tmp/fedora-coreos-29-base-2.qcow2 8G
[coreos-assembler]$ virt-install --connect=qemu:///session --name=coreos-inst-13281-1541708728 --noautoconsole --console=log.file=/srv/tmp/build/install.log --filesystem=source=/srv/tmp/build/tmp/anaconda,target=/mnt/logs,accessmode=mapped --filesystem=source=/srv/repo,target=/mnt/ostree-repo,accessmode=mapped --wait=10 --noreboot --nographics --memory=2048 --vcpus=sockets=4,cores=1,threads=1 --os-variant=rhel7 --rng=/dev/urandom --check path_in_use=off --network=none --disk=path=/srv/tmp/build/tmp/fedora-coreos-29-base-2.qcow2,cache=unsafe --location=/srv/installer/Fedora-Everything-netinst-x86_64-28-1.1.iso --initrd-inject=/srv/tmp/build/tmp/flattened.ks --extra-args "ks=file://flattened.ks console=tty0 console=ttyS0,115200n8 inst.cmdline inst.notmux" --dry-run --print-xml

It produced the attached xml. I'm working on the qemu-wrapper shim. I'll report back the results.

dry_run.txt

[dcode]

Sweet. Looks like it worked. If it works on non-vmware hypervisors can we just enable the CPU flag using the wrapper across the board?

[coreos-assembler]$ export  LIBGUESTFS_HV=$(pwd)/qemu-wrapper.sh
[coreos-assembler]$ export LIBGUESTFS_DEBUG=1
[coreos-assembler]$ cat qemu-wrapper.sh
#!/bin/bash -

i=0
while [ $# -gt 0 ]; do
    case "$1" in
    -cpu)
        shift 2;;
    *)
        args[i]="$1"
        (( i++ ))
        shift ;;
    esac
done

exec qemu-kvm -cpu host,pmu=off "${args[@]}"
[coreos-assembler]$ /usr/lib/coreos-assembler/gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /srv/tmp/fcos-qemu.qcow2 qemu

guestfish_success.log