RHCOS 4.11 failed to build qemu artifact

[miabbott]

All the arches are hitting this

+ ostree container unencapsulate --repo=/tmp/rootfs/ostree/repo ostree-unverified-image:oci-archive:/home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/builds/411.85.202203221537-0/x86_64/rhcos-411.85.202203221537-0-ostree.x86_64.ociarchive
�[0m�[31merror: �[0mImporting ostree-unverified-image:oci-archive:/home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/builds/411.85.202203221537-0/x86_64/rhcos-411.85.202203221537-0-ostree.x86_64.ociarchive: remote error: lstat /home/jenkins/agent/builds: no such file or directory
++ dump_err_info
++ lsblk -f
NAME   FSTYPE FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
vda    ext2   1.0              1bdbb906-48a5-41a7-a964-a3775736538c    3.9G    16% /
vdb                                                                                
|-vdb1                                                                             
|-vdb2 vfat   FAT16 EFI-SYSTEM E663-012B                             126.7M     0% /tmp/rootfs/boot/efi
|-vdb3 ext4   1.0   boot       96d15588-3596-4b3c-adca-a2ff7279ea63  338.7M     0% /tmp/rootfs/boot
`-vdb4 xfs          root       910678ff-f77e-4a7d-8d53-86f2ac47a823    3.3G     2% /tmp/rootfs

real	0m0.133s
user	0m0.030s
sys	0m0.043s
+ rm -rf /home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/tmp/build.qemu/supermin.out /home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/tmp/build.qemu/supermin.prepare /home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/tmp/build.qemu/supermin.build
+ '[' '!' -f /home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/tmp/build.qemu/rc ']'
++ cat /home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/tmp/build.qemu/rc
+ rc=1
+ return 1
+ rm -rf /home/jenkins/agent/workspace/rhcos-art/rhcos-art-rhcos-4.11/builds/411.85.202203221537-0/x86_64/.qemu.building
fatal: failed buildextend-qemu

Looks like fallout from coreos/coreos-assembler#2764

[miabbott]

@jlebon @cgwalters ^^

[cgwalters]

OK so there's something weird going on in the jenkins code with

                rm -rvf builds
                ln -svf /home/jenkins/agent/builds ${env.WORKSPACE}/builds

OK, this seems to relate to using a PVC...I think there's something here where we're not mounting that source path into the supermin environment? Definitely a hot mess. The real fix is to stop trying to build everything in one single pod.

So this probably worked before because we were pulling from tmp/repo which isn't in /home/jenkins.

[cgwalters]

Looking at a patch to cosa to have it follow symlinks for builds