RHCOS 4.11 s390x: failing tests (networking, reboot, kdump)

[miabbott]

Seeing failures in:

• ext.config.shared.networking.mtu-on-bond-ignition
• ext.config.shared.networking.force-persist-ip
• ext.config.shared.networking.prefer-ignition-networking
• ext.config.shared.reboot
• ext.config.shared.kdump.crash

Details

systemctl status kola-runext.service:
��� kola-runext.service
   Loaded: loaded (/etc/systemd/system/kola-runext.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2022-02-03 20:13:05 UTC; 1s ago
  Process: 1786 ExecStart=/usr/local/bin/kola-runext-mtu-on-bond-kargs (code=exited, status=10)
 Main PID: 1786 (code=exited, status=10)

Feb 03 20:13:05 localhost systemd[1]: Started kola-runext.service.
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: + . /var/opt/kola/extdata/commonlib.sh
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: + bond=bond0
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: + vlan=bond0.100
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: + for interface in $bond $vlan
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: + mtu=
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: ++ nmcli -g 802-3-ethernet.mtu connection show bond0
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: Error: bond0 - no such connection profile.
Feb 03 20:13:05 localhost kola-runext-mtu-on-bond-kargs[1786]: + mtu=
Feb 03 20:13:05 localhost systemd[1]: kola-runext.service: Main process exited, code=exited, status=10/n/a
Feb 03 20:13:05 localhost systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.shared.networking.mtu-on-bond-kargs (31.25s)
        harness.go:902: kolet failed: : kolet run-test-unit failed: Error: Unit kola-runext.service exited with code 10
2022-02-03T20:13:06Z cli: Unit kola-runext.service exited with code 10: Process exited with status 1

systemctl status kola-runext.service:
��� kola-runext.service
   Loaded: loaded (/etc/systemd/system/kola-runext.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2022-02-03 20:16:23 UTC; 1s ago
  Process: 1816 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
 Main PID: 1816 (code=exited, status=1/FAILURE)

Feb 03 20:16:23 localhost systemd[1]: Started kola-runext.service.
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + . /var/opt/kola/extdata/commonlib.sh
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + bond=bond0
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + vlan=bond0.100
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + for interface in $bond $vlan
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + mtu=
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ nmcli -g 802-3-ethernet.mtu connection show bond0
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + mtu=9000
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + '[' 9000 '!=' 9000 ']'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + ok 'bond0 mtu is correct'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + echo ok 'bond0 mtu is correct'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ok bond0 mtu is correct
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + for interface in $bond $vlan
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + mtu=
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ nmcli -g 802-3-ethernet.mtu connection show bond0.100
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + mtu=9000
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + '[' 9000 '!=' 9000 ']'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + ok 'bond0.100 mtu is correct'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + echo ok 'bond0.100 mtu is correct'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ok bond0.100 mtu is correct
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ get_ipv4_for_nic bond0.100
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ local nic_name=bond0.100
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: +++ jq -r '.[0].addr_info | map(select(.family == "inet")) | .[0].local'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: +++ ip -j addr show bond0.100
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ local ip=null
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ '[' -z null ']'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: ++ echo null
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + nic_ip=null
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + '[' null '!=' 10.10.10.10 ']'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + fatal 'Error: get bond0.100 ip = null, expected is 10.10.10.10'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + echo 'Error: get bond0.100 ip = null, expected is 10.10.10.10'
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: Error: get bond0.100 ip = null, expected is 10.10.10.10
Feb 03 20:16:23 localhost kola-runext-test.sh[1816]: + exit 1
Feb 03 20:16:23 localhost systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Feb 03 20:16:23 localhost systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.shared.networking.mtu-on-bond-ignition (28.68s)
        harness.go:902: kolet failed: : kolet run-test-unit failed: Error: Unit kola-runext.service exited with code 1

systemctl status kola-runext.service:
��� kola-runext.service
   Loaded: loaded (/etc/systemd/system/kola-runext.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2022-02-03 20:17:20 UTC; 1s ago
  Process: 1750 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
 Main PID: 1750 (code=exited, status=1/FAILURE)

Feb 03 20:17:20 localhost systemd[1]: Started kola-runext.service.
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: + . /var/opt/kola/extdata/commonlib.sh
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: + nic_name=eth1
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: ++ get_ipv4_for_nic eth1
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: ++ local nic_name=eth1
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: +++ jq -r '.[0].addr_info | map(select(.family == "inet")) | .[0].local'
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: +++ ip -j addr show eth1
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: Device "eth1" does not exist.
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: ++ local ip=
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: ++ '[' -z '' ']'
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: ++ echo 'Error: can not get ip for eth1'
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: ++ exit 1
Feb 03 20:17:20 localhost systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Feb 03 20:17:20 localhost kola-runext-test.sh[1750]: + nic_ip='Error: can not get ip for eth1'
Feb 03 20:17:20 localhost systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.shared.networking.force-persist-ip (28.68s)
        harness.go:902: kolet failed: : kolet run-test-unit failed: Error: Unit kola-runext.service exited with code 1
2022-02-03T20:17:21Z cli: Unit kola-runext.service exited with code 1: Process exited with status 1

systemctl status kola-runext.service:
��� kola-runext.service
   Loaded: loaded (/etc/systemd/system/kola-runext.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2022-02-03 20:20:30 UTC; 1s ago
  Process: 1750 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
 Main PID: 1750 (code=exited, status=1/FAILURE)

Feb 03 20:20:30 localhost systemd[1]: Started kola-runext.service.
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: + . /var/opt/kola/extdata/commonlib.sh
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: + nic_name=eth1
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: ++ get_ipv4_for_nic eth1
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: ++ local nic_name=eth1
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: +++ jq -r '.[0].addr_info | map(select(.family == "inet")) | .[0].local'
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: +++ ip -j addr show eth1
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: Device "eth1" does not exist.
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: ++ local ip=
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: ++ '[' -z '' ']'
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: ++ echo 'Error: can not get ip for eth1'
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: ++ exit 1
Feb 03 20:20:30 localhost systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Feb 03 20:20:30 localhost kola-runext-test.sh[1750]: + nic_ip='Error: can not get ip for eth1'
Feb 03 20:20:30 localhost systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.shared.networking.prefer-ignition-networking (29.02s)
        harness.go:902: kolet failed: : kolet run-test-unit failed: Error: Unit kola-runext.service exited with code 1
2022-02-03T20:20:31Z cli: Unit kola-runext.service exited with code 1: Process exited with status 1

systemctl status kola-runext.service:
��� kola-runext.service
   Loaded: loaded (/etc/systemd/system/kola-runext.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2022-02-03 20:25:01 UTC; 1s ago
  Process: 1736 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
 Main PID: 1736 (code=exited, status=1/FAILURE)

Feb 03 20:25:01 qemu0 systemd[1]: Started kola-runext.service.
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + . /var/opt/kola/extdata/commonlib.sh
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: ++ findmnt -nvr /var -o SOURCE
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + varsrc=/dev/vda4
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: ++ findmnt -nvr / -o SOURCE
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + rootsrc=/dev/vda4
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: ++ realpath /dev/vda4
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: ++ realpath /dev/vda4
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + [[ /dev/vda4 == /dev/vda4 ]]
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + ok '/var is backed by rootfs'
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + echo ok '/var is backed by rootfs'
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: ok /var is backed by rootfs
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + grep -q What=/dev/disk/by-uuid
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + systemctl cat boot.mount
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + ok 'boot mounted by UUID'
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + echo ok 'boot mounted by UUID'
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: ok boot mounted by UUID
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + '[' -f /boot/.root_uuid ']'
Feb 03 20:25:01 qemu0 kola-runext-test.sh[1736]: + '[' -f /boot/grub2/bootuuid.cfg ']'
Feb 03 20:25:01 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Feb 03 20:25:01 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
=== RUN   non-exclusive-tests/ext.config.files.conntrack-tools
=== RUN   non-exclusive-tests/ext.config.version.rhaos-pkgs-match-openshift
--- FAIL: ext.config.shared.reboot (26.82s)
        harness.go:902: kolet failed: : kolet run-test-unit failed: Error: Unit kola-runext.service exited with code 1
2022-02-03T20:25:02Z cli: Unit kola-runext.service exited with code 1: Process exited with status 1

--- FAIL: ext.config.shared.kdump.crash (647.70s)
        harness.go:902: kolet failed: : Waiting for reboot: machine "82b4428b-d2e9-4d88-8c6b-bc909c4cc3cc" failed to start: ssh journalctl failed: time limit exceeded
        harness.go:101: TIMEOUT[10m0s]: ssh: journalctl -t kola-runext-crash

kola_artifacts.zip

[miabbott]

cc: @jschintag

[jschintag]

So what i found on the failling tests so far is:

The following tests are newly added in rhcos-4.11:

• ext.config.shared.networking.prefer-ignition-networking
• ext.config.shared.networking.force-persist-ip
• ext.config.shared.networking.mtu-on-bond-kargs
• ext.config.shared.networking.mtu-on-bond-ignition
• ext.config.shared.var-mount.scsi-id

For the networking tests it seems that the keyfile is processed and the connection is created, but no device associated with it.

• ext.config.shared.kdump.crash
  Looking through the commits on the issue kola: ext.config.kdump investigate failures on aarch64 and cloud platforms coreos/fedora-coreos-tracker#860, i don't think this test has ever worked on s390x.

• ext.config.shared.reboot
  This test has since 4.10 been changed to include grub specific checks, causing it to fail on s390x

My Plan for now would be to disable the new tests to get the Pipeline running and fix the ext.config.shared.reboot test to skip the grup-specific parts.

[jschintag]

PR that fixes the tests: coreos/fedora-coreos-config#1498
This does not include ext.config.shared.var-mount.scsi-id, as that is passing on fcos.

[jschintag]

PR to bump fcos-config: #708

[jschintag]

When the changes are merged i will need to check ext.config.shared.var-mount.scsi-id, the test is passing in fcos but currently failling in rhcos-4.11

[jschintag]

So with the fcos-bump we are down to 2 failing tests:

• ext.config.shared.ignition.stable-boot (new)
• ext.config.shared.var-mount.scsi-id

The scis-id seems to fail to find the filesystem on the device:

Details

[   11.686561] systemd[1]: Starting Ignition (disks)...

[   11.763685] ignition[991]: Ignition 2.13.0

[   11.763850] ignition[991]: Stage: disks

[   11.764211] ignition[991]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"

[   11.773878] ignition[991]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"

[   11.777721] ignition[991]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   11.799745] ignition[991]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   11.799815] ignition[991]: disks: createPartitions: created device alias for "/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1": "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1" -> "/dev/sdb"

[   11.799833] ignition[991]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   11.799844] ignition[991]: disks: createPartitions: op(2): wiping partition table requested on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   11.799854] ignition[991]: disks: createPartitions: op(2): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   11.799866] ignition[991]: disks: createPartitions: op(2): op(3): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.812469] ignition[991]: disks: createPartitions: op(2): op(3): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.812557] ignition[991]: disks: createPartitions: op(2): op(4): [started]  reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.813180] ignition[991]: disks: createPartitions: op(2): op(4): [finished] reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.813194] ignition[991]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=1:0:+0 /run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   12.817938] ignition[991]: disks: createPartitions: op(2): running sgdisk with options: [--new=1:0:+0 --change-name=1:var /run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   12.817956] ignition[991]: disks: createPartitions: op(2): op(5): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   13.833635]  sdb: sdb1

[   13.835081] ignition[991]: disks: createPartitions: op(2): op(5): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   13.835148] ignition[991]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   13.835164] ignition[991]: disks: createFilesystems: op(6): [started]  waiting for devices [/dev/disk/by-partlabel/var]

...

[  104.004394] systemd[1]: Timed out waiting for device dev-disk-by\x2dpartlabel-var.device.

[  104.006415] ignition[991]: disks: createFilesystems: op(6): [failed]   waiting for devices [/dev/disk/by-partlabel/var]: device unit dev-disk-by\x2dpartlabel-var.device timeout

[  104.006446] ignition[991]: disks failed

[  104.006456] ignition[991]: Full config:

[  104.006463] ignition[991]: {

[  104.006469] ignition[991]:   "ignition": {

[  104.006474] ignition[991]:     "config": {

[  104.006480] ignition[991]:       "replace": {

[  104.006486] ignition[991]:         "verification": {}

[  104.006491] ignition[991]:       }

[  104.006498] ignition[991]:     },

[  104.006504] ignition[991]:     "proxy": {},

[  104.006509] ignition[991]:     "security": {

[  104.006515] ignition[991]:       "tls": {}

[  104.006520] ignition[991]:     },

[  104.006525] ignition[991]:     "timeouts": {},

[  104.006531] ignition[991]:     "version": "3.4.0-experimental"

[  104.006537] ignition[991]:   },

[  104.006544] ignition[991]:   "kernelArguments": {},

[  104.006549] ignition[991]:   "passwd": {

[  104.006555] ignition[991]:     "users": [

[  104.006560] ignition[991]:       {

[  104.006566] ignition[991]:         "gecos": "CoreOS Admin",

[  104.006573] ignition[991]:         "groups": [

[  104.006578] ignition[991]:           "adm",

[  104.006583] ignition[991]:           "sudo",

[  104.006589] ignition[991]:           "systemd-journal",

[  104.006594] ignition[991]:           "wheel"

[  104.006600] ignition[991]:         ],

[  104.006605] ignition[991]:         "name": "core",

[  104.006611] ignition[991]:         "sshAuthorizedKeys": [

[  104.006617] ignition[991]:           "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo/diWisyUW298OhkifoDPEDuMz62CJbo+fO7mEqdYBPldLszmKhvsG4oiIXvR30z74w3M8C497LA+ekcd7+Ho= core@default"

[  104.006624] ignition[991]:         ]

[  104.006630] ignition[991]:       }

[  104.006636] ignition[991]:     ]

[  104.006641] ignition[991]:   },

[  104.006652] ignition[991]:   "storage": {

[  104.006658] ignition[991]:     "disks": [

[  104.006665] ignition[991]:       {

[  104.006670] ignition[991]:         "device": "/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1",

[  104.006676] ignition[991]:         "partitions": [

[  104.006681] ignition[991]:           {

[  104.006687] ignition[991]:             "label": "var",

[  104.006693] ignition[991]:             "number": 1

[  104.006698] ignition[991]:           }

[  104.006704] ignition[991]:         ],

[  104.006709] ignition[991]:         "wipeTable": true

[  104.006715] ignition[991]:       }

[  104.006721] ignition[991]:     ],

[  104.006727] ignition[991]:     "filesystems": [

[  104.006732] ignition[991]:       {

[  104.006737] ignition[991]:         "device": "/dev/disk/by-partlabel/var",

[  104.006743] ignition[991]:         "format": "xfs",

[  104.006749] ignition[991]:         "label": "var",

[  104.006754] ignition[991]:         "path": "/var",

[  104.006760] ignition[991]:         "wipeFilesystem": true

[  104.006765] ignition[991]:       }

[  104.006771] ignition[991]:     ]

[  104.006777] ignition[991]:   },

[  104.006782] ignition[991]:   "systemd": {

[  104.006788] ignition[991]:     "units": [

[  104.006793] ignition[991]:       {

[  104.006800] ignition[991]:         "contents": "# Generated by Butane\n[Unit]\nBefore=local-fs.target\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var.service\n\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nType=xfs\n\n[Install]\nRequiredBy=local-fs.target",

[  104.006808] ignition[991]:         "enabled": true,

[  104.006813] ignition[991]:         "name": "var.mount"

[  104.006819] ignition[991]:       },

[  104.006824] ignition[991]:       {

[  104.006831] ignition[991]:         "contents": "[Unit]\n[Service]\nRemainAfterExit=yes\nEnvironmentFile=-/run/kola-runext-env\nEnvironment=KOLA_UNIT=kola-runext.service\nEnvironment=KOLA_TEST=ext.config.shared.var-mount.scsi-id\nEnvironment=KOLA_TEST_EXE=test.sh\nEnvironment=KOLA_EXT_DATA=/var/opt/kola/extdata\nExecStart=/usr/local/bin/kola-runext-test.sh\n",

[  104.006879] ignition[991]:         "enabled": false,

[  104.006884] ignition[991]:         "mask": false,

[  104.006890] ignition[991]:         "name": "kola-runext.service"

[  104.006896] ignition[991]:       }

[  104.006901] ignition[991]:     ]

[  104.006907] ignition[991]:   }

[  104.006917] ignition[991]: Ignition failed: failed to create filesystems: failed to wait on filesystems devs: device unit dev-disk-by\x2dpartlabel-var.device timeout

[  104.006925] ignition[991]: }

[  104.006988] systemd[1]: dev-disk-by\x2dpartlabel-var.device: Job dev-disk-by\x2dpartlabel-var.device/start failed with result 'timeout'.

[  104.007116] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE

[  104.007365] systemd[1]: ignition-disks.service: Failed with result 'exit-code'.

[  104.007576] systemd[1]: Failed to start Ignition (disks).

Similiar behaviour with stable boot, here it is timing out on creating the partition

Details

[   10.978877] systemd[1]: Starting Ignition (disks)...

[   11.047858] ignition[892]: Ignition 2.13.0

[   11.048033] ignition[892]: Stage: disks

[   11.048209] ignition[892]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"

[   11.050806] ignition[892]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"

[   11.052538] ignition[892]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/coreos-boot-disk]
...

[  101.092453] systemd[1]: Timed out waiting for device dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device.

[  101.094054] ignition[892]: disks: createPartitions: op(1): [failed]   waiting for devices [/dev/disk/by-id/coreos-boot-disk]: device unit dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device timeout

[  101.094077] ignition[892]: disks failed

[  101.094087] ignition[892]: Full config:

[  101.094093] ignition[892]: {

[  101.094103] ignition[892]:   "ignition": {

[  101.094109] ignition[892]:     "config": {

[  101.094114] ignition[892]:       "replace": {

[  101.094120] ignition[892]:         "verification": {}

[  101.094126] ignition[892]:       }

[  101.094132] ignition[892]:     },

[  101.094139] ignition[892]:     "proxy": {},

[  101.094144] ignition[892]:     "security": {

[  101.094150] ignition[892]:       "tls": {}

[  101.094155] ignition[892]:     },

[  101.094162] ignition[892]:     "timeouts": {},

[  101.094167] ignition[892]:     "version": "3.4.0-experimental"

[  101.094173] ignition[892]:   },

[  101.094179] ignition[892]:   "kernelArguments": {},

[  101.094184] ignition[892]:   "passwd": {

[  101.094190] ignition[892]:     "users": [

[  101.094195] ignition[892]:       {

[  101.094200] ignition[892]:         "gecos": "CoreOS Admin",

[  101.094206] ignition[892]:         "groups": [

[  101.094212] ignition[892]:           "adm",

[  101.094218] ignition[892]:           "sudo",

[  101.094224] ignition[892]:           "systemd-journal",

[  101.094229] ignition[892]:           "wheel"

[  101.094235] ignition[892]:         ],

[  101.094240] ignition[892]:         "name": "core",

[  101.094246] ignition[892]:         "sshAuthorizedKeys": [

[  101.094252] ignition[892]:           "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo/diWisyUW298OhkifoDPEDuMz62CJbo+fO7mEqdYBPldLszmKhvsG4oiIXvR30z74w3M8C497LA+ekcd7+Ho= core@default"

[  101.094258] ignition[892]:         ]

[  101.094264] ignition[892]:       }

[  101.094269] ignition[892]:     ]

[  101.094275] ignition[892]:   },

[  101.094281] ignition[892]:   "storage": {

[  101.094286] ignition[892]:     "disks": [

[  101.094291] ignition[892]:       {

[  101.094296] ignition[892]:         "device": "/dev/disk/by-id/coreos-boot-disk",

[  101.094302] ignition[892]:         "partitions": [

[  101.094307] ignition[892]:           {

[  101.094313] ignition[892]:             "label": "toor",

[  101.094320] ignition[892]:             "number": 5,

[  101.094326] ignition[892]:             "sizeMiB": 1024

[  101.094331] ignition[892]:           }

[  101.094336] ignition[892]:         ],

[  101.094342] ignition[892]:         "wipeTable": false

[  101.094347] ignition[892]:       }

[  101.094352] ignition[892]:     ],

[  101.094357] ignition[892]:     "filesystems": [

[  101.094363] ignition[892]:       {

[  101.094368] ignition[892]:         "device": "/dev/disk/by-partlabel/toor",

[  101.094375] ignition[892]:         "format": "ext4",

[  101.094380] ignition[892]:         "path": "/var/lib/toor"

[  101.094386] ignition[892]:       }

[  101.094391] ignition[892]:     ]

[  101.094396] ignition[892]:   },

[  101.094401] ignition[892]:   "systemd": {

[  101.094406] ignition[892]:     "units": [

[  101.094412] ignition[892]:       {

[  101.094419] ignition[892]:         "contents": "# Generated by Butane\n[Unit]\nBefore=local-fs.target\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-toor.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-toor.service\n\n[Mount]\nWhere=/var/lib/toor\nWhat=/dev/disk/by-partlabel/toor\nType=ext4\n\n[Install]\nRequiredBy=local-fs.target",

[  101.094426] ignition[892]:         "enabled": true,

[  101.094433] ignition[892]:         "name": "var-lib-toor.mount"

[  101.094439] ignition[892]:       },

[  101.094444] ignition[892]:       {

[  101.094454] ignition[892]:         "contents": "[Unit]\n[Service]\nRemainAfterExit=yes\nEnvironmentFile=-/run/kola-runext-env\nEnvironment=KOLA_UNIT=kola-runext.service\nEnvironment=KOLA_TEST=ext.config.shared.ignition.stable-boot\nEnvironment=KOLA_TEST_EXE=test.sh\nEnvironment=KOLA_EXT_DATA=/var/opt/kola/extdata\nExecStart=/usr/local/bin/kola-runext-test.sh\n",

[  101.094499] ignition[892]:         "enabled": false,

[  101.094506] ignition[892]:         "mask": false,

[  101.094512] ignition[892]:         "name": "kola-runext.service"

[  101.094517] ignition[892]:       }

[  101.094523] ignition[892]:     ]

[  101.094528] ignition[892]:   }

[  101.094537] ignition[892]: Ignition failed: create partitions failed: failed to wait on disks devs: device unit dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device timeout

[  101.094545] ignition[892]: }

[  101.094598] systemd[1]: dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device: Job dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device/start failed with result 'timeout'.

[  101.094717] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE

[  101.094848] systemd[1]: ignition-disks.service: Failed with result 'exit-code'.

[  101.095069] systemd[1]: Failed to start Ignition (disks).

Both have passed in fcos when i checked it in order to skip the tests.
kola_artifacts.zip

[jschintag]

I have created a seperate issue for those 2 tests, as they weren't included in the initial report and i also already divided and conquered the rest of the failing tests.
#710

[jschintag]

I would say we can close this issue as we have seperate issues for all problems that cropped up and with the tests disabled/snoozed/fixed we have successfully build 4.11

[miabbott]

/close

We're tracking the individual failures in separate issues now

[openshift-ci]

@miabbott: Closing this issue.

Details

In response to this:

/close

We're tracking the individual failures in separate issues now

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.