RHCOS 4.11 s390x: Failling disk tests

[jschintag]

The following tests are failling in the ignition disks stage and boot to emergency shell:

• ext.config.shared.ignition.stable-boot (new)
• ext.config.shared.var-mount.scsi-id

The scis-id seems to fail to find the filesystem on the device:

Details

[   11.686561] systemd[1]: Starting Ignition (disks)...

[   11.763685] ignition[991]: Ignition 2.13.0

[   11.763850] ignition[991]: Stage: disks

[   11.764211] ignition[991]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"

[   11.773878] ignition[991]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"

[   11.777721] ignition[991]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   11.799745] ignition[991]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   11.799815] ignition[991]: disks: createPartitions: created device alias for "/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1": "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1" -> "/dev/sdb"

[   11.799833] ignition[991]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   11.799844] ignition[991]: disks: createPartitions: op(2): wiping partition table requested on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   11.799854] ignition[991]: disks: createPartitions: op(2): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   11.799866] ignition[991]: disks: createPartitions: op(2): op(3): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.812469] ignition[991]: disks: createPartitions: op(2): op(3): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.812557] ignition[991]: disks: createPartitions: op(2): op(4): [started]  reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.813180] ignition[991]: disks: createPartitions: op(2): op(4): [finished] reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   12.813194] ignition[991]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=1:0:+0 /run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   12.817938] ignition[991]: disks: createPartitions: op(2): running sgdisk with options: [--new=1:0:+0 --change-name=1:var /run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1]

[   12.817956] ignition[991]: disks: createPartitions: op(2): op(5): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   13.833635]  sdb: sdb1

[   13.835081] ignition[991]: disks: createPartitions: op(2): op(5): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   13.835148] ignition[991]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1"

[   13.835164] ignition[991]: disks: createFilesystems: op(6): [started]  waiting for devices [/dev/disk/by-partlabel/var]

...

[  104.004394] systemd[1]: Timed out waiting for device dev-disk-by\x2dpartlabel-var.device.

[  104.006415] ignition[991]: disks: createFilesystems: op(6): [failed]   waiting for devices [/dev/disk/by-partlabel/var]: device unit dev-disk-by\x2dpartlabel-var.device timeout

[  104.006446] ignition[991]: disks failed

[  104.006456] ignition[991]: Full config:

[  104.006463] ignition[991]: {

[  104.006469] ignition[991]:   "ignition": {

[  104.006474] ignition[991]:     "config": {

[  104.006480] ignition[991]:       "replace": {

[  104.006486] ignition[991]:         "verification": {}

[  104.006491] ignition[991]:       }

[  104.006498] ignition[991]:     },

[  104.006504] ignition[991]:     "proxy": {},

[  104.006509] ignition[991]:     "security": {

[  104.006515] ignition[991]:       "tls": {}

[  104.006520] ignition[991]:     },

[  104.006525] ignition[991]:     "timeouts": {},

[  104.006531] ignition[991]:     "version": "3.4.0-experimental"

[  104.006537] ignition[991]:   },

[  104.006544] ignition[991]:   "kernelArguments": {},

[  104.006549] ignition[991]:   "passwd": {

[  104.006555] ignition[991]:     "users": [

[  104.006560] ignition[991]:       {

[  104.006566] ignition[991]:         "gecos": "CoreOS Admin",

[  104.006573] ignition[991]:         "groups": [

[  104.006578] ignition[991]:           "adm",

[  104.006583] ignition[991]:           "sudo",

[  104.006589] ignition[991]:           "systemd-journal",

[  104.006594] ignition[991]:           "wheel"

[  104.006600] ignition[991]:         ],

[  104.006605] ignition[991]:         "name": "core",

[  104.006611] ignition[991]:         "sshAuthorizedKeys": [

[  104.006617] ignition[991]:           "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo/diWisyUW298OhkifoDPEDuMz62CJbo+fO7mEqdYBPldLszmKhvsG4oiIXvR30z74w3M8C497LA+ekcd7+Ho= core@default"

[  104.006624] ignition[991]:         ]

[  104.006630] ignition[991]:       }

[  104.006636] ignition[991]:     ]

[  104.006641] ignition[991]:   },

[  104.006652] ignition[991]:   "storage": {

[  104.006658] ignition[991]:     "disks": [

[  104.006665] ignition[991]:       {

[  104.006670] ignition[991]:         "device": "/dev/disk/by-id/scsi-0NVME_VirtualMultipath_disk1",

[  104.006676] ignition[991]:         "partitions": [

[  104.006681] ignition[991]:           {

[  104.006687] ignition[991]:             "label": "var",

[  104.006693] ignition[991]:             "number": 1

[  104.006698] ignition[991]:           }

[  104.006704] ignition[991]:         ],

[  104.006709] ignition[991]:         "wipeTable": true

[  104.006715] ignition[991]:       }

[  104.006721] ignition[991]:     ],

[  104.006727] ignition[991]:     "filesystems": [

[  104.006732] ignition[991]:       {

[  104.006737] ignition[991]:         "device": "/dev/disk/by-partlabel/var",

[  104.006743] ignition[991]:         "format": "xfs",

[  104.006749] ignition[991]:         "label": "var",

[  104.006754] ignition[991]:         "path": "/var",

[  104.006760] ignition[991]:         "wipeFilesystem": true

[  104.006765] ignition[991]:       }

[  104.006771] ignition[991]:     ]

[  104.006777] ignition[991]:   },

[  104.006782] ignition[991]:   "systemd": {

[  104.006788] ignition[991]:     "units": [

[  104.006793] ignition[991]:       {

[  104.006800] ignition[991]:         "contents": "# Generated by Butane\n[Unit]\nBefore=local-fs.target\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var.service\n\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nType=xfs\n\n[Install]\nRequiredBy=local-fs.target",

[  104.006808] ignition[991]:         "enabled": true,

[  104.006813] ignition[991]:         "name": "var.mount"

[  104.006819] ignition[991]:       },

[  104.006824] ignition[991]:       {

[  104.006831] ignition[991]:         "contents": "[Unit]\n[Service]\nRemainAfterExit=yes\nEnvironmentFile=-/run/kola-runext-env\nEnvironment=KOLA_UNIT=kola-runext.service\nEnvironment=KOLA_TEST=ext.config.shared.var-mount.scsi-id\nEnvironment=KOLA_TEST_EXE=test.sh\nEnvironment=KOLA_EXT_DATA=/var/opt/kola/extdata\nExecStart=/usr/local/bin/kola-runext-test.sh\n",

[  104.006879] ignition[991]:         "enabled": false,

[  104.006884] ignition[991]:         "mask": false,

[  104.006890] ignition[991]:         "name": "kola-runext.service"

[  104.006896] ignition[991]:       }

[  104.006901] ignition[991]:     ]

[  104.006907] ignition[991]:   }

[  104.006917] ignition[991]: Ignition failed: failed to create filesystems: failed to wait on filesystems devs: device unit dev-disk-by\x2dpartlabel-var.device timeout

[  104.006925] ignition[991]: }

[  104.006988] systemd[1]: dev-disk-by\x2dpartlabel-var.device: Job dev-disk-by\x2dpartlabel-var.device/start failed with result 'timeout'.

[  104.007116] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE

[  104.007365] systemd[1]: ignition-disks.service: Failed with result 'exit-code'.

[  104.007576] systemd[1]: Failed to start Ignition (disks).

Similiar behaviour with stable boot, here it is timing out on creating the partition

Details

[   10.978877] systemd[1]: Starting Ignition (disks)...

[   11.047858] ignition[892]: Ignition 2.13.0

[   11.048033] ignition[892]: Stage: disks

[   11.048209] ignition[892]: reading system config file "/usr/lib/ignition/base.d/00-core.ign"

[   11.050806] ignition[892]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"

[   11.052538] ignition[892]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/coreos-boot-disk]
...

[  101.092453] systemd[1]: Timed out waiting for device dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device.

[  101.094054] ignition[892]: disks: createPartitions: op(1): [failed]   waiting for devices [/dev/disk/by-id/coreos-boot-disk]: device unit dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device timeout

[  101.094077] ignition[892]: disks failed

[  101.094087] ignition[892]: Full config:

[  101.094093] ignition[892]: {

[  101.094103] ignition[892]:   "ignition": {

[  101.094109] ignition[892]:     "config": {

[  101.094114] ignition[892]:       "replace": {

[  101.094120] ignition[892]:         "verification": {}

[  101.094126] ignition[892]:       }

[  101.094132] ignition[892]:     },

[  101.094139] ignition[892]:     "proxy": {},

[  101.094144] ignition[892]:     "security": {

[  101.094150] ignition[892]:       "tls": {}

[  101.094155] ignition[892]:     },

[  101.094162] ignition[892]:     "timeouts": {},

[  101.094167] ignition[892]:     "version": "3.4.0-experimental"

[  101.094173] ignition[892]:   },

[  101.094179] ignition[892]:   "kernelArguments": {},

[  101.094184] ignition[892]:   "passwd": {

[  101.094190] ignition[892]:     "users": [

[  101.094195] ignition[892]:       {

[  101.094200] ignition[892]:         "gecos": "CoreOS Admin",

[  101.094206] ignition[892]:         "groups": [

[  101.094212] ignition[892]:           "adm",

[  101.094218] ignition[892]:           "sudo",

[  101.094224] ignition[892]:           "systemd-journal",

[  101.094229] ignition[892]:           "wheel"

[  101.094235] ignition[892]:         ],

[  101.094240] ignition[892]:         "name": "core",

[  101.094246] ignition[892]:         "sshAuthorizedKeys": [

[  101.094252] ignition[892]:           "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo/diWisyUW298OhkifoDPEDuMz62CJbo+fO7mEqdYBPldLszmKhvsG4oiIXvR30z74w3M8C497LA+ekcd7+Ho= core@default"

[  101.094258] ignition[892]:         ]

[  101.094264] ignition[892]:       }

[  101.094269] ignition[892]:     ]

[  101.094275] ignition[892]:   },

[  101.094281] ignition[892]:   "storage": {

[  101.094286] ignition[892]:     "disks": [

[  101.094291] ignition[892]:       {

[  101.094296] ignition[892]:         "device": "/dev/disk/by-id/coreos-boot-disk",

[  101.094302] ignition[892]:         "partitions": [

[  101.094307] ignition[892]:           {

[  101.094313] ignition[892]:             "label": "toor",

[  101.094320] ignition[892]:             "number": 5,

[  101.094326] ignition[892]:             "sizeMiB": 1024

[  101.094331] ignition[892]:           }

[  101.094336] ignition[892]:         ],

[  101.094342] ignition[892]:         "wipeTable": false

[  101.094347] ignition[892]:       }

[  101.094352] ignition[892]:     ],

[  101.094357] ignition[892]:     "filesystems": [

[  101.094363] ignition[892]:       {

[  101.094368] ignition[892]:         "device": "/dev/disk/by-partlabel/toor",

[  101.094375] ignition[892]:         "format": "ext4",

[  101.094380] ignition[892]:         "path": "/var/lib/toor"

[  101.094386] ignition[892]:       }

[  101.094391] ignition[892]:     ]

[  101.094396] ignition[892]:   },

[  101.094401] ignition[892]:   "systemd": {

[  101.094406] ignition[892]:     "units": [

[  101.094412] ignition[892]:       {

[  101.094419] ignition[892]:         "contents": "# Generated by Butane\n[Unit]\nBefore=local-fs.target\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-toor.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-toor.service\n\n[Mount]\nWhere=/var/lib/toor\nWhat=/dev/disk/by-partlabel/toor\nType=ext4\n\n[Install]\nRequiredBy=local-fs.target",

[  101.094426] ignition[892]:         "enabled": true,

[  101.094433] ignition[892]:         "name": "var-lib-toor.mount"

[  101.094439] ignition[892]:       },

[  101.094444] ignition[892]:       {

[  101.094454] ignition[892]:         "contents": "[Unit]\n[Service]\nRemainAfterExit=yes\nEnvironmentFile=-/run/kola-runext-env\nEnvironment=KOLA_UNIT=kola-runext.service\nEnvironment=KOLA_TEST=ext.config.shared.ignition.stable-boot\nEnvironment=KOLA_TEST_EXE=test.sh\nEnvironment=KOLA_EXT_DATA=/var/opt/kola/extdata\nExecStart=/usr/local/bin/kola-runext-test.sh\n",

[  101.094499] ignition[892]:         "enabled": false,

[  101.094506] ignition[892]:         "mask": false,

[  101.094512] ignition[892]:         "name": "kola-runext.service"

[  101.094517] ignition[892]:       }

[  101.094523] ignition[892]:     ]

[  101.094528] ignition[892]:   }

[  101.094537] ignition[892]: Ignition failed: create partitions failed: failed to wait on disks devs: device unit dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device timeout

[  101.094545] ignition[892]: }

[  101.094598] systemd[1]: dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device: Job dev-disk-by\x2did-coreos\x2dboot\x2ddisk.device/start failed with result 'timeout'.

[  101.094717] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE

[  101.094848] systemd[1]: ignition-disks.service: Failed with result 'exit-code'.

[  101.095069] systemd[1]: Failed to start Ignition (disks).

Both have passed in fcos when i checked it in order to skip the tests.
kola_artifacts.zip

Initial issue:
#703

[miabbott]

@sohankunkerkar could have a look at these failures and see if you can triage them further?

[sohankunkerkar]

@sohankunkerkar could have a look at these failures and see if you can triage them further?

Sure, I will look into them

[nikita-dubrovskii]

Did initial check on kvm running rhcos-4.11.s390x. For some reason partition label is missing during udev:

DEVNAME=/dev/vda3
DEVPATH=/devices/css0/0.0.0001/0.0.0001/virtio1/block/vda/vda3
DEVTYPE=partition
MAJOR=252
MINOR=3
PARTN=3
SUBSYSTEM=block

Running same test on fcos-35.s390x succeeds:

DEVPATH=/devices/css0/0.0.0003/0.0.0003/virtio5/block/vda/vda3
DEVNAME=/dev/vda3
DEVTYPE=partition
DISKSEQ=1
PARTN=3
PARTNAME=boot
MAJOR=252
MINOR=3
SUBSYSTEM=block

[jschintag]

I tested it on a RHEL 8.6 based RHCOS 4.11 and ext.config.shared.var-mount.scsi-id passed, while ext.config.shared.ignition.stable-boot still fails.

[nikita-dubrovskii]

https://bugzilla.redhat.com/show_bug.cgi?id=2075085

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[jschintag]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale