ds-identify fails to detect NoCloud datastore with LABEL_FATBOOT instead of LABEL (change introduced recently in util-linux-2.33-rc1)

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1841466

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = 2020-08-25T19:31:08.570254+00:00
date_created = 2019-08-26T19:24:16.946096+00:00
date_fix_committed = 2020-08-25T19:31:08.570254+00:00
date_fix_released = 2020-08-25T19:31:08.570254+00:00
id = 1841466
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1841466
milestone = None
owner = flipsa
owner_name = Hans Dampf
private = False
status = fix_released
submitter = flipsa
submitter_name = Hans Dampf
tags = []
duplicates = []

Launchpad user Hans Dampf(flipsa) wrote on 2019-08-26T19:24:16.946096+00:00

Original bug report with detailed description was created for Xen Orchestra here: vatesfr/xen-orchestra#4449

Brief description:

On systems with util-linux-2.33-rc1 or younger (e.g. Debian 10 Buster), ds-identify fails to detect when a disk of a NoCloud datasource has a label written to the boot sector of the disk. Before util-linux-2.33-rc1, blkid showed "LABEL=cidata". With the change, blkid shows "LABEL_FATBOOT=cidata" (newly introduced / additional label).

I ran into this when using cloud-init together with Xen Orchestra v5.48 (Xen Orchestra is a management interface for xen; in my case xcp-ng v8.0.0). I created a VM template based on the recently released Debian 10.0 Buster, which uses util-linux 2.33.1. Upon boot, ds-identify fails to detect the NoCloud datasource / virtual disk which Xen Orchestra generated (the disk is created with this code from https://github.com/natevw/fatfs. With an older Debian 8 (util-linux-2.25.0) based template, ds-identify detects the NoCloud datasource disk fine.

Longer description / likely cause of the problem:

Xen Orchestra creates the NoCloud as a partition-less disk with a FAT16 filesystem which has the NoCloud user-data and meta-data files. The label "cidata" is written into the boot sector of the virtual disk. With the same disk, older versions of blkid report "LABEL=cidata" whereas newer versions detect "LABEL_FATBOOT=cidata". The ds-identify shell script checks only for the presence of the field called "LABEL" and not for "LABEL_FATBOOT".

Relevant commit message from the util-linux-2.33-rc1 changelog (commit f0ca7e80d7a171701d0d04a3eae22d97f15d0683): https://cdn.kernel.org/pub/linux/utils/util-linux/v2.33/v2.33-rc1-ChangeLog

libblkid: vfat: Change parsing label in special cases

• Use only label from the root directory and do not fallback to the label stored in boot sector. This is how MS-DOS 6.22, MS-DOS 7.10, Windows 98, Windows XP and also Windows 10 behave. Moreover Windows XP and Windows 10 do not touch label in boot sector anymore, so removing FAT label on those Windowses leads to having old label still stored in boot sector (which MS-DOS and Windows fully ignore).

• Label entry "NO NAME" in root directory is treated as label "NO NAME" instead of empty label. In root directory it has no special meaning. String "NO NAME" has a special meaning (empty label) only for label stored in boot sector.

 * Label from the boot sector is now stored into LABEL_FATBOOT field. So if there are applications which depends or needs to read this label, they have ability.

• After this change LABEL always correspondent to the label from the root directory and LABEL_FATBOOT to the label stored in the boot sector. If some of those labels is missing or is not present (e.g. "NO LABEL" in boot sector) then particular field is not set.

Possible fix:

I did a trivial change of 2 lines to ds-identify to check for LABEL_FATBOOT after the check for LABEL:

debian@cloudbuster:~$ diff /usr/lib/cloud-init/ds-identify.ORIGINAL /usr/lib/cloud-init/ds-identify
236a237,238

LABEL_FATBOOT=*) label="${line#LABEL_FATBOOT=}";
labels="${labels}${line#LABEL_FATBOOT=}${delim}";;

For me this solves the problem, as in: the cloud-init enabled VM boots up, ds-identify finds "LABEL_FATBOOT=cidata" and cloud-init correctly executes. In cases where both labels are written, the latter over-writes the former, which could be a theoretical problem if the values differ, but I am not sure how likely this case is.

Further debug information as requested by @rharper on IRC:

• cloud-init.tar.gz (Debian 10 / ds-identify fail)

• Debian version:

debian@cloudbuster:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

• util-linux version:

debian@cloudbuster:~$ sudo blkid -V
blkid from util-linux 2.33.1 (libblkid 2.33.1, 09-Jan-2019)

• blkid output:

debian@cloudbuster:~$ sudo blkid /dev/xvdb
/dev/xvdb: SEC_TYPE="msdos" LABEL_FATBOOT="cidata" UUID="355A-4FC2" TYPE="vfat"

• udevadm outout:
  debian@cloudbuster:~$ udevadm info --query=all /sys/class/block/xvdb
  P: /devices/vbd-832/block/xvdb
  N: xvdb
  L: 0
  S: disk/by-uuid/355A-4FC2
  E: DEVPATH=/devices/vbd-832/block/xvdb
  E: DEVNAME=/dev/xvdb
  E: DEVTYPE=disk
  E: MAJOR=202
  E: MINOR=16
  E: SUBSYSTEM=block
  E: USEC_INITIALIZED=4239917
  E: ID_FS_UUID=355A-4FC2
  E: ID_FS_UUID_ENC=355A-4FC2
  E: ID_FS_VERSION=FAT16
  E: ID_FS_TYPE=vfat
  E: ID_FS_USAGE=filesystem
  E: DEVLINKS=/dev/disk/by-uuid/355A-4FC2
  E: TAGS=:systemd:

Some experiments:

• This is interesting - dosfslabel incorrectly reports the label, while blkid (above) clearly shows the field is empty / not set:

debian@cloudbuster:~$ sudo dosfslabel /dev/xvdb
cidata

• Here I am first setting the label with dosfslabel to see what happens and then check blkid again:

debian@cloudbuster:~$ sudo dosfslabel /dev/xvdb cidata
fatlabel: warning - lowercase labels might not work properly with DOS or Windows

debian@cloudbuster:~$ sudo blkid /dev/xvdb
/dev/xvdb: SEC_TYPE="msdos" LABEL_FATBOOT="cidata" LABEL="cidata" UUID="355A-4FC2" TYPE="vfat"

Now blkid reports both labels

[ubuntu-server-builder]

Launchpad user Hans Dampf(flipsa) wrote on 2019-08-26T19:24:16.946096+00:00

Launchpad attachments: Debian 10 cloud-init collect-logs

[ubuntu-server-builder]

Launchpad user Hans Dampf(flipsa) wrote on 2019-08-26T20:00:11.885350+00:00

Debug info from the Debian 8 system where things are working (with the identical virtual disk as used with Debian 10 above):

debian@debian:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.2 (jessie)
Release: 8.2
Codename: jessie

debian@debian:~$ sudo blkid -V
blkid from util-linux 2.25.2 (libblkid 2.25.0, 24-Oct-2014)

debian@debian:~$ udevadm info --query=all /sys/class/block/xvdb
P: /devices/vbd-832/block/xvdb
N: xvdb
S: disk/by-label/cidata
S: disk/by-uuid/355A-4FC2
E: DEVLINKS=/dev/disk/by-label/cidata /dev/disk/by-uuid/355A-4FC2
E: DEVNAME=/dev/xvdb
E: DEVPATH=/devices/vbd-832/block/xvdb
E: DEVTYPE=disk
E: ID_FS_LABEL=cidata
E: ID_FS_LABEL_ENC=cidata
E: ID_FS_TYPE=vfat
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=355A-4FC2
E: ID_FS_UUID_ENC=355A-4FC2
E: ID_FS_VERSION=FAT16
E: MAJOR=202
E: MINOR=16
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=94333

[ubuntu-server-builder]

Launchpad user Hans Dampf(flipsa) wrote on 2019-08-26T20:35:55.480390+00:00

Launchpad attachments: dd image of the generated disk (xvdb)

[ubuntu-server-builder]

Launchpad user Braiam Peguero(braiampe) wrote on 2020-04-12T17:39:06.781806+00:00

Note, that while this fixes ds-identify script, NoCloud python script also search for this label.

[ubuntu-server-builder]

Launchpad user Marllus(marlluslustosa) wrote on 2020-07-21T20:22:56.425137+00:00

The fix is quite simple. Now he will be able to recognize both "LABEL" and "LABEL_FATBOOT".
These two patches solve the problem. Both in the DataSourceNoCloud.py script and in the ds-identify.
Tested cloudinit version: 20.2
Debian 10 (Buster)

Location of each file:

/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceNoCloud.py
/usr/lib/cloud-init/ds-identify

Patches have been attached.

Waiting for inclusion as a fix in the next version of CloudInit...
Launchpad attachments: patches-fix-label-cloudinit-.tar.gz

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2020-07-21T21:17:31.085075+00:00

@Marllus,
Thanks for taking the time to put that change together.
Please follow the doc at https://github.com/canonical/cloud-init/blob/master/HACKING.rst to officially submit the change for review.

Thanks,

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2020-07-22T13:46:46.860111+00:00

#500

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2020-07-29T21:27:58.183168+00:00

#513

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2020-08-25T19:31:10.391520+00:00

This bug is believed to be fixed in cloud-init in version 20.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.