cloud-init status --wait never returns if datasource not found

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1966085

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Bionic)', 'cloud-init (Ubuntu Focal)', 'cloud-init (Ubuntu Impish)', 'cloud-init (Ubuntu Jammy)']
assignee = falcojr
assignee_name = James Falcon
date_closed = 2022-03-24T16:21:36.503591+00:00
date_created = 2022-03-23T14:02:18.034768+00:00
date_fix_committed = 2022-03-24T02:25:16.032161+00:00
date_fix_released = 2022-03-24T16:21:36.503591+00:00
id = 1966085
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1966085
milestone = None
owner = falcojr
owner_name = James Falcon
private = False
status = fix_released
submitter = falcojr
submitter_name = James Falcon
tags = ['regression-update', 'verification-done-bionic', 'verification-done-focal', 'verification-done-impish']
duplicates = []

Launchpad user James Falcon(falcojr) wrote on 2022-03-23T14:02:18.034768+00:00

[Impact]

 * In environments where cloud-init-generator detects no viable datasources, cloud-init systemd units are not included in the boot process and cloud-init status --wait blocks indefinitely reporting "not run" state instead of "disabled". Utilities which rely on the blocking status will sit indefinitely as cloud-init will never run in these scenarios.

This upload ensures that the expected artifact of cloud-init-generator emits a /run/cloud-init/disabled file (which cloud-init status --wait expects to see when coud-init is disabled at generator timeframe) which allows cloud-init status --wait to return expected "disabled" status.

[Test Plan]

 * this scenario is reproducible by disabling LXD nocloud metadata templates and security.devlxd=false configuration which will allow cloud-init-generator to find no viable datasources during systemd generator time frame. Cloud-init status --wait in this case should not block, but return 'disabled' status immediately after systemd generator is run.

Procedure
 * lxc launch ubuntu-daily: test-series
 # To avoid LXD datasource detection don't allow /dev/lxd/sock in instance
 * lxc config set test-<series< security.devlxd=false
 # Remove nocloud-net cloud-init instance metadata to avoid NoCloud
 * lxc exec test- -- rm -rf /var/lib/cloud/seed/nocloud-net
 # clean reboot of cloud-init to ensure generator detection
 * lxc exec test- -- cloud-init clean --logs --reboot

 # make sure cloud-init status --wait doesn't block indefiinitely in this case

 * lxc exec tests- -- cloud-init status --wait

 This test procedure is captured by cloud-init integration test
https://github.com/canonical/cloud-init/blob/main/tests/integration_tests/cmd/test_status.py

It can be run with the following:

for $SERIES in bionic focal impish; do
  CLOUD_INIT_PLATFORM=lxd_container CLOUD_INIT_CLOUD_INIT_SOURCE=PROPOSED CLOUD_INIT_OS_IMAGE=$SERIES tox -e integration-tests tests/integration_tests/cmd/test_status.py
done

[Where problems could occur]

 * Problems can occur if cloud-init early exits from cloud-init status --wait without blocking if 3rd party apps/scripts which expect to block until cloud-init completes start running before cloud-init is "done" could could lead to unexpected system configuration or apt-lock issues if scripts attempt to install packages

[Other Info]

[Original Description]

Calling "cloud-init status --wait" never returns if ds-identify cannot find a datasource.

In #1162, we modified status checks to wait until we get an "enabled" or "disabled" file from ds-identiy. ds-identify never outputs a "disabled" file, so "status --wait" will wait indefinitely if no datasource is found.

A simple reproduction is to change "ret=1" on this line:
https://github.com/canonical/cloud-init/blob/main/tools/ds-identify#L1866

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-03-23T22:28:27.262247+00:00

Upstream fix for this posted to #1349

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-03-24T02:25:09.867962+00:00

Upstream fix merged at eee6032

[ubuntu-server-builder]

Launchpad user Christian Ehrhardt (paelzer) wrote on 2022-03-24T08:44:22.802572+00:00

Backports to Ubuntu releases started for Jammy at #1350

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-03-24T16:21:59.241483+00:00

Fix released to Jammy [ubuntu/jammy-proposed] cloud-init 22.1-14-g2e17a0d6-0ubuntu1~22.04.4 (Accepted)

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2022-03-24T20:29:15.548903+00:00

Hello James, or anyone else affected,

Accepted cloud-init into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/22.1-14-g2e17a0d6-0ubuntu1~21.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2022-03-24T20:31:32.949080+00:00

Hello James, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/22.1-14-g2e17a0d6-0ubuntu1~20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2022-03-24T20:32:53.944398+00:00

Hello James, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/22.1-14-g2e17a0d6-0ubuntu1~18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-03-24T21:51:54.160681+00:00

Uploads queued for -proposed testing on bionic focal and impish: verification will start once -proposed are published and accessible

$ rmadison cloud-init | grep proposed
cloud-init | 22.1-14-g2e17a0d6-0ubuntu118.04.2 | bionic-proposed | all
cloud-init | 22.1-14-g2e17a0d6-0ubuntu118.04.3 | bionic-proposed | source
cloud-init | 22.1-14-g2e17a0d6-0ubuntu120.04.2 | focal-proposed | all
cloud-init | 22.1-14-g2e17a0d6-0ubuntu120.04.3 | focal-proposed | source
cloud-init | 22.1-14-g2e17a0d6-0ubuntu121.10.2 | impish-proposed | all
cloud-init | 22.1-14-g2e17a0d6-0ubuntu121.10.3 | impish-proposed | source

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2022-03-24T22:16:52.879945+00:00

Verification on Bionic
Launchpad attachments: lp_1966085_bionic.txt

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2022-03-24T22:16:58.961790+00:00

Verification on Focal
Launchpad attachments: lp_1966085_focal.txt

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2022-03-24T22:17:05.205487+00:00

Verification on Impish
Launchpad attachments: lp_1966085_impish.txt

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-03-24T22:56:33.845083+00:00

Supplemental full integration test suite on bionic lxd-container to ensure no other functionality broke
Launchpad attachments: sru-22.1-lxd-container-full-tests.log

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2022-03-25T00:34:24.984979+00:00

sru-22.1-lxd-container-focal.log SUCCESS logs for full integration tests on lxd-container to ensure we didn't break anything outside of this minor changeset.
Launchpad attachments: sru-22.1-lxd-container-focal.log