Decrease azure ephemeral resource disk wait time by johnsonshi · Pull Request #800 · canonical/cloud-init

johnsonshi · 2021-02-04T00:14:26Z

Proposed Commit Message

Azure: Support for VMs without ephemeral resource disks.

Changes:

Only merge in default Azure cloud ephemeral disk configs
during DataSourceAzure._get_data() if the ephemeral disk
exists.
DataSourceAzure.address_ephemeral_resize() (which is
invoked in DataSourceAzure.activate() should only set up
the ephemeral disk if the disk exists.

Azure VMs may or may not come with ephemeral resource disks
depending on the VM SKU. For VM SKUs that come with
ephemeral resource disks, the Azure platform guarantees that
the ephemeral resource disk is attached to the VM before
the VM is booted. For VM SKUs that do not come with
ephemeral resource disks, cloud-init currently attempts
to wait and set up a non-existent ephemeral resource
disk, which wastes boot time. It also causes disk setup
modules to fail (due to non-existent references to the
ephemeral resource disk).

udevadm settle is invoked by cloud-init very early in boot.
udevadm settle is invoked very early, before
DataSourceAzure's _get_data() and activate() methods.

Within DataSourceAzure's _get_data() and activate() methods,
the ephemeral resource disk path should exist if the
VM SKU comes with an ephemeral resource disk.
The ephemeral resource disk path should not exist if the
VM SKU does not come with an ephemeral resource disk.

LP: #1901011

Additional Context

Problem

For Azure VMs, cloud-init's DataSourceAzure.py formats and addresses ephemeral disk resizing. It does this for all Azure VM SKUs. See code and code.

The code right now waits up to 120 seconds for the ephemeral disk to appear before either proceeding or giving up. It waits up to 120 secs for the symlink /dev/disk/cloud/azure_resource to appear.

For new Azure VM SKUs (such as Dv4, Dsv4, Ev4, Esv4) that do not come with ephemeral resource disks, cloud-init would wait up to 120 seconds before giving up. See LP: #1901011.

For these new Azure VM SKUs without ephemeral resource disks, the disk_setup module would also fail later in cloud-init because "builtin Azure ephemeral disk configs" are merged into DataSourceAzure metadata. These builtin configs reference the non-existent ephemeral disk, which causes the module to fail.

Why this approach was chosen

As of today, the Azure Instance Metadata Service (Azure IMDS) does not expose VM instance metadata indicating whether an ephemeral resource disk exists for the VM or not.

The Azure host also guarantees that the ephemeral resource disk is attached to the VM before it is booted during VM deployment.

Additionally, the ephemeral resource disk symlink (/dev/disk/cloud/azure_resource) that cloud-init waits for is actually created by a udev rule that comes with cloud-init. Additional relevant code, code, and code.

Because:

the Azure host guarantees that resource disks are attached for VM SKUs that have resource disks,
because the symlinks are created by udev rules (created as soon as the kernel detects the disk and sends the event to udev),
and because udevadm settle is invoked very early in boot by cloud-init (before DataSourceAzure runs),
it is guaranteed that the ephemeral resource disk symlink exists by the time DataSourceAzure runs.

Test Steps

No regression for VM SKUs with ephemeral resource disk

Created a custom image with this branch's cloud-init installed.
Deployed a Standard_DS1_V2 VM (has ephemeral resource disk) from this custom image.
Cloud-init logs here: https://paste.ubuntu.com/p/qdxkfwYKRD/

2021-02-03 23:23:31,913 - __init__.py[DEBUG]: Found unstable nic names: ['eth0']; calling udevadm settle
2021-02-03 23:23:31,914 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2021-02-03 23:23:31,933 - util.py[DEBUG]: Waiting for udev events to settle took 0.019 seconds
...
2021-02-03 23:23:32,823 - handlers.py[DEBUG]: finish: azure-ds/crawl_metadata: SUCCESS: crawl_metadata
2021-02-03 23:23:32,823 - util.py[DEBUG]: Crawl of metadata service took 1.117 seconds
...
2021-02-03 23:23:32,824 - azure.py[DEBUG]: Ephemeral resource disk '/dev/disk/cloud/azure_resource' exists. Merging default Azure cloud ephemeral disk configs.
...
2021-02-03 23:23:38,563 - handlers.py[DEBUG]: start: azure-ds/activate: activate
2021-02-03 23:23:38,563 - handlers.py[DEBUG]: start: azure-ds/address_ephemeral_resize: address_ephemeral_resize
2021-02-03 23:23:38,564 - azure.py[DEBUG]: Ephemeral resource disk '/dev/disk/cloud/azure_resource' exists.
...
2021-02-03 23:23:38,894 - handlers.py[DEBUG]: finish: azure-ds/address_ephemeral_resize: SUCCESS: address_ephemeral_resize

Fix for VM SKUs without ephemeral resource disk

Created a custom image with this branch's cloud-init installed.
Deployed a Standard_D2s_v4 VM (no ephemeral resource disk) from this custom image.
Cloud-init logs here: https://paste.ubuntu.com/p/GmCqBNTQrG/

2021-02-03 23:20:53,217 - __init__.py[DEBUG]: Found unstable nic names: ['eth0']; calling udevadm settle
2021-02-03 23:20:53,217 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2021-02-03 23:20:53,235 - util.py[DEBUG]: Waiting for udev events to settle took 0.018 seconds
...
2021-02-03 23:20:54,388 - handlers.py[DEBUG]: finish: azure-ds/crawl_metadata: SUCCESS: crawl_metadata
2021-02-03 23:20:54,389 - util.py[DEBUG]: Crawl of metadata service took 1.363 seconds
...
2021-02-03 23:20:54,389 - azure.py[DEBUG]: Ephemeral resource disk '/dev/disk/cloud/azure_resource' does not exist. Not merging default Azure cloud ephemeral disk configs.
...
2021-02-03 23:20:59,846 - handlers.py[DEBUG]: start: azure-ds/activate: activate
2021-02-03 23:20:59,846 - handlers.py[DEBUG]: start: azure-ds/address_ephemeral_resize: address_ephemeral_resize
2021-02-03 23:20:59,847 - azure.py[DEBUG]: Ephemeral resource disk '/dev/disk/cloud/azure_resource' does not exist.
2021-02-03 23:20:59,847 - handlers.py[DEBUG]: finish: azure-ds/address_ephemeral_resize: SUCCESS: address_ephemeral_resize

Checklist:

My code follows the process laid out in the documentation
I have updated or added any unit tests accordingly
I have updated or added any documentation accordingly

johnsonshi · 2021-02-04T00:16:11Z

@raharper @smoser @anhvoms The previous PR was closed due to inactivity as I was away for a short while. I have reopened it with the suggested changes. Dropped the wait entirely as the platform guarantees the existence of the resource disk for VMs that come with them, and udevadm settle is invoked anyways by cloud-init early in boot.

Also updated the PR message above with new info.

smoser

Given the unfortunate situation of not being able to know if we should expect a given disk or not, I think this is this approach is the best we can do.

My only concern is that we'll only know of failures when a user reports a problem, and they're not even likely to immediately notice a problem.

I guess the way to address that would be in cloud-tests. The test would be able to know (based on the requested instance-type or details) if it expected a disk or not. It could launch one of each and expect to find the disk or expect to have no WARN/errors in the log about one not being there.

@OddBloke or @TheRealFalcon is something like that possible?

TheRealFalcon · 2021-02-04T20:14:06Z

Yes, I think something like that shouldn't be too hard.

TheRealFalcon · 2021-02-19T22:41:01Z

@johnsonshi I created an example integration test that could be used/adapted here (note that is also requires the latest pycloudlib):

https://github.com/TheRealFalcon/cloud-init/blob/ephemeral-test/tests/integration_tests/bugs/test_lp1901011.py

OddBloke · 2021-02-22T20:47:36Z

+        assert 'azure_root -> ../../sda' in dev_links
+        assert 'azure_root-part1 -> ../../sda1' in dev_links
+        if is_ephemeral:
+            assert 'azure_resource -> ../../sdb' in dev_links
+            assert 'azure_resource-part1 -> ../../sdb1' in dev_links


I'm not certain, but I think the reason for these symlinks existing is that the underlying device names are not necessarily stable. I think this check is still good if you just s/ ->.*// each of the strings?

OddBloke · 2021-02-22T20:48:10Z

+        assert re.search('sda1.*part /', blks) is not None
+        if is_ephemeral:
+            assert re.search('sdb1.*part /mnt', blks) is not None


Given the above, probably want to dereference the named symlinks for these disks and check for those?

OddBloke

Integration test LGTM, and passes locally. Thanks!

smoser approved these changes Feb 4, 2021

View reviewed changes

TheRealFalcon force-pushed the decrease-azure-ephemeral-resource-disk-wait-time branch from 776844d to 01df49c Compare February 22, 2021 19:31

OddBloke reviewed Feb 22, 2021

View reviewed changes

OddBloke approved these changes Feb 22, 2021

View reviewed changes

johnsonshi and others added 9 commits February 22, 2021 16:40

Decrease Azure ephemeral disk wait time

719fbcd

Use LOG.debug for logging ephemeral disk not appearing

fdd65ad

Azure: Only process ephemeral disk if exists

05cd49d

Azure: Update log msg on presence of ephemeral disk

feecb2e

Azure: fix _get_data bug with undefined self.cfg ref

12e3ea0

Add UTs for ephemeral present and absent

625d20f

integration test for ephemeral instance

4cefabb

Added checking devices/mounts to test

a8192b2

updated test based on comments

1d25e98

TheRealFalcon force-pushed the decrease-azure-ephemeral-resource-disk-wait-time branch from cf3bb5f to 1d25e98 Compare February 22, 2021 21:41

TheRealFalcon merged commit a64b738 into canonical:master Feb 22, 2021

johnsonshi deleted the decrease-azure-ephemeral-resource-disk-wait-time branch February 23, 2021 19:23

This was referenced May 12, 2023

Release 21.1 #3846

Closed

Randomly set credentials written in cleartext to world-readable file #3854

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decrease azure ephemeral resource disk wait time#800

Decrease azure ephemeral resource disk wait time#800
TheRealFalcon merged 9 commits into
canonical:masterfrom
johnsonshi:decrease-azure-ephemeral-resource-disk-wait-time

johnsonshi commented Feb 4, 2021

Uh oh!

johnsonshi commented Feb 4, 2021

Uh oh!

smoser left a comment

Uh oh!

TheRealFalcon commented Feb 4, 2021

Uh oh!

TheRealFalcon commented Feb 19, 2021

Uh oh!

OddBloke Feb 22, 2021

Uh oh!

OddBloke Feb 22, 2021

Uh oh!

OddBloke left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

johnsonshi commented Feb 4, 2021

Proposed Commit Message

Additional Context

Problem

Why this approach was chosen

Test Steps

No regression for VM SKUs with ephemeral resource disk

Fix for VM SKUs without ephemeral resource disk

Checklist:

Uh oh!

johnsonshi commented Feb 4, 2021

Uh oh!

smoser left a comment

Choose a reason for hiding this comment

Uh oh!

TheRealFalcon commented Feb 4, 2021

Uh oh!

TheRealFalcon commented Feb 19, 2021

Uh oh!

OddBloke Feb 22, 2021

Choose a reason for hiding this comment

Uh oh!

OddBloke Feb 22, 2021

Choose a reason for hiding this comment

Uh oh!

OddBloke left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants