Add _netdev option to mount Azure ephemeral disk by otubo · Pull Request #1213 · canonical/cloud-init

otubo · 2022-01-26T11:40:04Z

Proposed Commit Message

The ephemeral disk depends on a functional network to be mounted. Even
though it depends on cloud-init.service, sometimes an ordering cycle is
noticed on the instance. If the option "_netdev" is added the problem is
gone.

rhbz: #1998445

Signed-off-by: Eduardo Otubo otubo@redhat.com

Additional Context

Test Steps

Deploy a RHEL-9/cloud-init-21.1 instance on Azure and observe that sometimes there's an ordering cycle on the logs and there's no network connectivity.

Checklist:

My code follows the process laid out in the documentation
I have updated or added any unit tests accordingly
I have updated or added any documentation accordingly

The ephemeral disk depends on a functional network to be mounted. Even though it depends on cloud-init.service, sometimes an ordering cycle is noticed on the instance. If the option "_netdev" is added the problem is gone. rhbz: #1998445 Signed-off-by: Eduardo Otubo <otubo@redhat.com>

otubo · 2022-01-26T12:08:08Z

For some reason tox didn't complain about the line length on my local copy. Anyways, just updated. Thanks!

anhvoms · 2022-01-30T18:07:46Z

@otubo Can you elaborate on why this is the case "The ephemeral disk depends on a functional network to be mounted"? Why is this an issue for RHEL 9 and not other distros (or earlier versions of RHEL)?
If this is indeed an issue that is specific to Azure, I think we should just scope it to DSAzure. DSAzure should pass in the mount options when it tries to mount the ephemeral resource disk.

otubo · 2022-01-31T08:57:19Z

@anhvoms you probably want to check the whole discussion, more specifically David's comment on the original BZ. TL;DR he says a recent update on systemd made it visible, but the problem might have been there all along.

cjp256 · 2022-01-31T13:48:30Z

@anhvoms you probably want to check the whole discussion, more specifically David's comment on the original BZ. TL;DR he says a recent update on systemd made it visible, but the problem might have been there all along.

Anh is out out for a couple of days. I don't have access to that bug so I cannot read along, but the mount already has a dependency on cloud-init.service, which itself should have a dependency on network being online. Has that changed for RHEL9?

RHEL 8.5:

# /usr/lib/systemd/system/cloud-init.service

[Unit]
Description=Initial cloud-init job (metadata service crawler)
Wants=cloud-init-local.service
Wants=sshd-keygen.service
Wants=sshd.service
After=cloud-init-local.service
After=NetworkManager.service network.service
After=NetworkManager-wait-online.service
...

Can you share the error you are seeing? I'm also not quite sure why networking needs to be online to mount the disk...

Thanks!

cjp256 · 2022-01-31T18:12:13Z

To summarize from the BZ...

A change in systemd (systemd/systemd@fa138f5) triggers an ordering cycle because local-fs.target will now set After=mnt.mount (and local-fs.target is required for cloud-init). Previously, it would just be Wants=mnt.mount, avoiding the order cycling.

systemd-248-7.el9: - No mnt.mount in "After"

systemctl show -p After local-fs.target

After=systemd-remount-fs.service tmp.mount run-user-0.mount
systemd-fsck-root.service boot-efi.mount local-fs-pre.target

systemd-249-2.el9: - Have mnt.mount in "After"

systemctl show -p After local-fs.target

After=systemd-fsck-root.service mnt.mount run-user-0.mount boot-efi.mount
run-user-1000.mount local-fs-pre.target systemd-remount-fs.service tmp.mount
-.mount

The argument made in the BZ is that due to a dependency on cloud-init.service, the ephemeral disk should not be considered local, but instead mark it as _netdev so that is not considered a local filesystem.

It seems to me that this grouping is slightly at odds with the x-systemd.requires configuration, but from what I gather reading the systemd.mount manpage, mounts fall into two groups: local-fs.target or remote-fs.target, depending whether the file system is local or remote. If our mount cannot make the local-fs.target, it should declare _netdev to group in with remote-fs.target.

Another option would be to instead use nofail so that it is not required by either local or remote targets - but that would likely cause other issues :)

From my perspective given what I've seen, this seems a reasonable change (and not specific to Azure). 👍 Are there possible side-effects from grouping with remote-fs.target?

anhvoms · 2022-02-02T15:22:46Z

@cjp256 thanks for the explanation. This looks reasonable to me

TheRealFalcon

Thanks for the explanation @cjp256 . Given that this module runs after networking has been applied, it looks like a reasonable change.

TheRealFalcon · 2022-02-02T22:23:36Z

@otubo , looks like there's still a formatting issue. Can you run tox -e format to fix it (or after a rebase it would be tox -e do_format)?

otubo · 2022-02-04T12:48:17Z

I hate style issues, sorry for the inconvenience guys. Should be good to go now. Thanks!

Fixes the spaces introduced in #1213 Signed-off-by: Chris Patterson <cpatterson@microsoft.com>

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com>

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes #26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes #28504

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes scylladb#28504 (cherry picked from commit 6d50e67)

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes #26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes #28504 (cherry picked from commit 6d50e67) Closes #29339

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes scylladb#28504 (cherry picked from commit 6d50e67) Closes scylladb#29339

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes scylladb#28504

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes #26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes #28504 (cherry picked from commit 6d50e67) Closes #29339 Closes #29354

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes scylladb#28504 (cherry picked from commit 6d50e67)

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes scylladb#28504 (cherry picked from commit 6d50e67) Closes scylladb#29339 Closes scylladb#29354

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes scylladb#26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes scylladb#28504 (cherry picked from commit 6d50e67) Closes scylladb#29339 Closes scylladb#29354 (cherry picked from commit 7ed7728)

When a Scylla node starts, the scylla-image-setup.service invokes the `scylla_swap_setup` script to provision swap. This script allocates a swap file and creates a swap systemd unit to delegate control to systemd. By default, systemd injects a Before=swap.target dependency into every swap unit, allowing other services to use swap.target to wait for swap to be enabled. On Azure, this doesn't work so well because we store the swap file on the ephemeral disk [1] which has network dependencies (`_netdev` mount option, configured by cloud-init [2]). This makes the swap.target indirectly depend on the network, leading to dependency cycles such as: swap.target -> mnt-swapfile.swap -> mnt.mount -> network-online.target -> network.target -> systemd-resolved.service -> tmp.mount -> swap.target This patch breaks the cycle by removing the swap unit from swap.target using DefaultDependencies=no. The swap unit will still be activated via WantedBy=multi-user.target, just not during early boot. Although this problem is specific to Azure, this patch applies the fix to all clouds to keep the code simple. Fixes #26519. Fixes SCYLLADB-1257 [1] scylladb/scylla-machine-image#426 [2] canonical/cloud-init#1213 (comment) Signed-off-by: Nikos Dragazis <nikolaos.dragazis@scylladb.com> Closes #28504 (cherry picked from commit 6d50e67) Closes #29339 Closes #29354 (cherry picked from commit 7ed7728) Closes #29517

otubo added 2 commits January 26, 2022 12:31

Breaking the line earlier so tox won't complain

cdb003d

TheRealFalcon approved these changes Feb 2, 2022

View reviewed changes

otubo added 2 commits February 4, 2022 13:01

Fixing style issues.

398e619

Another style problem fixed

bc0b5a7

TheRealFalcon merged commit a1cfd1b into canonical:main Feb 4, 2022

cjp256 mentioned this pull request Feb 7, 2022

mounts: fix mount opts string for ephemeral disk #1250

Merged

blackboxsw pushed a commit that referenced this pull request Feb 10, 2022

mounts: fix mount opts string for ephemeral disk (#1250)

8d96687

Fixes the spaces introduced in #1213 Signed-off-by: Chris Patterson <cpatterson@microsoft.com>

ubuntu-server-builder mentioned this pull request May 12, 2023

Release 22.1 #3948

Closed

ndragazis mentioned this pull request Feb 1, 2026

After soft_reboot_node nemesis Unable to get azure kms key due to DNS timeouts on Azure scylladb/scylladb#26519

Closed

fruch mentioned this pull request Feb 2, 2026

Fix Azure DNS timeout and systemd swap dependency cycle scylladb/scylla-machine-image#891

Closed

2 tasks

ndragazis mentioned this pull request Feb 2, 2026

SCYLLADB-1257 scylla_swap_setup: Remove Before=swap.target dependency from swap unit scylladb/scylladb#28504

Closed

scylladbbot mentioned this pull request Apr 5, 2026

[Backport 2026.1] SCYLLADB-1257 scylla_swap_setup: Remove Before=swap.target dependency from swap unit scylladb/scylladb#29339

Closed

scylladbbot mentioned this pull request Apr 6, 2026

[Backport 2025.4] SCYLLADB-1257 scylla_swap_setup: Remove Before=swap.target dependency from swap unit scylladb/scylladb#29354

Closed

scylladbbot mentioned this pull request Apr 16, 2026

[Backport 2025.3] SCYLLADB-1257 scylla_swap_setup: Remove Before=swap.target dependency from swap unit scylladb/scylladb#29511

Closed

scylladbbot mentioned this pull request Apr 16, 2026

[Backport 2025.3] SCYLLADB-1257 scylla_swap_setup: Remove Before=swap.target dependency from swap unit scylladb/scylladb#29512

Closed

scylladbbot mentioned this pull request Apr 16, 2026

[Backport 2025.1] SCYLLADB-1257 scylla_swap_setup: Remove Before=swap.target dependency from swap unit scylladb/scylladb#29517

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add _netdev option to mount Azure ephemeral disk#1213

Add _netdev option to mount Azure ephemeral disk#1213
TheRealFalcon merged 4 commits into
canonical:mainfrom
otubo:1998445

otubo commented Jan 26, 2022

Uh oh!

otubo commented Jan 26, 2022

Uh oh!

anhvoms commented Jan 30, 2022 •

edited

Loading

Uh oh!

otubo commented Jan 31, 2022

Uh oh!

cjp256 commented Jan 31, 2022

Uh oh!

cjp256 commented Jan 31, 2022

systemctl show -p After local-fs.target

systemctl show -p After local-fs.target

Uh oh!

anhvoms commented Feb 2, 2022

Uh oh!

TheRealFalcon left a comment

Uh oh!

TheRealFalcon commented Feb 2, 2022 •

edited

Loading

Uh oh!

otubo commented Feb 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

otubo commented Jan 26, 2022

Proposed Commit Message

Additional Context

Test Steps

Checklist:

Uh oh!

otubo commented Jan 26, 2022

Uh oh!

anhvoms commented Jan 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

otubo commented Jan 31, 2022

Uh oh!

cjp256 commented Jan 31, 2022

Uh oh!

cjp256 commented Jan 31, 2022

systemctl show -p After local-fs.target

systemctl show -p After local-fs.target

Uh oh!

anhvoms commented Feb 2, 2022

Uh oh!

TheRealFalcon left a comment

Choose a reason for hiding this comment

Uh oh!

TheRealFalcon commented Feb 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

otubo commented Feb 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anhvoms commented Jan 30, 2022 •

edited

Loading

TheRealFalcon commented Feb 2, 2022 •

edited

Loading