feat(hetzner): enable hotplug support and prepare IPv6 integration

[DarkPhily]

Proposed Commit Message

feat(hetzner): enable hotplug support and prepare IPv6 integration

This PR tries to accomplish hotplug support for the Hetzner cloud. cloud-init should pick up the newly attached private network and configure the interface accordingly. Furthermore, this PR also prepares cloud-init to use the IPv6 Hetzner metadata-service

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[DarkPhily]

One test is failing, because of the new Attributes to DataSourceHetzner.

FAILED tests/unittests/test_upgrade.py::TestUpgrade::test_all_ds_init_vs_unpickle_attributes[mode0] - AssertionError: New Hetzner attributes need unpickle coverage: {'max_wait', 'sleep_time', 'extra_hotplug_udev_rules', 'metadata_path', 'userdata_path', 'metadata_private_networks_path'}
assert not {'max_wait', 'sleep_time', 'extra_hotplug_udev_rules', 'metadata_path', 'userdata_path', 'metadata_private_networks_path'}

I'm not sure where I would need to make the changes for the test to pass and would be happy if you could point me to the right direction.

[aciba90]

Thanks for the feature!

Drive-by comment: The testing issue is that the urls code gets evaluated at module import-time which happens prior to mocking, to fix it something in the lines of:

diff --git a/cloudinit/sources/DataSourceHetzner.py b/cloudinit/sources/DataSourceHetzner.py
index 09a0eb938..c29f03515 100644
--- a/cloudinit/sources/DataSourceHetzner.py
+++ b/cloudinit/sources/DataSourceHetzner.py
@@ -16,10 +16,11 @@ from cloudinit.net.ephemeral import EphemeralIPNetwork
 
 LOG = logging.getLogger(__name__)
 
-BASE_URLS_V1 = [
-    f"http://[fe80::a9fe:a9fe%25{net.find_fallback_nic()}]/hetzner/v1/",
-    "http://169.254.169.254/hetzner/v1/",
-]
+def base_urls_v1():
+    return (
+        f"http://[fe80::a9fe:a9fe%25{net.find_fallback_nic()}]/hetzner/v1/",
+        "http://169.254.169.254/hetzner/v1/",
+    )
 
 
 BUILTIN_DS_CONFIG = {
@@ -85,6 +86,7 @@ class DataSourceHetzner(sources.DataSource):
         if not on_hetzner:
             return False
 
+        base_urls = base_urls_v1()
         try:
             with EphemeralIPNetwork(
                 self.distro,
@@ -97,13 +99,13 @@ class DataSourceHetzner(sources.DataSource):
                             url, "metadata/instance-id"
                         )
                     }
-                    for url in BASE_URLS_V1
+                    for url in base_urls
                 ],
             ):
                 url, contents = hc_helper.get_metadata(
                     [
                         url_helper.combine_url(url, self.metadata_path)
-                        for url in BASE_URLS_V1
+                        for url in base_urls
                     ],
                     max_wait=self.max_wait,
                     timeout=self.timeout,
@@ -116,7 +118,7 @@ class DataSourceHetzner(sources.DataSource):
                         url_helper.combine_url(
                             url, self.metadata_private_networks_path
                         )
-                        for url in BASE_URLS_V1
+                        for url in base_urls
                     ],
                     max_wait=self.max_wait,
                     timeout=self.timeout,
@@ -129,7 +131,7 @@ class DataSourceHetzner(sources.DataSource):
                 url, ud = hc_helper.get_metadata(
                     [
                         url_helper.combine_url(url, self.userdata_path)
-                        for url in BASE_URLS_V1
+                        for url in base_urls
                     ],
                     max_wait=self.max_wait,
                     timeout=self.timeout,

[aciba90]

I will peak at the other testing issue later and a full review.

[aciba90]

As we are introducing new attributes to the datasource class, we need to introduce un unpickle method to restore them for cached data sources:

diff --git a/cloudinit/sources/DataSourceHetzner.py b/cloudinit/sources/DataSourceHetzner.py
index 09a0eb938..4c2e91549 100644
--- a/cloudinit/sources/DataSourceHetzner.py
+++ b/cloudinit/sources/DataSourceHetzner.py
@@ -79,12 +80,24 @@ class DataSourceHetzner(sources.DataSource):
 
         self.extra_hotplug_udev_rules = EXTRA_HOTPLUG_UDEV_RULES
 
+    def _unpickle(self, ci_pkl_version: int) -> None:
+        super()._unpickle(ci_pkl_version)
+        self.extra_hotplug_udev_rules = EXTRA_HOTPLUG_UDEV_RULES
+        self.max_wait = self.ds_cfg.get("max_wait", MD_MAX_WAIT)
+        self.metadata_path = self.ds_cfg.get("metadata_path")
+        self.metadata_private_networks_path = self.ds_cfg.get(
+            "metadata_private_networks_path"
+        )
+        self.sleep_time = self.ds_cfg.get("sleep_time", MD_SLEEP_TIME)
+        self.userdata_path = self.ds_cfg.get("userdata_path")
+
     def _get_data(self):
         (on_hetzner, serial) = get_hcloud_data()

[DarkPhily]

Which test is your first comment referring to? The Lint Test / Check mypy?

Looking at the CI output, the issue isn't in the DataSourceHetzner.py, but in helpers/hetzner.py and I don't quite get, why it's complaining about no return, since there clearly is a return 😅
The raise of any other URLError was missing.

[aciba90]

Which test is your first comment referring to? The Lint Test / Check mypy?

tests/unittests/sources/test_hetzner.py::TestDataSourceHetzner::test_read_data

The mock in the line:

m_fallback_nic.return_value = "eth0"

doesn't get applied because the function is called at import-time.

Looking at the CI output, the issue isn't in the DataSourceHetzner.py, but in helpers/hetzner.py and I don't quite get, why it's complaining about no return, since there clearly is a return 😅

That's another one, the get_metadata function doesn't have an explicit return when len(urls) == 0.

[DarkPhily]

The mock in the line:

  m_fallback_nic.return_value = "eth0"

doesn't get applied because the function is called at import-time.

How did you catch that one? 😅 I don't see a failed test for this

[DarkPhily]

@aciba90 Should the docs be updated in this PR too?

I noticed, that you state the currently supported DataSources for hotplug support in doc/module_docs/cc_install_hotplug/data.yml

Currently supported datasources: Openstack, EC2

[DarkPhily]

@aciba90
Do you need something else to be done? :)
I also wanted to ask if you can see which CLA the job found. My personal or the company CLA?

[aciba90]

The changes look great, thanks for the improvements. I left in-line comments to improve the code, in addition to that could you please:

• Provide logs of a machine that has exercised this code (attaching a NIC to a running instance), if feasible.
• Give more information of what this PR accomplishes in the proposed PR commit message.

---

The mock in the line:

m_fallback_nic.return_value = "eth0"

doesn't get applied because the function is called at import-time.

How did you catch that one? 😅 I don't see a failed test for this

My laptop doesn't have an ethernet connection, so my default nic name was different from eth0. :)

I also wanted to ask if you can see which CLA the job found. My personal or the company CLA?

It looks like your username is detected there, but I not completely sure which one is picked: https://github.com/canonical/cloud-init/actions/runs/17413361821/job/49575901996

[DarkPhily]

I have a hard time to install cloud-init to the system, since I merged the main branch, because you changed the build-system in the meantime 😅 .
Maybe you can provide more insight how I could install it now, while testing.

Nonetheless, I manually triggered the hotplug event for now, to generate the log:
cloud-init.tar.gz

[aciba90]

LGTM, thanks!

Thanks for attaching logs of a manual run.

I have a hard time to install cloud-init to the system, since I merged the main branch, because you changed the build-system in the meantime 😅 . Maybe you can provide more insight how I could install it now, while testing.

Doesn't ./packages/bddeb -d from https://cloudinit.readthedocs.io/en/latest/development/package_testing.html#package-testing work for you?

[DarkPhily]

Unfortunately not.
The script complains, that I don't have permissions to execute debian/rules

Creating a temp tarball using the 'make-tarball' helper
Extracting temporary tarball 'cloud-init_25.2-62-gf57e6c26.orig.tar.gz'
Creating a debian/ folder in '/run/cloud-init/tmp/tmpgb3d7mvw/cloud-init-25.2-62-gf57e6c26'
Running 'debuild -d -us -uc' in '/run/cloud-init/tmp/tmpgb3d7mvw/cloud-init-25.2-62-gf57e6c26'
Traceback (most recent call last):
  File "/root/cloud-init/./packages/bddeb", line 387, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/cloud-init/./packages/bddeb", line 361, in main
    subp.subp(cmd, capture=capture)
  File "/root/cloud-init/cloudinit/subp.py", line 291, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['debuild', '--preserve-envvar', 'INIT_SYSTEM', '-d', '-us', '-uc']
Exit code: 29
Reason: -
Stdout:  dpkg-buildpackage -d -us -uc -ui
        dpkg-buildpackage: info: source package cloud-init
        dpkg-buildpackage: info: source version 25.2-62-gf57e6c26-1~bddeb
        dpkg-buildpackage: info: source distribution UNRELEASED
        dpkg-buildpackage: info: source changed by Scott Moser <smoser@ubuntu.com>
         dpkg-source --before-build .
        dpkg-buildpackage: info: host architecture amd64
         debian/rules clean
        Can't exec "debian/rules": Permission denied at /usr/bin/dpkg-buildpackage line 913.
        dpkg-buildpackage: error: debian/rules clean subprocess failed with unknown status code -1
Stderr: debuild: fatal error at line 1184:
        dpkg-buildpackage -d -us -uc -ui failed

Even when I explicitly use sudo it complains about this.

I worked around this by using pip install:

uv pip install -e . --system --break-system-packages

But this isn't working anymore, because you removed the build-backend in pyproject.toml

Should I maybe open an issue regarding the bddeb not working correctly?

[aciba90]

Thanks for the information. That error is the same as #6141, I think.

I believe there is something going on in packages/bddeb. Would you mind dropping a breakpoint and inspecting the contents of xdir? In particurar ls -hal $xdir/debian, as it seems that for some reason, your user doesn't have permission to execute debian/rules in that dir.

diff --git a/packages/bddeb b/packages/bddeb
index 4e06378c7..77f81d4c7 100755
--- a/packages/bddeb
+++ b/packages/bddeb
@@ -361,6 +361,7 @@ def main():
             "Running 'debuild %s' in %r" % (" ".join(args.debuild_args), xdir)
         )
         with util.chdir(xdir):
+            import pdb; pdb.set_trace()
             cmd = ["debuild", "--preserve-envvar", "INIT_SYSTEM"]
             if args.debuild_args:
                 cmd.extend(args.debuild_args)

[aciba90]

@DarkPhily, is there anything else to be done here? Or are good to merge? Thanks.

[DarkPhily]

We are good to go :)

PS: I can't merge it myself