feat(raspberry-pi-os): Add Raspberry Pi OS support by paulober · Pull Request #5827 · canonical/cloud-init

paulober · 2024-10-16T15:43:27Z

This commit adds support for the Raspberry Pi OS debian based distribution. It includes a distro definition, 3 modules and integration into other systems like config generation and docs.

I have signed the CLA: https://ubuntu.com/legal/contributors
I have added my Github username to tools/.github-cla-signers
I have included a comprehensive commit message using the guide below
I have added unit tests to cover the new behavior under tests/unittests/
- Test files should map to source files i.e. a source file cloudinit/example.py should be tested by tests/unittests/test_example.py
- Run unit tests with tox -e py3
I have kept the change small, avoiding unnecessary whitespace or non-functional changes.
I have added a reference to issues that this PR relates to in the PR message
I have updated the documentation with the changed behavior.
- If the change doesn't change the user interface and is trivial, this step may be skipped.
- Cloud-config documentation is generated from the jsonschema.
- Generate docs with tox -e doc.

Proposed Commit Message

<type>(raspberry-pi-os): <summary>  Add support for Raspberry Pi OS

This commit adds support for the Raspberry Pi OS distro.
It includes a distro integration and detection including
an additional configuration module (cc_raspberry_pi) for
platform specific configuration.

Additional Context

@tdewey-rpi, @XECDesign, @ghollingworth, @will-v-pi

Merge type

Squash merge using "Proposed Commit Message"
Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

paulober · 2024-10-17T08:57:31Z

Is there an option if a setting for rpi_interfaces has been set, to tell the power_state_change module that a reboot is required?

tdewey-rpi · 2024-10-17T09:49:11Z

Some additional context:

This PR (and the linked GitHub PRs from other repos) are part of a wider desire to move to using cloud-init as the customisation scheme for Raspberry Pi OS, with associated infrastructure in Raspberry Pi Imager to ensure we maintain feature parity (if not an increased feature set) with the current 'firstrun' style customisation scheme.

Going forward, assuming this PR is merged and our migration proceeds along happily enough, my plan is to recommend to each distro vendor supplying OS images to the Raspberry Pi Imager that they also adopt cloud-init as their customisation scheme. This may result in additional PRs adding distro features to cloud-init in a similar manner to this PR - but should absolutely improve the experience for end users across the board.

holmanb · 2024-10-17T16:43:28Z

    renderers: ['network-manager', 'networkd']
+{% elif variant == "raspberry-pi-os" %}
+  network:
+    dhcp_client_priority: [dhclient]


Upstream dhclient is abandoned. Is raspberry-pi-os's plan to support it forever?

This was included as an error, and dhclient is not actually used by Raspberry Pi OS - instead, we use network-manager's built in DHCP client. I believe this has been corrected in the latest version of this PR.

holmanb · 2024-10-17T16:44:34Z

+{% elif variant == "raspberry-pi-os" %}
+  network:
+    dhcp_client_priority: [dhclient]
+    renderers: ['netplan', 'network-manager', 'networkd', 'eni']


Which of these does raspberry-pi-os support and plan to use?

Which of these does raspberry-pi-os support and plan to use?

In the updated PR, this has been reduced to netplan & network-manager, both of which we want to offer.

holmanb · 2024-11-12T22:57:38Z

There are some merge conflicts now.

paulober · 2024-11-14T13:23:25Z

There are some merge conflicts now.

Resolved.

github-actions · 2024-12-05T00:28:53Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

paulober · 2024-12-05T11:58:00Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

@TheRealFalcon

TheRealFalcon

Thanks for this contribution. I left some first pass inline comments.

In general, I would like to consolidate the config and modules as much as possible. When it comes to the config, let's put anything raspberry-pi specific under a rpi or similar top-level key. E.g.,

rpi:
  enable_rpi_connect: true
  interfaces:
    ...
  piwiz:
    ...

Do all the modules need to run at the boot stages they've been added? If not, can any of the modules be combined? E.g., cc_rpi_connect.py runs a single command. Can that command not be incorporated into the cc_rpi_userdata.py module?

TheRealFalcon · 2024-12-10T21:33:39Z

@@ -0,0 +1,129 @@
+# Copyright (C) 2024, Raspberry Pi Ltd.


I don't understand why this file is necessary. Cloud-init writes 50-cloud-init.yaml, but as far as I know, all network-manager -> netplan files start with '90-'. Doesn't this mean any changes made in network-manager will automatically override the default netplan config? Are you aware of cases where this doesn't work?

If I remember correctly, the problem was if you edit the netplan generated NetworkManager (created by cloud-init) config in a desktop GUI the updated config will be written back to netplan with a generated UUID and not touching the original file. This ment changes made by a user are lost after a reboot because the cloud-init netplan config will always be preferred and overrides the NM generated one. By triggering the save operation for every cloud-init generated config after first boot and then removing the original file this issue can be resolved.

I'm curious if things are different on raspberry-pi-os, but on my desktop at least, I see the network manager files get written with a '90-' first, e.g., 90-NM-141e654c-21b3-4e52-9505-e3c94be335c0.yaml so anything in that file would override what is in 50-cloud-init.yaml. If it doesn't work this way, I would think this would be a netplan bug, or the patch was applied incorrectly. Is this something you can check to see if you get different results?

Ok I just checked without the netplan_nm_patch, and yes the updated config is written to 90-NM... and also visible in the UI. But after a reboot the 50-cloud-init will take over => changes are not effective.

But 50-cloud-init.yaml shouldn't "take over" unless there's something wrong with your netplan implementation. I'm assuming 50-cloud-init.yaml has some level of base configuration and then the 90-* files are being layered on top? What does netplan get show when you're in this state? How is it different from what you would expect with how netplan's layering works?

I still don't think that a module is the best way to go about this. If you don't want cloud-init to write networking at all, then it'd be best to update your image accordingly. If you need different network-manager behavior, perhaps choose the network-manager renderer if netplan's layering isn't working correctly for you.

As a last resort, raspberry-pi-os.py could override _write_network_state() directly along with a long explaining why it's doing what it's doing.

Just found out that my problem is related to a duplicate access-point issue that has been fixed with a PR for 0.107 the problem is 0.107 of libnetplan0 already requires libc6 >= 2.38. But everything except wifi works also without the netplan_nm_patch.

TheRealFalcon · 2024-12-10T21:46:42Z

-{% if variant in ["ubuntu", "unknown", "debian"] %}
+{% if variant in ["ubuntu", "unknown", "debian", "raspberry-pi-os"] %}
 After=networking.service
+{% if variant == "raspberry-pi-os" %}


Why is this section needed? After=NetworkManager... is already a few line below.

I will have to dig into this again. But I think it was need because I've had huge problems getting the cloud-init network stage run after network manager is available but not to late so it wont get skipped.

TheRealFalcon · 2024-12-10T21:52:33Z

 Before=sshd.service
 Before=systemd-user-sessions.service
-{% if variant in ["ubuntu", "unknown", "debian"] %}
+{% if variant in ["ubuntu", "unknown", "debian", "raspberry-pi-os"] %}


I don't think raspberry-pi-os is needed in this section. Based on the line below you don't want Before=sysinit.target and the two after that are part of DefaultDependencies. Can you add raspberry-pi-os to the line about default dependencies?

Thanks for the advice. I'll test it.

TheRealFalcon · 2024-12-10T22:03:19Z

+        LOG.error("Failed to configure serial console: %s", e)
+
+
+def enable_ssh(cfg: Config, enable: bool) -> None:


Do we need the separate tool to enable SSH? Cloud-init has built in SSH support. Is there a reason we can't use that? If so, is there a reason that module can't be expanded to do what this tool is doing?

TheRealFalcon · 2024-12-10T22:08:36Z

 {% endif %}
  - locale
 {% endif %}
+{% if variant == "raspberry-pi-os" %}


rpi_interfaces was also defined in the network stage. Does this module need to run twice? Why?

You're right, it doesn't. I'll remove the duplicate.

TheRealFalcon · 2024-12-10T22:10:22Z

        "lxd",
        "mcollective",
        "mounts",
+        "netplan-nm-patch",


We only need the _ version of all the keys. The - versions exist in other modules for backwards compatibility only.

TheRealFalcon · 2024-12-10T22:26:58Z

@@ -0,0 +1,228 @@
+# Copyright (C) 2024, Raspberry Pi Ltd.


Much of this module is a bit strange to me. We're using a one-shot first-boot system configuration service (cloud-init) to configure how a one-shot first-boot system configuration service (piwiz) should work. Why would people want to use both in tandem? Does piwiz do anything that cloud-init currently can't? If not, then why not just use cloud-init for everything? If no user data is passed, I can see still starting piwiz so it can be used interactively, but if a user provides cloud-init user data, I see no reason to then also use a separate configuration service.

In particular, I think allowing username and password configuration multiple ways is very confusing. Cloud-init will create a default user automatically that can be overridden to provide a different user and password. Why would we then also expose additional cloud-init user data to allow piwiz to set the same thing? Let's have one configuration option for applying these things.

Thank you for raising these points. The userconf-pi scripts handle more than just the setup wizard. They also rename the default user shipped with Raspbian OS and manage other configuration tasks that are specific to the OS. Overwriting the user setup logic directly in the distribution's Python script seemed more complex.

The module runes even if not configuration is supplied to ensure the scripts are launched correctly, especially since the systemd hook is disabled on builds with cloud-init, allowing for this behavior to be modified.

That said, your suggestion to streamline this makes sense. I agree that having multiple methods to configure the username and password can be confusing. While piwiz provides interactive setup capabilities, cloud-init is designed for automated configuration. Ideally, there should be a single clear configuration pathway.

If I find the time, I’ll look into the feasibility of removing the userconf module or aligning it more closely with cloud-init. This will likely require some adjustments on the OS side, but it’s worth exploring. I’ll revisit this and update you once I have a concrete solution.

github-actions · 2025-01-07T00:25:02Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

paulober · 2025-01-07T12:41:07Z

Hi @TheRealFalcon, do you have any recommendation on how I can build cloud-init deb pkgs for testing. My current build script always results in (and I had to modify a requirements.txt as it always tries to locate python3-pyserial instead of python3-serial which should normally be auto-renamed by the config in the debs.json if I understand correctly):

dpkg-buildpackage: info: host architecture arm64
         debian/rules clean
        Can't exec "debian/rules": Permission denied at /usr/bin/dpkg-buildpackage line 834.
        dpkg-buildpackage: error: debian/rules clean subprocess failed with unknown status code -1
Stderr: debuild: fatal error at line 1182:
        dpkg-buildpackage -us -uc -ui failed

my build.sh:

#!/bin/bash
cd cloud-init
VARIANT=raspberry-pi-os
make deb VARIANT=raspberry-pi-os
dpkg-deb -R cloud-init_all.deb package_dir
make render-template VARIANT=raspberry-pi-os | tail -n +2 > package_dir/etc/cloud/cloud.cfg

SYSTEM_DIR="./package_dir/lib/systemd/system"
GENERATORS_DIR="./package_dir/lib/systemd/system-generators"

# Iterate over all .tmpl files in the ./systemd directory
for tmpl_file in ./systemd/*.tmpl; do
    # Get the base name of the file without the directory and extension
    base_name=$(basename "$tmpl_file" .tmpl)

    # Check if the file is the generator file
    if [ "$base_name" = "cloud-init-generator" ]; then
        # Render the template and copy it to the system-generators directory
        output_file="$GENERATORS_DIR/$base_name"
        ./tools/render-template --variant="${VARIANT}" "$tmpl_file" > "$output_file"
        echo "Rendered generator template $tmpl_file to $output_file"
    else
        # Render the template and copy it to the system directory
        output_file="$SYSTEM_DIR/$base_name"
        ./tools/render-template --variant="${VARIANT}" "$tmpl_file" > "$output_file"
        echo "Rendered service template $tmpl_file to $output_file"
    fi
done

make clean_packaging
dpkg-deb -b package_dir cloud-init-custom.deb
rm -rf package_dir
cp cloud-init-custom.deb ~/cloud-init/pi-gen/stage2/04-cloud-init/files
cd ..

Platform: Debian 12 arm64

Many thanks,
Paul

TheRealFalcon · 2025-01-07T20:11:03Z

@paulober , for throwaway test debs, we have a script for that. Usage for me usually looks something like

DEB_BUILD_OPTIONS=nocheck packages/bddeb -d

Would that work for you?

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

paulober · 2025-05-15T10:58:33Z

@TheRealFalcon I now re-authored all commits. Let me know if you also need me to reword the commit messages.

TheRealFalcon · 2025-05-15T11:33:42Z

I signed the CLA on behalf of Raspberry Pi

@ghollingworth , did it involve docusign? If not, it wasn't using the new process for organizations. If that happened, let me know and I can make sure the email domain is accepted.

ghollingworth · 2025-05-15T11:42:52Z

Yes, this was done seven days ago... I've not heard anything since

TheRealFalcon · 2025-05-15T12:01:30Z

@ghollingworth, thanks for the confirmation. I'll make sure things get updated internally.

@paulober, everything else looks good here. Thank you for all of the work and patience on this PR. Once I get the CLA worked out, I'll merge this.

github-actions · 2025-05-30T00:28:07Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

TheRealFalcon

@paulober , thanks for your patience on this one! We are finally good to merge!

This commit adds support for the Raspberry Pi OS distro. It includes a distro integration and detection including an additional configuration module (cc_raspberry_pi) for platform specific configuration. Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

github-actions Bot added the documentation This Pull Request changes documentation label Oct 16, 2024

This was referenced Oct 16, 2024

Cloud-Init support RPi-Distro/pi-gen#797

Closed

Cloud-init support for Raspberry Pi OS raspberrypi/rpi-imager#950

Closed

holmanb added the CLA signed The submitter of the PR has signed the CLA label Oct 16, 2024

paulober force-pushed the raspios-support branch 2 times, most recently from c475d72 to 683ffc8 Compare October 17, 2024 08:34

paulober marked this pull request as ready for review October 17, 2024 08:41

holmanb reviewed Oct 17, 2024

View reviewed changes

paulober force-pushed the raspios-support branch 2 times, most recently from fcd247a to 9bb2b18 Compare October 18, 2024 10:42

paulober force-pushed the raspios-support branch from 9bb2b18 to 7cdcb3c Compare November 14, 2024 13:22

paulober force-pushed the raspios-support branch from 7cdcb3c to cdbb40b Compare November 20, 2024 11:09

github-actions Bot added the stale-pr Pull request is stale; will be auto-closed soon label Dec 5, 2024

paulober force-pushed the raspios-support branch from cdbb40b to 0769b9c Compare December 5, 2024 12:00

TheRealFalcon removed the stale-pr Pull request is stale; will be auto-closed soon label Dec 5, 2024

paulober force-pushed the raspios-support branch from 0769b9c to 1da5ee6 Compare December 10, 2024 11:58

TheRealFalcon suggested changes Dec 10, 2024

View reviewed changes

TheRealFalcon self-assigned this Dec 10, 2024

github-actions Bot added the stale-pr Pull request is stale; will be auto-closed soon label Jan 7, 2025

TheRealFalcon removed the stale-pr Pull request is stale; will be auto-closed soon label Jan 7, 2025

paulober added 16 commits May 15, 2025 12:52

Add cc_netplan_nm_patch module

2acce87

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Fix some lint errors

18b3d10

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Simplify rpios integration

74c5111

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Remove netplan_nm_patch

9a0cb1c

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Remove left-over

bb0f7c8

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Fix tox errors in PR files

aa3a9c7

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

raspberry-pi-os: Added unit tests + requested changes

799c497

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Using caplog + more logging

502088d

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Formatting

be06751

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Fix module naming

ee7975f

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Fix reboot request and recommendations

3b26d73

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Add more tests

a9a8997

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Remove custom exit

9bbe41b

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Fix reboot cmd exec

1d287e5

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Fix tests

3387cb2

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

Update examples

c8a0828

Signed-off-by: paulober <44974737+paulober@users.noreply.github.com>

paulober force-pushed the raspios-support branch from f9f02c1 to c8a0828 Compare May 15, 2025 10:55

github-actions Bot added the stale-pr Pull request is stale; will be auto-closed soon label May 30, 2025

TheRealFalcon removed the stale-pr Pull request is stale; will be auto-closed soon label May 30, 2025

TheRealFalcon approved these changes Jun 4, 2025

View reviewed changes

TheRealFalcon merged commit c61e42a into canonical:main Jun 4, 2025
21 of 22 checks passed

		LOG.error("Failed to configure serial console: %s", e)


		def enable_ssh(cfg: Config, enable: bool) -> None:

Conversation

paulober commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Commit Message

Additional Context

Merge type

Uh oh!

paulober commented Oct 17, 2024

Uh oh!

tdewey-rpi commented Oct 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

holmanb commented Nov 12, 2024

Uh oh!

paulober commented Nov 14, 2024

Uh oh!

github-actions Bot commented Dec 5, 2024

Uh oh!

paulober commented Dec 5, 2024

Uh oh!

TheRealFalcon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulober Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheRealFalcon Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 7, 2025

Uh oh!

paulober commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheRealFalcon commented Jan 7, 2025

Uh oh!

paulober commented May 15, 2025

Uh oh!

TheRealFalcon commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

paulober commented Oct 16, 2024 •

edited

Loading

paulober Dec 11, 2024 •

edited

Loading

TheRealFalcon Dec 10, 2024 •

edited

Loading

paulober commented Jan 7, 2025 •

edited

Loading

TheRealFalcon commented May 15, 2025 •

edited

Loading