Skip to content

fix(raspberry-pi-os): Systemd network service template fix#6459

Merged
blackboxsw merged 1 commit into
canonical:mainfrom
paulober:fix-rpios-network-service
Dec 3, 2025
Merged

fix(raspberry-pi-os): Systemd network service template fix#6459
blackboxsw merged 1 commit into
canonical:mainfrom
paulober:fix-rpios-network-service

Conversation

@paulober
Copy link
Copy Markdown
Contributor

@paulober paulober commented Sep 8, 2025

Proposed Commit Message

fix(raspberry-pi-os): adjust systemd network ordering; drop obsolete deps

Align systemd network ordering with current Raspberry Pi OS behavior.
The previous sequencing was designed around older cloud-init and
setup-wizard requirements, but recent upstream and downstream
changes make those workarounds unnecessary. The updated ordering
matches default expectations and works reliably with newer
NetworkManager/Netplan fixes.

Remove the no-longer-needed Before=avahi-daemon.service dependency,
simplifying the unit and bringing it closer to other distros.

Run the network units before sysinit.target, which is consistent with how
RPi OS boots and has been validated in current downstream images.

Additional Context

Test Steps

Merge type

  • Squash merge using "Proposed Commit Message"
  • Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

@tdewey-rpi

Signed-off-by: paulober <paul.oberosler@raspberrypi.com>
@github-actions
Copy link
Copy Markdown

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging blackboxsw, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag blackboxsw to reopen it.)

@github-actions github-actions Bot added the stale-pr Pull request is stale; will be auto-closed soon label Sep 23, 2025
@holmanb
Copy link
Copy Markdown
Member

holmanb commented Sep 25, 2025

Hi @paulober, can you please clarify the need for this PR?

Remove networkd reference and direct NM dependency.
Also schedule the network stage before sysinit.target
like on ubuntu and debian.

This says what the code does, but not why. With service file changes especially, it is really important to understand the reasons for changes like this. What does this fix? Did it work before, and if not how did it get merged? What was done to validate this that would have caught the issue before? Etc, etc.

@holmanb holmanb added incomplete Action required by submitter and removed stale-pr Pull request is stale; will be auto-closed soon labels Sep 25, 2025
@paulober
Copy link
Copy Markdown
Contributor Author

paulober commented Sep 25, 2025

@holmanb Sure no problem. Originally I build cloud-init support for Raspberry Pi OS based on Debian bookworm with maybe potential to also run on bullseye. But that changed this year and cloud-init will now be included from the beginning when Raspberry Pi OS based on Trixie launches. And this of course brought many changes with it. So i'm now updating everything to best support the upcoming release and some tooling around it. This is also the reason for my other PRs. I did not want to combine everything in one PR because that would probably take months to review and adjust like last year. So no specific issue, just improving of boot to screen time and best compatibility with the upcoming release and tooling.

Hope this explains it. Let me know if you have any other questions.

@paulober
Copy link
Copy Markdown
Contributor Author

@blackboxsw Any updates on getting this PR merged?

@github-actions
Copy link
Copy Markdown

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging blackboxsw, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag blackboxsw to reopen it.)

@github-actions github-actions Bot added the stale-pr Pull request is stale; will be auto-closed soon label Oct 30, 2025
@paulober
Copy link
Copy Markdown
Contributor Author

paulober commented Oct 30, 2025

@blackboxsw

@github-actions github-actions Bot removed the stale-pr Pull request is stale; will be auto-closed soon label Oct 31, 2025
@github-actions
Copy link
Copy Markdown

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging blackboxsw, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag blackboxsw to reopen it.)

@github-actions github-actions Bot added the stale-pr Pull request is stale; will be auto-closed soon label Nov 14, 2025
@paulober
Copy link
Copy Markdown
Contributor Author

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging blackboxsw, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag blackboxsw to reopen it.)

@blackboxsw

@blackboxsw
Copy link
Copy Markdown
Collaborator

@paulober anytime we see systemd ordering changes it gives us pause when we don't have specific justification for the issues that are being avoided or resolved as there have been many occasions when systemd ordering cycles get introduced by a seemingly innocuous change which result in one of cloud-init's boot stages getting ejecting from boot goals. These ordering cycle issues show up due to third party software or services that may not be present in default OS images and are hard to discover in base image testing. Hence the request by @holmanb to provide specific justification about the symptom being addressed or motivations for the PR.

Your statement above is vague and doesn't detail the improvement achieved, why RPI OS doesn't care about ordering Before=avahi anymore : So no specific issue, just improving of boot to screen time and best compatibility with the upcoming release and tooling.

The comment above is not something tangible we can point to justify the re-ordering systemd services.

Note as well that Ubuntu Desktop behaves slightly different as a downstream of cloud-init than ubuntu server images as Ubuntu Desktop has to take into account ordering before=NetworkManager.service which is documented in ubuntu's livecd-rootfs systemd dropins/overrides for cloud-init. I only reference this Ubuntu Desktop image systemd ordering for cloud-init to further show the impact of trying to move cloud-init-network service earler in boot and the unexpected relationship with both dbus.socket/service and sysinit.target which we need to take into account on this PR

In this PR's "suggested commit message" can you please describe at least items 1 and 2:

  1. Why doesn't RPI OS consider ordering After NetworkManager or networkd (are they not services RPI OS expects to support?). I see on config/cloud.cfg.tmpl that raspberry-pi-os sets network renderers to [netplan, network-manager] so I expect we'd have some interaction with that NetworkManager backend and we may need ordering after that service becomes available if RaspberryPi OS may attempt to reach out to the network to obtain cloud-init configuration/user-data or meta-data.

  2. Why is Before=avahi-daemon.service ordering dropped?

Also, the scheduling Before=sysinit.target does represent that network backends and possibly dbus or other early-boot services may not yet be fully operational. What is the behavior on RaspberryPI OS if network-config were provided via nocloud configuration seed files in /var/lib/cloud/seed/nocloud/(meta-data,user-data,network-config)

If possible, can we get the output of the following in RaspberryPi OS with your changeset applied?

  • systemd-analyze blame
  • systemd-analyze critical-chain sysinit.target
  • systemd-analyze critical-chain cloud-init-network.service

Or alternatively, just providing the /var/log/cloud-init.log and journalctl -b0 output (which can be collected in a gz with cloud-init collect-logs).

@blackboxsw blackboxsw self-assigned this Nov 17, 2025
@blackboxsw blackboxsw removed the stale-pr Pull request is stale; will be auto-closed soon label Nov 17, 2025
@blackboxsw
Copy link
Copy Markdown
Collaborator

CC: @tdewey-rpi for visibility to the 2 open requests/questions before we can land this PR

@paulober
Copy link
Copy Markdown
Contributor Author

Hi @blackboxsw,

thanks for taking another look at this PR. I’ll try to summarize the background and reasoning.

Last year was the first time I worked with the cloud-init codebase when adding initial Raspberry Pi OS support based on Bookworm. Since cloud-init wasn’t part of any official RPi OS images at the time, real-world testing was very limited. Combined with my inexperience with cloud-init back then, a few mistakes made it into the initial implementation (which is currently upstream).

This year, cloud-init has been included in Raspberry Pi OS based on Trixie, which gave us the opportunity for much broader testing — and with more testers. During that testing we identified issues that simply didn’t surface last year, and this PR corrects those mistakes. All changes included here (and in the other open PRs) are already shipped downstream in the current RPi OS images and have been working reliably.

To address your questions:

1. About NetworkManager/networkd ordering

The ordering rules in last year’s implementation were primarily introduced to ensure that cloud-init ran in the right order, specifically before the Raspberry Pi OS setup wizards (both desktop and console). This was required to make the user setup in cloud-init function correctly in that environment.

Since then, the cloud-init user setup logic for RPi OS has been updated, and there have also been complementary changes outside cloud-init in RPi OS tooling. With these updates in place, we can now align the ordering more closely with the upstream/default expectations. The requirement for correct sequencing still exists, but the workaround from last year is no longer necessary in its previous form. Also there have been downstream fixes for NetworkManager Netplan which result in the config changes provided in this PR being the one that works better with that new env and how we expect the startup to perform.

2. Dropping Before=avahi-daemon.service

If I remember correctly then this ordering was part of the earlier workaround related to the setup wizards or because I thought it would be required for some networking stuff. With the revised user setup flow and improved integration, I confirmed that this dependency is no longer needed, so it was removed to simplify stuff and adjust more to what other distros do.

3. Running before sysinit.target

Yes, this means that some early-boot services may not yet be fully available.
But on RPi OS this ordering behaves correctly.
Not sure if that should make a difference but you can test it with the current Raspberry Pi OS images where the patches are applied downstream to see if it behaves correctly.

4. Logs and systemd-analyze output

At the moment I don’t have the time to rebuild the full test environment from scratch and re-run everything without the downstream patches — especially since all of these changes are already deployed in the current RPi OS images and evaluated in that real-world environment. Given the explanations above and the fact that the ordering logic has been validated through actual downstream usage, these specific logs shouldn’t be required to justify the updated ordering anymore.


I hope this provides a clear explanation of how the ordering evolved and why the updated approach works for RPi OS.

@blackboxsw
Copy link
Copy Markdown
Collaborator

blackboxsw commented Dec 2, 2025

Thank you Paul for the update here. What I'd like to do for posterity is capturing some portion reasoning in the proposed commit message description at the top of this PR which will be used as the git commit message.

I'd suggest something like the following but maybe you would like to word it a bit differently?

fix(raspberry-pi-os): adjust systemd network ordering; drop obsolete deps

Align systemd network ordering with current Raspberry Pi OS behavior.
The previous sequencing was designed around older cloud-init and
setup-wizard requirements, but recent upstream and downstream
changes make those workarounds unnecessary. The updated ordering
matches default expectations and works reliably with newer
NetworkManager/Netplan fixes.

Remove the no-longer-needed Before=avahi-daemon.service dependency,
simplifying the unit and bringing it closer to other distros.

Run the network units before sysinit.target, which is consistent with how
RPi OS boots and has been validated in current downstream images.

@paulober
Copy link
Copy Markdown
Contributor Author

paulober commented Dec 3, 2025

@blackboxsw Done!

@blackboxsw
Copy link
Copy Markdown
Collaborator

Thank you much @paulober for your patience here. Let's work through this list of open PRs this week. I'll get you feedback on others today.

@blackboxsw blackboxsw merged commit 78c68f5 into canonical:main Dec 3, 2025
21 checks passed
blackboxsw pushed a commit to blackboxsw/cloud-init that referenced this pull request Dec 12, 2025
…deps (canonical#6459)

Align systemd network ordering with current Raspberry Pi OS behavior.
The previous sequencing was designed around older cloud-init and
setup-wizard requirements, but recent upstream and downstream
changes make those workarounds unnecessary. The updated ordering
matches default expectations and works reliably with newer
NetworkManager/Netplan fixes.

Remove the no-longer-needed Before=avahi-daemon.service dependency,
simplifying the unit and bringing it closer to other distros.

Run the network units before sysinit.target, which is consistent with how
RPi OS boots and has been validated in current downstream images.

Signed-off-by: paulober <paul.oberosler@raspberrypi.com>
holmanb pushed a commit that referenced this pull request Dec 18, 2025
…deps (#6459)

Align systemd network ordering with current Raspberry Pi OS behavior.
The previous sequencing was designed around older cloud-init and
setup-wizard requirements, but recent upstream and downstream
changes make those workarounds unnecessary. The updated ordering
matches default expectations and works reliably with newer
NetworkManager/Netplan fixes.

Remove the no-longer-needed Before=avahi-daemon.service dependency,
simplifying the unit and bringing it closer to other distros.

Run the network units before sysinit.target, which is consistent with how
RPi OS boots and has been validated in current downstream images.

Signed-off-by: paulober <paul.oberosler@raspberrypi.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

incomplete Action required by submitter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants