DNS failure while trying to fetch user-data

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #2008952

Launchpad details

affected_projects = ['netplan', 'subiquity', 'livecd-rootfs (Ubuntu)', 'systemd (Ubuntu)']
assignee = chad.smith
assignee_name = Chad Smith
date_closed = 2023-04-12T03:53:12.154194+00:00
date_created = 2023-03-02T00:09:30.336453+00:00
date_fix_committed = 2023-04-12T03:53:12.154194+00:00
date_fix_released = 2023-04-12T03:53:12.154194+00:00
id = 2008952
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/2008952
milestone = None
owner = ken-vandine
owner_name = Ken VanDine
private = False
status = fix_released
submitter = ken-vandine
submitter_name = Ken VanDine
tags = ['foundations-todo']
duplicates = []

Launchpad user Ken VanDine(ken-vandine) wrote on 2023-03-02T00:09:30.336453+00:00

In testing netboot + autoinstall of the new ubuntu desktop subiquity based installer for 23.04 I found cloud-init is failing to retrieve user-data because it can't resolved the hostname in the URL. This same configuration does work for 22.04 based subiquity, so seems a regression.

From the ipxe config:

imgargs vmlinuz initrd=initrd
ip=dhcp
iso-url=http://cdimage.ubuntu.com/daily-live/pending/lunar-desktop-amd64.iso
fsck.mode=skip
layerfs-path=minimal.standard.live.squashfs
autoinstall
'ds=nocloud-net;s=http://boot.linuxgroove.com/ubuntu/23.04/' \

That fails, but if we replace boot.linuxgroove.com with the IP it works.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2023-03-07T04:11:05.741103+00:00

There are a couple of significant issues that may be leading to this symptom:

1. On Desktop live server images, something is laying down a competing file for management of the wired NIC on my laptop:

/etc/netlan/01-network-manager-all.yaml

Let NetworkManager manage all devices on this system

network:
version: 2
renderer: NetworkManager

Yet /etc/cloud/cloud.cfg specifies priority order of network renderers to prefer using netplan insteaad of NetworkManager:
networks:
renderers: ['netplan', 'eni', 'sysconfig']
activators: ['netplan', 'eni', 'network-manager', 'networkd']

This causes cloud-init to also write out /etc/netplan/50-cloud-init.yaml which also tries to claim management of enp0s31f6 under systemd-networkd:

network:
version: 2
ethernets:
enp0s21f6:
dhcp4: true
match:
macaddress: 6c:24:08:9e:54:e7
set-name: enp0s31f6

The result is undetermined behavior in early boot as NetworkManager and systemd-networkd fight over who owns the wired device. This could be the resulting DNS lookup failures we are seeing.

Resolution: I think desktop images want to avoid telling cloud-init to render networking using netplan. I believe we are missing this proposed fix to livecd rootfs:

https://code.launchpad.net/~dbungert/livecd-rootfs/+git/livecd-rootfs/+merge/427445

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2023-03-07T04:19:23.423043+00:00

journalctl on a failed laptop is showwing a lot of throttling logs from NetworkManager such as :

stat change: unavilable -> disconnected (reason 'carrier changed', sys-iface-state: 'managed')

This makes be believe that networkd and NetworkManager are attempting to grab management of the NIC multiple times throughout boot thereby rendering the network and carrier down.

I see cloud-init reflecting the 'down' carrier state of en0s31f6 in the 'init' timeframe too by

Cloud-init v. 23.1.1-0ubuntu1 running 'init' at Tue, 07 Mar 2023 03:17:56 +0000. Up 62.47 seconds.
ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++
ci-info: +-----------+-------+-----------+-----------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +-----------+-------+-----------+-----------+-------+-------------------+
ci-info: | enp0s31f6 | False | . | . | . | 6c:24:08:9e:54:e6 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: | wlp0s20f3 | False | . | . | . | 38:7a:0e:2d:d0:bb |
ci-info: +-----------+-------+-----------+-----------+-------+-------------------+

2. The second interesting configuration 'error' to be aware of is that netplan doesn't like the 644 permissions on the rendered /etc/netplan/01-network-manager-all.yaml.

** (process:2015): WARNING **: 03:18:08.594: Permissions for /etc/netplan/01-network-manager-all.yaml are too open. Netplan configuration should NOT be accessible by others.

Full log paste: https://dpaste.com/APGZH46PN

[ubuntu-server-builder]

Launchpad user Dan Bungert(dbungert) wrote on 2023-03-07T20:00:26.776415+00:00

Looking at the desktop ISOs, while the system is configured with Netplan to use NetworkManager, there seems to be some conflict here where systemd-networkd is trying to interact with the device. Some feedback would be appreciated. The practical result is that early boot cloud-init is not able to fetch data that it should be able to retrieve.

[ubuntu-server-builder]

Launchpad user Lukas Märdian(slyon) wrote on 2023-03-08T15:56:12.617068+00:00

I disagree with comment #1

/etc/netlan/01-network-manager-all.yaml and /etc/netplan/50-cloud-init.yaml should not be in conflict with each other, but will be merged by Netplan to produce the following configuration:

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp0s21f6:
      dhcp4: true
      match:
        macaddress: 6c:24:08:9e:54:e7
      set-name: enp0s31f6

Which should render its config in /run/NetworkManager/system-connections instead of /run/systemd/network (which would be the case if the global renderer setting wasn't changed).

So I wonder what is the exact sequence of events for running the Netplan generator (by systemd), installing the 50-cloud-init.yaml file (by cloud-init), installing the 01-network-manager-all.yaml file (by the installer?), executing netplan apply (by cloud-init).

Also, what are the artifacts generated by Netplan in /run/systemd/network/ and /run/NetworkManager/system-connections/ ?

I assume some kind of race condition, where:

• cloud-init installs 50-cloud-init.yaml
• Netplan generator being run (as a systemd generator during early boot)
• systemd-network is now controlling the interface
• the installer putting the 01-network-manager-all.yaml file
• cloud-init calling netplan apply at runtime
• network configuration is being changed and NetworkManager is supposed to take over control
• Interfaces still managed by networkd from the earlier stage, and therefore getting into conflict

[ubuntu-server-builder]

Launchpad user Dan Bungert(dbungert) wrote on 2023-03-08T17:50:02.522200+00:00

So while this bug talks about netboot, I think it isn't necessary to involve that for testing purposes. I suggest simply booting the desktop daily-live with the nocloud kernel command line listed above and seeing how things go. Are the various network tools doing what we expect?

Note that gnome-boxes is a convenient tool for running this ISO in a VM, it seems to have a better video driver than what kvm would suggest by default. It's still possible to give it kernel command line with config like:
/srv/iso/lunar/vmlinuz
/srv/iso/lunar/initrd
autoinstall layerfs-path=minimal.standard.live.squashfs ds=nocloud-net;s=http://boot.linuxgroove.com/ubuntu/23.04/

[ubuntu-server-builder]

Launchpad user Dan Bungert(dbungert) wrote on 2023-03-08T18:07:06.913458+00:00

So I wonder what is the exact sequence of events for running the Netplan generator (by systemd), installing the 50-cloud-init.yaml file (by cloud-init), installing the 01-network-manager-all.yaml file (by the installer?), executing netplan apply (by cloud-init).

By the time Subiquity has started, the bad interaction has already taken place.
In this nocloud case, cloud-init should have been able to retrieve the user-data and other things, that failed. So at Subiquity start time, we ask for the autoinstall and get an empty answer. An empty answer is quite common - that is what happens in a normal interactive install - so it's not immediately obvious that a misbehavior has taken place.

[ubuntu-server-builder]

Launchpad user Lukas Märdian(slyon) wrote on 2023-03-09T16:17:33.919786+00:00

01-network-manager-all.yaml seems to be shipped by livecd-rootfs

[ubuntu-server-builder]

Launchpad user Danilo Egea Gondolfo(danilogondolfo) wrote on 2023-03-09T21:51:14.925816+00:00

It doesn't seem to be caused by a race between networkd and NetworkManager.

I reproduced the issue with qemu here and I see the name resolution failure happening few seconds before NetworkManager started.

From /var/log/cloud-init.log:

2023-03-09 19:17:58,443 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloudNet'> failed
Traceback (most recent call last):
...
cloudinit.url_helper.UrlError: HTTPConnectionPool(host='boot.linuxgroove.com', port=80): Max retries exceeded with url: /ubuntu/23.04/meta-data (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f650b663610>:
Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

From systemd's journal
Mar 09 19:18:01 ubuntu systemd[1]: Starting NetworkManager.service - Network Manager...

(edit)

resolved wasn't running as well

2023-03-09T19:18:01.712656+00:00 ubuntu systemd[1]: Starting systemd-resolved.service - Network Name Resolution...

but it seems my instance had a nameserver:

2023-03-09 19:17:58,500 - stages.py[INFO]: Applying network configuration from initramfs bringup=True: {'config': [{'type': 'physical', 'name': 'ens3', 'subnets': [{'type': 'dhcp', 'control': 'manual', 'netmask': '255.255.255.0', 'broadc
ast': '10.0.2.255', 'gateway': '10.0.2.2', 'dns_nameservers': ['10.0.2.3']}], 'mac_address': '52:54:00:12:34:56'}], 'version': 1}

[ubuntu-server-builder]

Launchpad user Danilo Egea Gondolfo(danilogondolfo) wrote on 2023-03-09T22:29:20.442624+00:00

As /etc/resolv.conf is a symlink, is it possible that the nameservers received via DHCP in the early boot stages are never stored in /etc/resolv.conf?

cloud-init tries to resolve that address before resolved is started and there is nothing at /etc/resolv.conf.

Does that make sense?

[ubuntu-server-builder]

Launchpad user Lukas Märdian(slyon) wrote on 2023-03-14T10:13:33.147997+00:00

Maybe we need some additional systemd service ordering, to make systemd-resolved start before calling into the DHCP client, so that it can properly receive the DNS servers.

[ubuntu-server-builder]

Launchpad user Danilo Egea Gondolfo(danilogondolfo) wrote on 2023-03-14T13:57:14.111431+00:00

Launchpad attachments: systemd_journal.txt

[ubuntu-server-builder]

Launchpad user Danilo Egea Gondolfo(danilogondolfo) wrote on 2023-03-14T13:57:29.059302+00:00

Launchpad attachments: cloud-init.log

[ubuntu-server-builder]

Launchpad user Danilo Egea Gondolfo(danilogondolfo) wrote on 2023-03-14T13:57:40.892864+00:00

Launchpad attachments: timeline.txt

[ubuntu-server-builder]

Launchpad user Danilo Egea Gondolfo(danilogondolfo) wrote on 2023-03-14T14:01:23.195758+00:00

I attached some logs that might be useful. After checking it again I realized that the syslog timestamps are a little off when compared to the systemd journal.

As shown in the timeline.txt (attached), cloud-init and systemd-resolved are starting at the same time.

So the name resolution might not be working yet when cloud-init needs it.