cloud-init 19.2.36 fails with python exception "Not all expected physical devices present ..." during bionic image deployment from MAAS

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1846535

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Bionic)', 'cloud-init (Ubuntu Disco)', 'cloud-init (Ubuntu Eoan)']
assignee = oddbloke
assignee_name = Dan Watkins
date_closed = 2019-12-19T23:00:05.207528+00:00
date_created = 2019-10-03T17:19:42.795903+00:00
date_fix_committed = 2019-10-04T15:34:46.009163+00:00
date_fix_released = 2019-12-19T23:00:05.207528+00:00
id = 1846535
importance = critical
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1846535
milestone = None
owner = nikolay.vinogradov
owner_name = Nikolay Vinogradov
private = False
status = fix_released
submitter = nikolay.vinogradov
submitter_name = Nikolay Vinogradov
tags = ['cpe-onsite', 'regression-update', 'sts', 'verification-done', 'verification-done-bionic', 'verification-done-disco', 'verification-done-xenial']
duplicates = [1846661]

Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T17:19:42.795903+00:00

[Impact]

Any instances launched with bridges or bonds in their network configuration will fail to bring up networking.

[Test Case]

Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage.

This results in a maas machine deployment failure and the machine gets released

Procedure:

Alternate steps on a maas machine with a bridge already created

A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP
A2. click deploy -> bionic
A3. Once manual deployment fails go to step 2 below

Alternative 2 juju bootstrap failure on maas

B1: juju bootrap mymaas --no-gui
B2: Once bootstrap fails go to step 2

2. After deployment failure and machine is powered off click on the failed/released node in the MAAS UI
3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI
4. Grab the IP from the interfaces tab
5. ssh ubuntu@ -- cloud-init status --long

Expect failure message

6. Click Exit Rescue Mode on the node in MAAS UI.

7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages:

system_upgrade: {enabled: True}
apt:
  sources:
    proposed.list:
       source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed

8. Repeat step 1 and expect bootstrap success

expect to see MAASDatasource from bootstrapped machine and no errors

9. juju ssh 0 -- cloud-init status --long

Additional verification checks to avoid regression
 - DONE oracle
 - DONE ec2
 - DONE openstack
 - DONE gce
 - DONE azure

• DONE nocloud kvm
• DONE nocloud lxd

[Regression Potential]

The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle.

[Original Report]

Symptoms

After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state:

$ juju ssh 0
ERROR retrieving SSH host keys for "0": keys not found

$ juju machines
Machine State DNS Inst id Series AZ Message
0 pending 172.20.10.125 block-3 bionic AZ3 Deployed
1 pending 172.20.10.124 block-2 bionic AZ2 Deployed
2 pending 172.20.10.126 block-1 bionic AZ1 Deployed
3 pending 172.20.10.127 object-2 bionic AZ1 Deployed
4 pending 172.20.10.128 object-1 bionic AZ2 Deployed
5 pending 172.20.10.129 object-3 bionic AZ3 Deployed

It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails.

We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.

[ubuntu-server-builder]

Launchpad user Michał Ajduk(majduk) wrote on 2019-10-03T17:55:46.025567+00:00

Issue was introduced in Cloud-init v 19.2-36-g059d049c-0ubuntu118.04.1.
It was not present in Cloud-init v. 19.2-24-ge7881d5c-0ubuntu118.04.1

Symptoms:
2019-10-03 13:10:59,100 - init.py[WARNING]: Not all expected physical devices present: {'3c:fd:fe:d5:7a:42', '3c:fd:fe:d5:70:d9', '3c:fd:fe:d5:7a:41', '3c:fd:fe:d5:70:d8', '3c:fd:fe:d5:7a:40', '3c:fd:fe:d5:70:da'}

It seems that following change causes that:

• New upstream snapshot. (LP: #1844334)
  • net: add is_master check for filtering device list

[ubuntu-server-builder]

Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:00:56.483600+00:00

The deployments with bonds seems to be affected. I think since slaves MAC address change after a bond is created cloud-init fails to find the interfaces with original MAC address (they're present in netplan though).

[ubuntu-server-builder]

Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:05:17.152526+00:00

The cloud-init commit that introduced the change: b3a87fc

It affects branches : 19.2-36 and 19.2-25

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2019-10-03T18:19:59.511335+00:00

Confirmed I can successfully install bionic on baremetal with 19.2-36 without seeing this issue directly in MAAS. I'll try now using Juju to deploy the bare metal machine now to see if I can reproduce the failure.

Nikolay do you have access to cloud-init collect-logs (or minimally /var/log/cloud-init.log) on the failed system?

[ubuntu-server-builder]

Launchpad user David Coronel(davecore) wrote on 2019-10-03T18:31:10.178956+00:00

subscribed ~field-high

Issue seems to be affecting deploys with network bonds. Workaround is to use an older cloud image with an older cloud-init version.

[ubuntu-server-builder]

Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:40:15.387335+00:00

cloud-init-output.log attached
Launchpad attachments: cloud-init-output.log

[ubuntu-server-builder]

Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:41:29.250470+00:00

Note mac addresses of bond slaves
Launchpad attachments: configuration with bonds

[ubuntu-server-builder]

Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:42:14.384169+00:00

Note mac addresses again. See also the error in attached cloud-init-output.log.
Launchpad attachments: after removal of the bonds

[ubuntu-server-builder]

Launchpad user Jason Hobbs(jason-hobbs) wrote on 2019-10-03T18:44:49.587069+00:00

sub'd to field critical. this is blocking all of our tests.

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2019-10-03T22:14:17.068098+00:00

EOD update from the cloud-init team:

We've identified what the problem is: in the problematic code path, we have started filtering out network devices that have a "master", which accidentally also includes physical interfaces that are members of a bridge or bond. As suggested in comment #1, it's the "net: add is_master check for filtering device list" commit that introduced the issue. Unfortunately, that commit was a fix for a critical issue (bug 1844191) from another source, so we cannot simply revert it and take our time to land a different fix.

We are currently pursuing three potential options for fixes: (a) completely circumvent the now-incorrect function in the code path that is raising the exception, (b) specifically avoid excluding bridge and bond member interfaces from the list, and (c) match more closely on the type of interface the 'master' check was intended to exclude. In order of preference, we would prefer (c) over (b) over (a). (Because (a) does not obviate the need for something along the lines of (b) or (c) for other code paths, and because we may have to play whack-a-mole with other cases that (b) needs to expand to include.)

We have initial implementations of (a) and (b) which we are testing (though they are not yet ready to land in trunk in their current form). We are investigating (c), but it's likely that we won't reach a conclusion on that investigation fast enough to warrant waiting for its completion before addressing this issue.

[ubuntu-server-builder]

Launchpad user Nivedita Singhvi(niveditasinghvi) wrote on 2019-10-04T09:16:18.960810+00:00

Bumped up the importance due to more reports in production env
hitting this problem (apparently, not fully confirmed).

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2019-10-04T13:03:00.541354+00:00

Status changed to 'Confirmed' because the bug affects multiple users.

[ubuntu-server-builder]

Launchpad user Server Team CI bot(server-team-bot) wrote on 2019-10-04T15:34:44.351035+00:00

This bug is fixed with commit a7d8d03 to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=a7d8d032