Azure IMDS publicKeys contain \r\n which prevents ssh access to vms using cloud-generated ssh keys.

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1910835

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Bionic)', 'cloud-init (Ubuntu Focal)', 'cloud-init (Ubuntu Groovy)', 'cloud-init (Ubuntu Hirsute)']
assignee = None
assignee_name = None
date_closed = 2021-01-12T20:54:27.491085+00:00
date_created = 2021-01-09T03:29:43.495573+00:00
date_fix_committed = 2021-01-12T20:54:27.491085+00:00
date_fix_released = 2021-01-12T20:54:27.491085+00:00
id = 1910835
importance = critical
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1910835
milestone = None
owner = chad.smith
owner_name = Chad Smith
private = False
status = fix_released
submitter = chad.smith
submitter_name = Chad Smith
tags = ['verification-done', 'verification-done-bionic', 'verification-done-focal', 'verification-done-groovy', 'verification-done-xenial']
duplicates = []

Launchpad user Chad Smith(chad.smith) wrote on 2021-01-09T03:29:43.495573+00:00

== Begin SRU Template ==
[Impact]
The previous version of cloud-init used OpenSSL to process the SSH keys provided by the Azure platform. cloud-init 20.4 replaced that code with a more efficient implementation, but one which did not use OpenSSL: this meant that users passing certificates to instances, or users generating SSH keys in Azure's web UI (which inserts \r\n sequences into the public key content), were regressed: their certificates and malformed SSH keys were no longer handled, so they could fail to gain access to newly-launched instances.

This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible

  * Azure: cherry-pick 4f62ae8: Fix regression with handling of IMDS ssh keys
    (#760) (LP: #1910835)

The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3.

[Test Case]

As this is a single commit backport, the cloud-init SRU exception need not apply. An upstream integration test has been written for this issue (https://github.com/canonical/cloud-init/blob/master/tests/integration_tests/bugs/test_lp1910835.py).

A full run of the upstream test suite on Azure will therefore regression test the update generally and test this issue specifically: a log of a test run for each suite will be attached.

[Regression Potential]

The proposed change only modifies code paths used on Azure, specifically to revert to a previous behaviour: users unaffected by the bug should see no change (their keys will get to their instance via a different route), and users affected by the bug would have been unable to access their instances before (so cannot be relying on this behaviour in a way which we could break by fixing it).

[Discussion]
This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the az cli

Any other cloud-platform is not affected by this change.

== End SRU Template ==

  * cherry-pick 4f62ae8: Fix regression with handling of IMDS ssh keys
    (#760) (LP: #1910835)

== Original Description ==

cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys.

To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key.

When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys.

the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out.

Here is the IMDS value provided to cloud-init

cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}'

[{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}]

cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M
9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M
lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M
4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M
6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M
iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M
6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M
S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure

this prevents ssh from actually reading the right key from azure:

$ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys

If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered:

$ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys
3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA)

If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-01-09T03:49:03.305700+00:00

To work around this issue in cloud-init 20.4 on azure, one can launch a VM providing cloud-config userdata which contains the unsplit pubkey content.

userdata can be added in the Advanced tab in Azure portal dashboard or on the az cli via the cmdline parameters --custom-data mycloudconfig.yaml

For the auto-generated key listed in the above bug description the following would be the "right" #cloud-config to provide at VM launch time

#cloud-config
ssh_authorized_keys:

• ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1dk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUWlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh76xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhuiKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU=

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-01-09T03:53:42.922377+00:00

For those who have access to the pem file auto-generated on their local system you can get your public key from the pem file with the following command:

ssh-keygen -f ~/Downloads/my-generated-azure_key.pem -y

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-01-09T15:35:01.176390+00:00

The Azure auto-generated public key can also be pasted directly into the portal dashboard at instance launch time by setting "Use existing public key" in the "SSH Public Key Source" option and copying the output from ssh-keygen -f ~/Downloads/my-generated-azure_key.pem -y into the textarea.

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2021-01-11T23:08:23.158334+00:00

Hello Chad, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2021-01-11T23:09:36.419365+00:00

Hello Chad, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2021-01-11T23:19:38.670798+00:00

Hello Chad, or anyone else affected,

Accepted cloud-init into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~20.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2021-01-11T23:38:31.121007+00:00

Hello Chad, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-01-12T16:48:54.200071+00:00

Launchpad attachments: xenial verification log

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-01-12T16:49:59.660266+00:00

Launchpad attachments: bionic verification log

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-01-12T16:50:17.376629+00:00

Launchpad attachments: focal verification log

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-01-12T16:50:34.313233+00:00

Launchpad attachments: groovy verification log

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-01-12T17:27:34.877335+00:00

I've just attached the output of our verification testing runs for each series.

Each series exhibits two test failures, neither of which is material to the change in question (and both of which reproduce against the cloud-init currently in the archive).

test_datasource_rbx_no_stacktrace is testing a non-Azure datasource but doesn't include the appropriate marks to exclude it from an Azure test run. As such, it's an expected failure (which we will fix upstream: https://bugs.launchpad.net/cloud-init/+bug/1911230).

TestSeedRandomData.test_seed_random_data is implemented incorrectly for Azure. cc_seed_random behaves differently on platforms which provide random data to cloud-init (such as Azure), and that is not taken into account in the test code (or described in the documentation). https://bugs.launchpad.net/cloud-init/+bug/1911227 captures addressing the documentation and test issue.

Given that all other tests pass, including the one for this bug, I consider verification of these cloud-init SRUs to be complete.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2021-01-12T18:55:54.794372+00:00

This bug was fixed in the package cloud-init - 20.4-0ubuntu1~20.10.2

---

cloud-init (20.4-0ubuntu1~20.10.2) groovy; urgency=medium

• cherry-pick 4f62ae8: Fix regression with handling of IMDS ssh keys
  (Fix regression by forcing no IMDS ssh keys #760) (LP: #1910835)

-- Daniel Watkins oddbloke@ubuntu.com Mon, 11 Jan 2021 17:10:13 -0500