sshd_config authorizedkeysfile setting is not honored after v18.5

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1917817

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = 2021-10-16T18:35:58.637323+00:00
date_created = 2021-03-05T00:20:47.237531+00:00
date_fix_committed = 2021-10-16T18:35:58.637323+00:00
date_fix_released = 2021-10-16T18:35:58.637323+00:00
id = 1917817
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1917817
milestone = None
owner = adil-hindistan
owner_name = Adil Hindistan
private = False
status = fix_released
submitter = adil-hindistan
submitter_name = Adil Hindistan
tags = []
duplicates = []

Launchpad user Adil Hindistan(adil-hindistan) wrote on 2021-03-05T00:20:47.237531+00:00

1. Cloud Provider: AWS
2. The only non-default settings we have are:

runcmd:

• [ /bin/chmod, 755, /etc/ssh/auth_keys ]

system_info:
distro: rhel
default_user:
name: ec2-user
lock_passwd: true
gecos: EC2 Default User
groups: [ wheel, adm, systemd-journald ]
sudo: [ "ALL=(ALL) NOPASSWD:ALL" ]
shell: /bin/bash
paths:
cloud_dir: /var/lib/cloud
templates_dir: /etc/cloud/templates
ssh_svcname: sshd

Relevant setting from /etc/ssh/sshd_config:

AuthorizedKeysFile /etc/ssh/auth_keys/%u

3. Unfortunately, policies do not allow me to upload logs but I will try describe the problem below.

I am using RHEL version of cloud-init and between RHEL version upgrades ssh keys stopped working on
EC2. We had no config changes to cloud-init but cloud-init version got upgraded from 18.5 to 19.4

Our investigation showed that authorizedkeysfile in our sshd_config (/etc/ssh/auth_keys/%u) does not get populated when cloud-init is run. Instead, it populates the default user's .ssh directory at /home/{user}/.ssh/authorized_keys.

I also tested with RHEL 8.3 and the same problem exists there as well.
OS | rpm version | Status
RHEL 7.7 | cloud-init-18.5-6.el7.x86_64 | OK
RHEL 7.9 | cloud-init-19.4-7.el7_9.2.x86_64 | Problem
RHEL 8.3 | cloud-init-19.4-11.el8_3.1.noarch | Problem

As a side note, older RHEL 7.5 also works fine, which led me to believe problem was introduced either in 19.1 or in 19.4 update.

Looking into code-base, ssh_util changes looked like a suspect to me but I did not see any tests for ssh_util so I am not sure where exactly the problem is or what the function outputs are.

Any thoughts?

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-03-05T20:29:43.918555+00:00

I believe this was introduced by f1094b1. Specifically, we went from returning auth_key_fn (which was constructed from AuthorizedKeysFile in sshd_config) to returning default_authorizedkeys_file which is unconditionally /home/$user/.ssh/authorized_keys.

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-03-05T20:31:36.876794+00:00

I believe that b0e7381 was intended to address (at least) this, but it ended up being reverted due to introducing a security vulnerability.

So we need to have another go at addressing this, I think: perhaps a targetted fix for this regression, rather than attempting to address this alongside other feature work.

[ubuntu-server-builder]

Launchpad user Adil Hindistan(adil-hindistan) wrote on 2021-03-09T17:45:31.231355+00:00

Dan -
Thanks for your help with confirming the bug. This workaround seems to work fine for now:

runcmd:

• 'for i in ls /home/; do cp -p /home/$i/.ssh/authorized_keys /etc/ssh/auth_keys/$i ; done'

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2021-10-16T18:35:52.192361+00:00

This issue should be fixed with #956 along with the other PRs/bugs referenced in that PR.