cloud-init in impish makes /home/ubuntu/.ssh root.root

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1940233

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Bionic)', 'cloud-init (Ubuntu Focal)', 'cloud-init (Ubuntu Hirsute)', 'cloud-init (Ubuntu Impish)']
assignee = None
assignee_name = None
date_closed = 2021-08-23T20:16:44.132864+00:00
date_created = 2021-08-17T08:42:10.109458+00:00
date_fix_committed = 2021-08-21T03:32:21.196115+00:00
date_fix_released = 2021-08-23T20:16:44.132864+00:00
id = 1940233
importance = critical
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1940233
milestone = None
owner = paelzer
owner_name = Christian Ehrhardt 
private = False
status = fix_released
submitter = paelzer
submitter_name = Christian Ehrhardt 
tags = ['rls-ii-incoming']
duplicates = []

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-08-17T08:42:10.109458+00:00

Hi,
I got to this by my systems complaining to be unable to do ssh-keygen
after deployment. Example:
$ uvt-kvm ssh --insecure impish-kvm 'ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N '''''''
Saving key "/home/ubuntu/.ssh/id_rsa" failed: Permission denied

I found that is due to permissions after guest spawning:
/home/ubuntu/.ssh changed

Old:
drwx------ 2 ubuntu ubuntu 4096 Aug 17 08:20 .ssh/

New:
drwxr-xr-x 2 root root 4096 Aug 17 08:17 .ssh/

That beaks later things like ssh-keygen.

uvt-kvm only does instruct cloud-init to place a key.
This uses ssh_authorized_keys from
https://cloudinit.readthedocs.io/en/latest/topics/modules.html?highlight=ssh_authorized_keys#authorized-keys

Checked a few guests:
I've seen this on

• impish x86
• impish s390x

I've not seen this on

• bionic
• focal
• impish

You might say - wait a minute impish in both lists.
But it is the date:

Bad
com.ubuntu.cloud.daily:server:21.10:amd64 20210815
cloud-init 21.2-69-g65607405-0ubuntu1

Good
com.ubuntu.cloud.daily:server:21.10:amd64 20210706
cloud-init 21.2-3-g899bfaa9-0ubuntu2

And either this cloud-init version is broken or the underlying new impish image.
I mounted the underlying cloud-image (without customization by cloud-init)
and found that /home is empty (true for all those images).

So to me that seems to be an issue in the new cloud-init that now is in
those images.

Steps to reproduce

if your host has no keys to push to the guest run ssh-keygen

sync the latest broken images

$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=impish

spawn guest

$ uvt-kvm create --password=ubuntu i release=impish arch=amd64 label=daily

wait for it and check the permissions

$ uvt-kvm wait i
$ uvt-kvm ssh i "ls -laF /home/ubuntu/"
drwxr-xr-x 2 root root 4096 Aug 17 08:17 .ssh/

[ubuntu-server-builder]

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-08-17T08:45:52.648973+00:00

Collected logs according to https://cloudinit.readthedocs.io/en/latest/topics/bugs.html
(automatic collection seems broken - see 1940235)
Launchpad attachments: cloud-init.tar

[ubuntu-server-builder]

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-08-17T09:10:14.880252+00:00

Since it might affect the ability to log into a new spawned system I think this could be release-critical for Impish. Therefore I flagged it like that for now - please feel free to downgrade once the root cause is known and you think a lower rating is appropriate.

[ubuntu-server-builder]

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-08-17T09:16:06.319361+00:00

paride found this change:
00dbaf1

It say in the message:
"the directories in the path that do not exist must be root owned and with permission 755."

That is wrong, it needs to be user owned or the user can later on not change things anymore missing write permissions on his own ~/.ssh.

[ubuntu-server-builder]

Launchpad user Paride Legovini(paride) wrote on 2021-08-17T10:10:38.810472+00:00

Thanks Christian for filing this.

Yes there's where we introduced the problem. I think we are missing a check in check_create_path() so that directories under the home directory are created as user-owned. Something like:

newparent_uid = root_pwent.pw_uid
newparent_gid = root_pwent.pw_gid
if parent.startswith(home_folder):
    newparent_uid = user_pwent.pw_uid
    newparent_gid = user_pwent.pw_gid

and then:

util.chownbyid(parent_folder, newparent_uid, newparent_gid)

Maybe we should also check/test how check_create_path() behaves when the user's home directory does not exist, or it is not user-owned and user-readable-and-writable. I'd say that check_create_path() should return False, as it is not this module's duty to create the home directory.

[ubuntu-server-builder]

Launchpad user Paride Legovini(paride) wrote on 2021-08-17T15:08:00.236246+00:00

Turns out this bug is also the reason why the HA test jobs are failing, as they run on Impish and and an extra ssh key is added to the instances under test. The failure mode is the same Christian hit.

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2021-08-17T15:08:59.843318+00:00

PR up: #984

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-08-21T03:33:07.728457+00:00

PR landed, expected upload to Ubuntu Impish 21.10 on Monday Aug 23

[ubuntu-server-builder]

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-08-23T06:49:04.791388+00:00

Thank you all for the prompt handling!

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2021-08-23T20:16:45.089425+00:00

This bug is believed to be fixed in cloud-init in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2021-08-24T05:08:18.601729+00:00

This bug was fixed in the package cloud-init - 21.3-1-g6803368d-0ubuntu1

---

cloud-init (21.3-1-g6803368d-0ubuntu1) impish; urgency=medium

• New upstream snapshot.
  • testing: Fix ssh keys integration test (testing: Fix ssh keys integration test #992)
  • Release 21.3 (Release 21.3 #993) (LP: #1940839)
  • Azure: During primary nic detection, check interface status continuously
    before rebinding again (Azure: During primary nic detection, check interface status continuously before rebinding again #990) [aswinrajamannar]
  • Fix home permissions modified by ssh module (SC-338) (Fix home permissions modified by ssh module (SC-338) #984)
    (LP: #1940233)
  • Add integration test for sensitive jinja substitution (Add integration test for sensitive jinja substitution (SC-352) #986)
  • Ignore hotplug socket when collecting logs (Ignore hotplug socket when collecting logs (SC-345) #985) (LP: #1940235)
  • testing: Add missing mocks to test_vmware.py (testing: Add missing mock to test_vmware.py (SC-348) #982)
  • add Zadara Edge Cloud Platform to the supported clouds list (added Zadara Edge Cloud Platform #963)
    [sarahwzadara]
  • testing: skip upgrade tests on LXD VMs (testing: skip upgrade tests on LXD VMs #980)

-- James Falcon james.falcon@canonical.com Mon, 23 Aug 2021 16:53:23 -0500

[ubuntu-server-builder]

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-08-25T05:39:54.991868+00:00

While this is important and fixed in the SRU upload of bug 1940871 - the bad code never reached Bionic/Focal/Hirsute as it was in the interim 21.3 versions. Therefore the bug starte for these releases is invalid (it isn't there now, and won't be added in the ongoing SRU).
Setting the states to invalid.