Skip to content

cmd/initContainer: Do not try to remount /etc/machine-id as ro#1354

Merged
debarshiray merged 2 commits intocontainers:mainfrom
alatiera:altiera/machineid-ro-mount
Aug 22, 2023
Merged

cmd/initContainer: Do not try to remount /etc/machine-id as ro#1354
debarshiray merged 2 commits intocontainers:mainfrom
alatiera:altiera/machineid-ro-mount

Conversation

@alatiera
Copy link
Copy Markdown
Contributor

Followup to 1cc9e07

Sometimes the parent location might be mounted with nosuid,nodev,noexec and trying to remount it as ro would remove those and thus fails.

See commit mentioned above for more details.

#911

@alatiera
Copy link
Copy Markdown
Contributor Author

Here's the error I was getting ftr:

level=debug msg="Resolved /etc/localtime to /run/host/usr/share/zoneinfo/Europe/Athens"
level=debug msg="Creating regular file /etc/machine-id"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
mount: /etc/machine-id: filesystem was mounted, but any subsequent operation failed: Unknown error 5005.
Error: failed to bind /etc/machine-id to /run/host/etc/machine-id

and here's what my mounts look like:

TARGET                        SOURCE           FSTYPE          OPTIONS
/                             /dev/dm-1        btrfs           rw,nodev,relatime,space_cache=v2,subvolid=5,subvol=/
├─/tmp                        tmpfs            tmpfs           rw,nosuid,nodev,nr_inodes=1048576
├─/dev                        devtmpfs         devtmpfs        rw,nosuid,size=4096k,nr_inodes=8215431,mode=755
│ ├─/dev/hugepages            hugetlbfs        hugetlbfs       rw,nosuid,nodev,relatime,pagesize=2M
│ ├─/dev/mqueue               mqueue           mqueue          rw,nosuid,nodev,noexec,relatime
│ ├─/dev/shm                  tmpfs            tmpfs           rw,nosuid,nodev
│ └─/dev/pts                  devpts           devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
├─/sys                        sysfs            sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/selinux           selinuxfs        selinuxfs       rw,nosuid,noexec,relatime
│ ├─/sys/kernel/tracing       tracefs          tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/debug         debugfs          debugfs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security      securityfs       securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup            cgroup2          cgroup2         rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/firmware/efi/efivars efivarfs         efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf               bpf              bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ └─/sys/fs/fuse/connections  fusectl          fusectl         rw,nosuid,nodev,noexec,relatime
├─/proc                       proc             proc            rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc  systemd-1        autofs          rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1150
├─/run                        tmpfs            tmpfs           rw,nosuid,nodev,size=13154028k,nr_inodes=819200,mode=755
│ ├─/run/user/1000            tmpfs            tmpfs           rw,nosuid,nodev,relatime,size=6577012k,nr_inodes=1644253,mode=700,uid=1000,gid=1000
│ │ ├─/run/user/1000/gvfs     gvfsd-fuse       fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
│ │ └─/run/user/1000/doc      portal           fuse.portal     rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
│ └─/run/media/alatiera/disk  /dev/sda5        squashfs        ro,nosuid,nodev,relatime,errors=continue
├─/boot                       systemd-1        autofs          rw,relatime,fd=48,pgrp=1,timeout=120,minproto=5,maxproto=5,direct,pipe_ino=8369
├─/home                       /dev/mapper/home btrfs           rw,nodev,relatime,space_cache=v2,subvolid=5,subvol=/
└─/usr                        /dev/mapper/usr  squashfs        ro,relatime,errors=continue
  └─/usr                      sysext           overlay         ro,nodev,relatime,lowerdir=/run/systemd/sysext/meta/usr:/run/systemd/sysext/extensions/toolbox/usr:/run/systemd/sysext/extensions/shell_dot

And a link to the issue when found out

@alatiera
Copy link
Copy Markdown
Contributor Author

I am not entirely sure about the fix, since /etc/machine-id is a bit different and gets mounted in /run/host but seems to be working.

@alatiera alatiera force-pushed the altiera/machineid-ro-mount branch from 1fc4c7b to 24e5365 Compare August 20, 2023 05:22
@debarshiray
Copy link
Copy Markdown
Member

That's odd. The CentOS Stream 9 and Fedora tests on Zuul CI didn't run.

@debarshiray
Copy link
Copy Markdown
Member

recheck

@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/e593247b73b548ba860bbc722c271d46

unit-test RETRY_LIMIT in 32s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 01s
unit-test-restricted RETRY_LIMIT in 35s
system-test-fedora-rawhide RETRY_LIMIT in 32s
✔️ system-test-fedora-38 SUCCESS in 27m 50s
✔️ system-test-fedora-37 SUCCESS in 27m 19s

Copy link
Copy Markdown
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down, @alatiera ! The patch looks good to me.

I have only one question about the root cause. It seems to me that on this particular build of GNOME OS, the entire / on the host is mounted with ...,nodev,.... Since /etc isn't a separate mount point, /etc is also nodev, and this runs into problems with the ro flag as it's used here. ie., mount --rbind -o ro /run/host/etc/machine-id /etc/machine-id tries to remove that nodev and fails. Did I understand that right?

@alatiera
Copy link
Copy Markdown
Contributor Author

That's my understanding as well. Only /usr is a separate mount in this build yes. Both /var and /etc are just under /

containers#1330

Signed-off-by: Nieves Montero <nmontero@redhat.com>
@debarshiray debarshiray force-pushed the altiera/machineid-ro-mount branch from 24e5365 to 3ed0af5 Compare August 22, 2023 20:31
On new builds of GNOME OS [1], the host's / is mounted with 'nodev,...'
and those flags are also inherited by /etc because it's not a separate
mount point.  This leads to the same problem with /etc/machine-id that
was seen before with /var/lib/flatpak, /var/lib/systemd/coredump and
/var/log/journal [2].

Therefore, use the same approach [2] to handle /etc/machine-id.

[1] https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues/718

[2] Commit 1cc9e07
    containers@1cc9e07b7c36fe9f
    containers#1340

containers#911
containers#1354

Signed-off-by: Jordan Petridis <jordan@centricular.com>
@debarshiray debarshiray force-pushed the altiera/machineid-ro-mount branch from 3ed0af5 to 219f5b4 Compare August 22, 2023 20:33
@debarshiray
Copy link
Copy Markdown
Member

debarshiray commented Aug 22, 2023

That's my understanding as well. Only /usr is a separate mount in this build yes. Both /var and /etc are just under /

Thanks for confirming that! I took the liberty to add that to the commit message, because I will surely mix up the details very soon. :)

Let's wait for the CI to finish.

Note that the tests run on Fedora Rawhide nodes are failing because of the same reasons as in #1344 and #1331 , and the root cause appears to be rsync: https://bugzilla.redhat.com/show_bug.cgi?id=2229654 So, I am going to temporarily ignore them.

@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/b41168d20aae4c16a5ab850fecb8a9e0

unit-test RETRY_LIMIT in 33s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 05s
unit-test-restricted RETRY_LIMIT in 32s
system-test-fedora-rawhide RETRY_LIMIT in 32s
✔️ system-test-fedora-38 SUCCESS in 28m 49s
✔️ system-test-fedora-37 SUCCESS in 29m 13s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants