SCOS: dbus-broker.service `Failed to set up mount namespacing: /run/systemd/unit-root/dev: Read-only file system`

[travier]

Split from #909

From #905

[HuijingHei]

But after qcow2 VM booted, dbus-broker.service works. Is there any difference from pxe-install?

$ rpm -q selinux-policy dbus-broker
selinux-policy-34.1.37-1.el9.noarch
dbus-broker-28-5.el9.x86_64

$ systemctl status dbus-broker
● dbus-broker.service - D-Bus System Message Bus
     Loaded: loaded (/usr/lib/systemd/system/dbus-broker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-07-26 13:17:26 UTC; 39min ago
TriggeredBy: ● dbus.socket
       Docs: man:dbus-broker-launch(1)
   Main PID: 1435 (dbus-broker-lau)
      Tasks: 2 (limit: 5520)
     Memory: 1.9M
        CPU: 64ms
     CGroup: /system.slice/dbus-broker.service
             ├─1435 /usr/bin/dbus-broker-launch --scope system --audit
             └─1441 dbus-broker --log 4 --controller 9 --machine-id e3f19a9f0de54ebebcaf3301cd636d4d --max-bytes 536870912 --max-fds 4096 --max-matches 13107>

Jul 26 13:17:26 localhost systemd[1]: Starting D-Bus System Message Bus...
Jul 26 13:17:26 localhost dbus-broker-launch[1435]: Looking up NSS user entry for 'dbus'...
Jul 26 13:17:26 localhost dbus-broker-launch[1435]: NSS returned NAME 'dbus' and UID '81'
Jul 26 13:17:26 localhost systemd[1]: Started D-Bus System Message Bus.
Jul 26 13:17:26 localhost dbus-broker-lau[1435]: Ready

[HuijingHei]

Build c9s locally, and run kola testiso -S --scenarios pxe-install --console, get failed service coreos-installer, which is caused by NetworkManager(but does not show in Failed Units), which is caused by dbus-broker

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Press Enter for maintenance
(or press Control-D to continue): 
[systemd]
Failed Units: 4
  coreos-installer.service
  dbus-broker.service
  systemd-network-generator.service
  dbus.socket

[root@localhost ~]# systemctl status coreos-installer.service | cat
× coreos-installer.service - CoreOS Installer
     Loaded: loaded (/usr/lib/systemd/system/coreos-installer.service; static)
     Active: failed (Result: exit-code) since Wed 2022-07-27 12:25:09 UTC; 25s ago
    Process: 1085 ExecStart=/usr/libexec/coreos-installer-service (code=exited, status=1/FAILURE)
   Main PID: 1085 (code=exited, status=1/FAILURE)
        CPU: 56ms

Jul 27 12:25:08 localhost.localdomain systemd[1]: Starting CoreOS Installer...
Jul 27 12:25:08 localhost.localdomain coreos-installer-service[1085]: coreos-installer install /dev/vda --ignition-url http://192.168.76.2:43111/config.ign -e
Jul 27 12:25:09 localhost.localdomain coreos-installer-service[1097]: Error: downloading source Ignition config http://192.168.76.2:43111/config.ign
Jul 27 12:25:09 localhost.localdomain coreos-installer-service[1097]: Caused by:
Jul 27 12:25:09 localhost.localdomain coreos-installer-service[1097]:     Read-only file system (os error 30)
Jul 27 12:25:09 localhost.localdomain systemd[1]: coreos-installer.service: Main process exited, code=exited, status=1/FAILURE
Jul 27 12:25:09 localhost.localdomain systemd[1]: coreos-installer.service: Failed with result 'exit-code'.
Jul 27 12:25:09 localhost.localdomain systemd[1]: Failed to start CoreOS Installer.
Jul 27 12:25:09 localhost.localdomain systemd[1]: coreos-installer.service: Triggering OnFailure= dependencies.

[root@localhost ~]# systemctl status dbus-broker.service | cat
× dbus-broker.service - D-Bus System Message Bus
     Loaded: loaded (/usr/lib/systemd/system/dbus-broker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2022-07-27 12:25:08 UTC; 1min 4s ago
   Duration: 22.776s
TriggeredBy: × dbus.socket
       Docs: man:dbus-broker-launch(1)
    Process: 1083 ExecStart=/usr/bin/dbus-broker-launch --scope system --audit (code=exited, status=226/NAMESPACE)
   Main PID: 1083 (code=exited, status=226/NAMESPACE)
        CPU: 0

Jul 27 12:25:08 localhost.localdomain systemd[1]: Starting D-Bus System Message Bus...
Jul 27 12:25:08 localhost.localdomain systemd[1083]: dbus-broker.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Read-only file systm
Jul 27 12:25:08 localhost.localdomain systemd[1083]: dbus-broker.service: Failed at step NAMESPACE spawning /usr/bin/dbus-broker-launch: Read-only file system
Jul 27 12:25:08 localhost.localdomain systemd[1]: dbus-broker.service: Main process exited, code=exited, status=226/NAMESPACE
Jul 27 12:25:08 localhost.localdomain systemd[1]: dbus-broker.service: Failed with result 'exit-code'.
Jul 27 12:25:08 localhost.localdomain systemd[1]: Failed to start D-Bus System Message Bus.
Jul 27 12:25:08 localhost.localdomain systemd[1]: dbus-broker.service: Start request repeated too quickly.
Jul 27 12:25:08 localhost.localdomain systemd[1]: dbus-broker.service: Failed with result 'exit-code'.
Jul 27 12:25:08 localhost.localdomain systemd[1]: Failed to start D-Bus System Message Bus.

[root@localhost ~]# ls -lZd /dev
drwxr-xr-x. 17 root root system_u:object_r:device_t:s0 2840 Jul 27 12:14 /dev

[HuijingHei]

Seems dbus-broker.service works well in installed os, but failed in live.
cc @travier

[travier]

We already have https://github.com/openshift/os/blob/master/manifest-c9s.yaml#L114. Maybe we need something else or undo it.

[HuijingHei]

Here are full logs, hope they can be helpful
console.txt
journal.txt

[HuijingHei]

Start live.iso (embed console=ttyS0 and autologin ign), can reproduce the issue

virt-install --name=test --vcpus=2 --ram=4096 \
--graphics=none --cdrom /home/fedora/data/c9s/builds/412.91.202207260751-0/x86_64/custom1.iso \
--disk=/home/fedora/data/test.qcow2 --os-variant centos8

[core@localhost ~]$ systemctl status NetworkManager
Failed to connect to bus: Connection refused
$ journalctl -xe | grep dbus-broker
Jul 28 05:31:36 localhost systemd[2385]: dbus-broker.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Read-only file system
Jul 28 05:31:36 localhost systemd[2385]: dbus-broker.service: Failed at step NAMESPACE spawning /usr/bin/dbus-broker-launch: Read-only file system

[HuijingHei]

Build rhel9 locally with systemd-250.8-1.fc36.x86_64, can not reproduce the issue

kola testiso -S --scenarios pxe-install --output-dir tmp/kola-metal
Testing scenarios: [pxe-install]
Ignoring verification of signature on metal image
PASS: pxe-install (bios + metal) (4m36.122s)

[HuijingHei]

https://bugzilla.redhat.com/show_bug.cgi?id=2111863

[cgwalters]

One thing that I'd highlight here is our automated test suite is correctly finding critical bugs! In particular, our coverage of the ISO and PXE flow is so important because otherwise we'd only find these types of things when people provision real bare metal systems, and that's expensive.

[HuijingHei]

Maybe can close this as ee4fba5 PR is merged, also close BZ#2111863 as notabug

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale