[BUG] [AIO] failure to boot with podman / btrfs filesystem (fedora 35 and podman on osx)

[mangelajo]

What happened:

microshift/kubelet fails to start

What you expected to happen:

MicroShift starting correctly

How to reproduce it (as minimally and precisely as possible):

Try to start microshift AIO on podman, under a fedora core 35 system with encrypted btrfs

Environment:

• Microshift version (use microshift version):
• Hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

Relevant Logs

Mar 09 13:57:31 a2fec82806c3 microshift[1429]: I0309 13:57:31.303535    1429 state_mem.go:36] "Initialized new in-memory state store"
Mar 09 13:57:31 a2fec82806c3 microshift[1429]: I0309 13:57:31.303680    1429 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Mar 09 13:57:31 a2fec82806c3 microshift[1429]: I0309 13:57:31.303692    1429 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Mar 09 13:57:31 a2fec82806c3 microshift[1429]: I0309 13:57:31.303698    1429 policy_none.go:44] "None policy: Start"
Mar 09 13:57:31 a2fec82806c3 microshift[1429]: W0309 13:57:31.303720    1429 fs.go:588] stat failed on /dev/mapper/luks-4306042c-0955-4e0c-bbcf-770e8903dfc7 with error: no such file or directory
Mar 09 13:57:31 a2fec82806c3 microshift[1429]: E0309 13:57:31.303737    1429 kubelet.go:1389] "Failed to start ContainerManager" err="failed to get rootfs info: failed to get device for dir \"/var/lib/kubelet\": could not find device with major: 0, minor: 35 in cached partitions map"
Mar 09 13:57:31 a2fec82806c3 systemd[1]: microshift.service: Main process exited, code=exited, status=1/FAILURE
Mar 09 13:57:31 a2fec82806c3 systemd[1]: microshift.service: Failed with result 'exit-code'.
Mar 09 13:57:31 a2fec82806c3 systemd[1]: microshift.service: Consumed 24.056s CPU time
Mar 09 13:57:31 a2fec82806c3 systemd[1]: microshift.service: Service RestartSec=100ms expired, scheduling restart.
Mar 09 13:57:31 a2fec82806c3 systemd[1]: microshift.service: Scheduled restart job, restart counter is at 22.
Mar 09 13:57:31 a2fec82806c3 systemd[1]: Stopped MicroShift.

Relevant mounts inside AIO pod:

/dev/mapper/luks-4306042c-0955-4e0c-bbcf-770e8903dfc7 on /var/lib/containers/storage/overlay type btrfs (rw,nosuid,nodev,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=257,subvol=/root)
/dev/mapper/luks-4306042c-0955-4e0c-bbcf-770e8903dfc7 on /var/lib/kubelet type btrfs (rw,nosuid,nodev,relatime,seclabel,compress=zstd:1,ssd,space_cache,subvolid=257,subvol=/root).

similar issue in kind: kubernetes-sigs/kind#2411

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jiridanek]

/reopen

[openshift-ci]

@jiridanek: You can't reopen an issue/PR unless you authored it or you are a collaborator.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.