MicroShift 4.21 startup fails due to service-ca errors in bootc containers

[ggiguash]

The problem does not seem to occur when running on a host.

# oc describe pod -n openshift-service-ca service-ca-7df55db7f9-qw6vg | grep -A10 Events:
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  3h10m (x3 over 3h10m)   default-scheduler  no nodes available to schedule pods
  Normal   Pulled            4m5s (x855 over 3h9m)   kubelet            Container image "quay.io/okd/scos-content@sha256:83ba25f4b709a53932183342431986fdc1419cc4a960f70a3e72251b0c67f7dd" already present on machine
  Warning  Failed            3m52s (x857 over 3h9m)  kubelet            Error: container create failed: mount_setattr `/etc/hosts` (maybe the file system used doesn't support idmap mounts on this kernel?): Invalid argument

[ggiguash]

The latest service-ca operator is running non-root and is getting a pod user namespace.
I suspect openshift/service-ca-operator@7f239cc is causing this new behavior.

• With user namespaces, kubelet creates /etc/hosts etc. via idmapped mounts under /var/lib/kubelet/pods
• Inside the bootc environment, that directory ends up on a filesystem that doesn’t support idmap mounts (tmpfs or overlayfs)
• On the host MicroShift install, /var/lib/kubelet is on ext4/xfs that does support idmapped mounts, so we don’t see the error.

[ggiguash]

Looking at other pods, service-ca seems to be the only one with hostUsers set to false

# oc get pods -A   -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.hostUsers}{"\n"}{end}'
cert-manager    cert-manager-5ff79b64d4-g6n87
cert-manager    cert-manager-cainjector-7ff6579955-bcndt
cert-manager    cert-manager-webhook-6677bc56dd-54jsk
kube-kindnet    kube-kindnet-ds-wp8kf
kube-proxy      kube-proxy-mw7bf
kube-system     csi-snapshot-controller-f7998d4c7-k5bpg
openshift-dns   dns-default-7kzt4
openshift-dns   node-resolver-wjvkt
openshift-ingress       router-default-b7c7d5b5b-zk8qq
openshift-service-ca    service-ca-7df55db7f9-s5z2w     false
topolvm-system  topolvm-controller-6684c669f7-zxkgc
topolvm-system  topolvm-lvmd-0-mhg2h
topolvm-system  topolvm-node-fr9gm

# oc get pods -A   -o jsonpath='{range .items[?(@.spec.hostUsers==false)]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.hostUsers}{"\n"}{end}'
openshift-service-ca    service-ca-7df55db7f9-s5z2w     false

[ggiguash]

Using -v /var/lib/kubelet:/var/lib/kubelet:Z option when starting the bootc container works around this issue. We need to investigate how to solve this problem in a scalable way without host directory dependency.