[BUG] serivce-ca crash looping

[cooktheryan]

What happened:

openshift-service-ca            service-ca-84b44986cb-wsqxs           0/1     CrashLoopBackOff   5          16m

What you expected to happen:

Service ca stays online

How to reproduce it (as minimally and precisely as possible):

1. Deploy from install.sh or build from main
2. kubectl get po -wA

Anything else we need to know?:

Environment:

• Microshift version (use microshift version): All
• Hardware configuration: amd64
• OS (e.g: cat /etc/os-release): 8.4 rhel and fedora
• Kernel (e.g. uname -a): 4.18.0-305.el8.x86_64
• Others:

Relevant Logs

[cooktheryan]

This seems to have resolved itself with the latest build

[fzdarsky]

Unfortunately, it's still an issue in 4.8.0-0.microshift-2021-11-19-115908-10-ga0f69b3.

[fzdarsky]

The reason for service-ca crash-looping is that it gets assigned an IP address from the wrong subnet and therefore can't connect which makes the process segfault:

[vagrant@fedora ~]$ oc logs pod/service-ca-7bffb6f6bf-sbnxl -n openshift-service-ca
W1221 13:18:36.329434       1 cmd.go:204] Using insecure, self-signed certificates
I1221 13:18:36.832263       1 observer_polling.go:159] Starting file observer
W1221 13:18:39.963082       1 builder.go:209] unable to get owner reference (falling back to namespace): Get "https://10.43.0.1:443/api/v1/namespaces/openshift-service-ca/pods": dial tcp 10.43.0.1:443: connect: no route to host
I1221 13:18:39.963316       1 builder.go:240] service-ca-controller version v3.11.0-357-gb66c450-
F1221 13:19:10.812187       1 cmd.go:129] unable to load configmap based request-header-client-ca-file: Get "https://10.43.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.43.0.1:443: connect: no route to host
goroutine 1 [running]:
[...]

[vagrant@fedora ~]$ oc get pods -A -o wide
NAMESPACE                       NAME                                  READY   STATUS             RESTARTS   AGE   IP          NODE     NOMINATED NODE   READINESS GATES
kube-system                     kube-flannel-ds-5pv9w                 1/1     Running            0          13m   10.0.2.15   fedora   <none>           <none>
kubevirt-hostpath-provisioner   kubevirt-hostpath-provisioner-j98x6   1/1     Running            0          13m   10.85.0.2   fedora   <none>           <none>
openshift-dns                   dns-default-95tm7                     2/2     Running            0          13m   10.42.0.2   fedora   <none>           <none>
openshift-dns                   node-resolver-bz4js                   1/1     Running            0          13m   10.0.2.15   fedora   <none>           <none>
openshift-ingress               router-default-6c96f6bc66-7prlx       1/1     Running            0          13m   10.0.2.15   fedora   <none>           <none>
openshift-service-ca            service-ca-7bffb6f6bf-sbnxl           0/1     CrashLoopBackOff   9          13m   10.85.0.3   fedora   <none>           <none>

Note the 10.85.0.x IP, which is from CRI-O's default CIDR vs. 10.42.0.x, which is the default service CIDR.

Note further that kubecirt-hostpath-provider receives a wrong IP, too, so this is not limited to service-ca!

[fzdarsky]

A workaround is to delete both pods. They receive a correct IP after restart.

[fzdarsky]

Fixed in #519 and #525.