Fix occasional CI failures due to TopoLVM not initializing properly

[ggiguash]

It seems to happen more frequently in the isolated CI jobs. See this run for example.

Note that the problem can be reproduced locally by running the following sequence:

make rpm image EMBED_CONTAINER_IMAGES=1

while make clean run run-ready run-healthy ISOLATED_NETWORK=1 ; do sleep 0 ; done

Note: Run make rpm image EMBED_CONTAINER_IMAGES=1 WITH_KINDNET=0 command to test with OVN-k networking.

[ggiguash]

Looking at the sos report from the job, we observe that topolvm-controller does not start properly.

$ omc get pods -A
NAMESPACE              NAME                                       READY   STATUS             RESTARTS   AGE
cert-manager           cert-manager-5f864bbfd-r5pnk               1/1     Running            0          15m
cert-manager           cert-manager-cainjector-589dc747b5-2nzn9   1/1     Running            0          15m
cert-manager           cert-manager-webhook-5987c7ff58-qs2dk      1/1     Running            0          15m
kube-kindnet           kube-kindnet-ds-7lgh8                      1/1     Running            0          15m
kube-proxy             kube-proxy-cgdlb                           1/1     Running            0          15m
kube-system            csi-snapshot-controller-7886549469-lzqjl   1/1     Running            0          15m
openshift-dns          dns-default-599wx                          1/2     Running            0          15m
openshift-dns          node-resolver-phzxp                        1/1     Running            0          15m
openshift-ingress      router-default-59544b6969-z8c5v            1/1     Running            0          15m
openshift-service-ca   service-ca-74b47c98cc-jlz5l                1/1     Running            0          15m
topolvm-system         topolvm-controller-9cd8649c9-5t5h7         4/5     CrashLoopBackOff   28         15m
topolvm-system         topolvm-lvmd-0-pngdg                       1/1     Running            0          15m
topolvm-system         topolvm-node-sr2t6                         3/3     Running            0          15m

The container logs have the following errors:

$ omc logs -n topolvm-system topolvm-controller-9cd8649c9-5t5h7 --all-containers
2025-10-18T15:17:29.401319477Z {"level":"info","ts":"2025-10-18T15:17:29Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/pod/mutate"}
2025-10-18T15:17:29.401319477Z {"level":"info","ts":"2025-10-18T15:17:29Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/pvc/mutate"}
2025-10-18T15:17:29.403468421Z {"level":"error","ts":"2025-10-18T15:17:29Z","logger":"setup","msg":"unable to create controller","controller":"Node","error":"failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get \"https://10.43.0.1:443/api/v1\": tls: failed to verify certificate: x509: cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs","stacktrace":"github.com/topolvm/topolvm/cmd/topolvm-controller/app.subMain\n\t/workdir/cmd/topolvm-controller/app/run.go:111\ngithub.com/topolvm/topolvm/cmd/topolvm-controller/app.init.func1\n\t/workdir/cmd/topolvm-controller/app/root.go:63\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041\ngithub.com/topolvm/topolvm/cmd/topolvm-controller/app.Execute\n\t/workdir/cmd/topolvm-controller/app/root.go:70\nmain.main\n\t/workdir/cmd/hypertopolvm/main.go:44\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
2025-10-18T15:17:29.403468421Z Error: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get "https://10.43.0.1:443/api/v1": tls: failed to verify certificate: x509: cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs
2025-10-18T15:17:29.403489240Z failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get "https://10.43.0.1:443/api/v1": tls: failed to verify certificate: x509: cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs
2025-10-18T15:20:58.386338464Z W1018 15:20:58.386232       1 feature_gate.go:354] Setting GA feature gate Topology=true. It will be removed in a future release.
2025-10-18T15:20:58.386338464Z I1018 15:20:58.386323       1 csi-provisioner.go:154] Version: 
2025-10-18T15:20:58.386338464Z I1018 15:20:58.386332       1 csi-provisioner.go:177] Building kube configs for running in cluster...
2025-10-18T15:20:58.432318938Z I1018 15:20:58.431586       1 main.go:108] "Version" version=""
2025-10-18T15:20:58.483686494Z I1018 15:20:58.483576       1 main.go:108] Version: 
2025-10-18T15:06:38.931944026Z I1018 15:06:38.931886       1 connection.go:253] "Still connecting" address="unix:///run/topolvm/csi-topolvm.sock"
2025-10-18T15:06:48.931708356Z I1018 15:06:48.931656       1 connection.go:253] "Still connecting" address="unix:///run/topolvm/csi-topolvm.sock"
..
<Many lines omitted>
..
2025-10-18T15:20:58.932039496Z I1018 15:20:58.931987       1 connection.go:253] "Still connecting" address="unix:///run/topolvm/csi-topolvm.sock"

[ggiguash]

Another CI failure can be found here.

[ggiguash]

Here's a certificate SAN for a working system:

# oc get --raw /api | \
    openssl s_client -connect 10.43.0.1:443 -showcerts </dev/null 2>/dev/null | \
    openssl x509 -noout -text | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name: 
                DNS:api-int.example.com, DNS:api.example.com, DNS:kubernetes, DNS:kubernetes.default, 
                DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:openshift,
                DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, 
                DNS:10.43.0.1, DNS:10.44.0.0, IP Address:10.43.0.1, IP Address:10.44.0.0

Here's a certificate SAN for a failed system:

# oc get --raw /api | \
    openssl s_client -connect 10.43.0.1:443 -showcerts </dev/null 2>/dev/null | \
    openssl x509 -noout -text | grep -A1 "Subject Alternative Name"
            X509v3 Subject Alternative Name: 
                DNS:*.apps.example.com

[ggiguash]

The system usually reconciles itself after a relatively long period of time (.e.g. 30m).
It causes the greenboot checks to fail and declare the boot invalid. Looks like some timing issue because all the container images are present locally.

[ggiguash]

@copejon
The problem still happens. See this job log for example.