[BUG] Cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs

[kevchu3]

What happened?

I am using the MetalLB load balancer and found that the speaker is attempting to connect to my Kubernetes API endpoint at https://10.43.0.1:443 but cannot validate the certificate for 10.43.0.1 due to this error: Get \"https://10.43.0.1:443/api?timeout=32s\": x509: cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs

I've encountered this twice after upgrading my MicroShift RPMs. The first time was when upgrading from 4.12.3 to 4.12.4 and to resolve the issue, I reverted my MicroShift version back to 4.12.2. During the upgrade, I also noticed my /etc/microshift/config.yaml was malformed, so I corrected it. A couple of weeks and upgrades later, I've noticed that I'm running into the same issue again, and the /etc/microshift/config.yaml appears to be fine this time around.

Any suggestions on what this issue might be?

What did you expect to happen?

API call at https://10.43.0.1:443/api should not run into an IP SAN issue as I would expect the API certificate to assume 10.43.0.1 to be trusted.

How to reproduce it (as minimally and precisely as possible)?

1. Installed a MicroShift 4.12 cluster
2. Installed MetalLB version 0.13.9 through these instructions:

$ oc apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.9/config/manifests/metallb-native.yaml
$ oc adm policy add-scc-to-user privileged -z controller -n metallb-system
$ oc adm policy add-scc-to-user privileged -z speaker -n metallb-system

3. Upgrade MicroShift package version (sudo dnf update). I encountered this issue when upgrading from 4.12.3 to 4.12.4 and again from 4.12.5 to 4.12.6.

Anything else we need to know?

Environment

• MicroShift version (use microshift version):

$ rpm -qa | grep microshift
microshift-selinux-4.12.6-202303012057.p0.g50997a2.assembly.4.12.6.el8.noarch
microshift-networking-4.12.6-202303012057.p0.g50997a2.assembly.4.12.6.el8.x86_64
microshift-4.12.6-202303012057.p0.g50997a2.assembly.4.12.6.el8.x86_64

• Hardware configuration:
• OS (e.g: cat /etc/os-release):

# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.7 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

• Kernel (e.g. uname -a):

# uname -a
Linux <hostname obfuscated> 4.18.0-425.13.1.el8_7.x86_64 #1 SMP Thu Feb 2 13:01:45 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

• Others:

Relevant logs

$ cat /etc/microshift/config.yaml
dns:
  baseDomain: <obfuscated>
network:
  clusterNetwork:
    - cidr: 10.42.0.0/16
  serviceNetwork:
    - 10.43.0.0/16
  serviceNodePortRange: 30000-32767
node:
  hostnameOverride: '<obfuscated>'
  nodeIP: '<obfuscated>'
apiServer:
  subjectAltNames: []
debugging:
  logLevel: "Normal"

The speaker pod happens to be an infrastructure application that is calling the API endpoint:

$ oc logs speaker-7zs9f -n metallb-system
{"branch":"dev","caller":"main.go:93","commit":"dev","goversion":"gc / go1.19.5 / amd64","level":"info","msg":"MetalLB speaker starting version 0.13.9 (commit dev, branch dev)","ts":"2023-03-07T22:40:28Z","version":"0.13.9"}
{"caller":"announcer.go:115","event":"createARPResponder","interface":"wlp2s0","level":"info","msg":"created ARP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"caller":"announcer.go:124","event":"createNDPResponder","interface":"wlp2s0","level":"info","msg":"created NDP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"caller":"announcer.go:115","event":"createARPResponder","interface":"br-int","level":"info","msg":"created ARP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"caller":"announcer.go:115","event":"createARPResponder","interface":"ovn-k8s-mp0","level":"info","msg":"created ARP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"caller":"announcer.go:115","event":"createARPResponder","interface":"virbr0","level":"info","msg":"created ARP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"caller":"announcer.go:115","event":"createARPResponder","interface":"br-ex","level":"info","msg":"created ARP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"caller":"announcer.go:124","event":"createNDPResponder","interface":"br-ex","level":"info","msg":"created NDP responder for interface","ts":"2023-03-07T22:40:28Z"}
{"level":"error","ts":"2023-03-07T22:40:28Z","msg":"Failed to get API Group-Resources","error":"Get \"https://10.43.0.1:443/api?timeout=32s\": x509: cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs","stacktrace":"sigs.k8s.io/controller-runtime/pkg/cluster.New\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/cluster/cluster.go:161\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/manager/manager.go:351\ngo.universe.tf/metallb/internal/k8s.New\n\t/go/go.universe.tf/metallb/internal/k8s/k8s.go:129\nmain.main\n\t/go/go.universe.tf/metallb/speaker/main.go:156\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"error","ts":"2023-03-07T22:40:28Z","logger":"setup","msg":"unable to start manager","error":"Get \"https://10.43.0.1:443/api?timeout=32s\": x509: cannot validate certificate for 10.43.0.1 because it doesn't contain any IP SANs","stacktrace":"go.universe.tf/metallb/internal/k8s.New\n\t/go/go.universe.tf/metallb/internal/k8s/k8s.go:149\nmain.main\n\t/go/go.universe.tf/metallb/speaker/main.go:156\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

[kevchu3]

I've dug a bit deeper into the issue and found that the API service at 10.43.0.1:443 is serving the ingress-ca certificate instead of the API certificate.

Just to confirm the service that should be listening on 10.43.0.1 (on the cluster I'm reporting an issue on):

$ oc get svc -n default
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   45d

Here's what I'm seeing on that same cluster:

$ echo |     openssl s_client -servername 10.43.0.1 -connect 10.43.0.1:443 2>/dev/null |     openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 9 (0x9)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = ingress-ca
        Validity
            Not Before: Mar  8 16:53:02 2023 GMT
            Not After : Mar  7 16:53:03 2024 GMT
        Subject: CN = *.apps.example.com
...
            X509v3 Subject Alternative Name: 
                DNS:*.apps.example.com

I looked at another MicroShift reference cluster and this is the certificate I expect to see:

$ echo |     openssl s_client -servername 10.43.0.1 -connect 10.43.0.1:443 2>/dev/null |     openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kube-apiserver-service-network-signer
        Validity
            Not Before: Jan 31 17:32:52 2023 GMT
            Not After : Jan 31 17:32:53 2024 GMT
        Subject: CN = 10.43.0.1
...
            X509v3 Subject Alternative Name: 
                DNS:api-int.example.com, DNS:api.example.com, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:10.43.0.1, IP Address:10.43.0.1

And in my reference cluster, I'm running this version of MicroShift:

$ rpm -qa | grep microshift
microshift-networking-4.12.1-202301261525.p0.g3db9e81.assembly.4.12.1.el8.x86_64
microshift-selinux-4.12.1-202301261525.p0.g3db9e81.assembly.4.12.1.el8.noarch
microshift-4.12.1-202301261525.p0.g3db9e81.assembly.4.12.1.el8.x86_64

[kevchu3]

I've reproduced the issue fairly easily, here are the steps.

1. I installed RHEL 8.7 from scratch and followed the documentation here.

2. I installed MicroShift 4.12.6 (which is the latest at the moment).

$ rpm -qa | grep microshift
microshift-selinux-4.12.6-202303012057.p0.g50997a2.assembly.4.12.6.el8.noarch
microshift-networking-4.12.6-202303012057.p0.g50997a2.assembly.4.12.6.el8.x86_64
microshift-4.12.6-202303012057.p0.g50997a2.assembly.4.12.6.el8.x86_64

The issue is that the API endpoint at https://10.43.0.1 is serving the ingress-ca certificate rather than the kube-apiserver-service-network-signer certificate. To verify:

$ echo |     openssl s_client -servername 10.43.0.1 -connect 10.43.0.1:443 2>/dev/null |     openssl x509 -text

...
            X509v3 Subject Alternative Name: 
                DNS:*.apps.example.com

3. To confirm that the issue is with a recent version of MicroShift, I went back and reinstalled my RHEL 8.7 server from scratch and installed an older version of MicroShift .

$ rpm -qa | grep microshift
microshift-selinux-4.12.1-202301261525.p0.g3db9e81.assembly.4.12.1.el8.noarch
microshift-networking-4.12.1-202301261525.p0.g3db9e81.assembly.4.12.1.el8.x86_64
microshift-4.12.1-202301261525.p0.g3db9e81.assembly.4.12.1.el8.x86_64

And I can also verify that the same command to check the certificate produces the expected one:

$ echo |     openssl s_client -servername 10.43.0.1 -connect 10.43.0.1:443 2>/dev/null |     openssl x509 -text

...
            X509v3 Subject Alternative Name: 
                DNS:api-int.example.com, DNS:api.example.com, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:openshift, DNS:openshift.default, DNS:openshift.default.svc, DNS:openshift.default.svc.cluster.local, DNS:10.43.0.1, IP Address:10.43.0.1

4. As I suspected, the MetalLB project did not directly cause this regression, but it's another way to confirm. I apply the MetalLB manifests:

$ oc apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.9/config/manifests/metallb-native.yaml
$ oc adm policy add-scc-to-user privileged -z controller -n metallb-system
$ oc adm policy add-scc-to-user privileged -z speaker -n metallb-system

And verified that the speaker pod has crashed.

$ oc logs speaker-qzqrk -n metallb-system
...
{"level":"error","ts":"2023-03-09T19:19:05Z","msg":"Failed to get API Group-Resources","error":"Get \"https://10.43.0.1:443/api?timeout=32s\": dial tcp 10.43.0.1:443: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/cluster.New\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/cluster/cluster.go:161\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/manager/manager.go:351\ngo.universe.tf/metallb/internal/k8s.New\n\t/go/go.universe.tf/metallb/internal/k8s/k8s.go:129\nmain.main\n\t/go/go.universe.tf/metallb/speaker/main.go:156\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"error","ts":"2023-03-09T19:19:05Z","logger":"setup","msg":"unable to start manager","error":"Get \"https://10.43.0.1:443/api?timeout=32s\": dial tcp 10.43.0.1:443: connect: connection refused","stacktrace":"go.universe.tf/metallb/internal/k8s.New\n\t/go/go.universe.tf/metallb/internal/k8s/k8s.go:149\nmain.main\n\t/go/go.universe.tf/metallb/speaker/main.go:156\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

[dhellmann]

See #1478

[pacevedom]

For a more detailed description of the issue check the first commit message in the PR @dhellmann linked.