support alternate controller-manager flags in kubecertagent controller (e.g. for RKE2)

[cfryanr]

The controller responsible for creating the kube-cert-agent pod parses the CLI flags of the controller-manager pod to discover the file paths for the cert and key files. However, the controller-manager binary offers two different sets of CLI flags that can be used to specify these paths. Previously, Pinniped only looked for the original flags --cluster-signing-cert-file, and --cluster-signing-key-file. Now also look for the alternate flags --cluster-signing-kube-apiserver-client-key-file and --cluster-signing-kube-apiserver-client-cert-file. According to https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ you can specify either of these flag sets, but not both.

This change will hopefully allow the Pinniped Concierge to support more Kubernetes distributions without needing to resort to using the impersonation proxy. For example, RKE2 uses these alternate controller-manager flags.

Note: Instead of using the Pinniped Concierge on RKE2, you could configure the API server flags to use the Pinniped Supervisor as its OIDC provider for authentication. See https://pinniped.dev/docs/tutorials/supervisor-without-concierge-demo/ for details. This does not require installing the Concierge onto the cluster, and therefore would not require the workarounds documented below. Or alternatively, you could install the Concierge but enable the impersonation proxy and use that instead. However, the impersonation proxy has the downside of requiring a means of ingress be configured for incoming HTTPS network traffic.

Note that RKE2 clusters do not have the standard cluster-info configmap. The Pinniped Concierge wants to read this configmap to get the cluster's endpoint and CA bundle. As a workaround, it can be created by the administrator using this shell script. Note that if you have made the cluster accessible over the network (i.e. not localhost), then the server should be the URL that you use to access the cluster's API server over the network.

#!/usr/bin/env bash
set -euo pipefail

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-info
  namespace: kube-public
data:
  kubeconfig: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: $(kubectl config view --minify=true --raw -o json | jq -r '.clusters[0].cluster["certificate-authority-data"]')
        server: $(kubectl config view --minify=true --raw -o json | jq -r '.clusters[0].cluster.server')
      name: ""
    contexts: null
    current-context: ""
    kind: Config
    preferences: {}
    users: null
EOF

# Echo it back to the screen.
echo "Showing created cluster-info configmap ..."
kubectl get configmap -n kube-public cluster-info -o yaml

This allows you to get further into the authentication flow, but unfortunately it still fails.

Now it fails because the kube-api-server pod specifies the --anonymous-auth=false CLI flag to the kube-apiserver command by default in RKE2. This prevents the user from being able to make calls to the Concierge's TokenCredentialRequest API to complete their authentication using the Concierge. This Concierge authentication endpoint receives the user's proof of identity and returns a credential that they can use to auth directly to the Kube API server as that identity, so it must be allowed to be called without prior authentication.

To allow anonymous auth for the Kubernetes API in RKE2, edit (or create) the config file:

sudo vim /etc/rancher/rke2/config.yaml

Add this line to the file. Then save the file and exit the editor.

kube-apiserver-arg: "anonymous-auth=true"

Restart the cluster:

sudo systemctl restart rke2-server

After the cluster restarts, make sure that the API server pod has the --anonymous-auth=true CLI flag.

kubectl get pods -l 'component=kube-apiserver' -n kube-system -o 'jsonpath={.items[0].spec.containers[0].args}'

Now you will be able to use the Pinniped Concierge on your RKE cluster without being required to enable the Concierge impersonation proxy.

Note: The full implications of enabling anonymous auth for the Kube API server (not discussed here) should be considered before doing this on a production cluster.

Thanks to @Oğuz Yarımtepe on Kubernetes Slack for helping us investigate this on RKE2 clusters.

Release note:

TBD

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 31.19%. Comparing base (59c36ee) to head (5e6f6a1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2043      +/-   ##
==========================================
+ Coverage   31.17%   31.19%   +0.01%     
==========================================
  Files         366      366              
  Lines       61174    61191      +17     
==========================================
+ Hits        19072    19089      +17     
  Misses      41569    41569              
  Partials      533      533              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.