VC-34488: Add volumes and volumeMounts for using custom CA bundles in the Venafi Kubernetes Agent

[maelvls]

Ref: https://venafi.atlassian.net/browse/VC-34488

We found that there is currently no “knob” in the Venafi Kubernetes Agent values.yaml file to configure a custom CA bundle. @hawksight created a GitHub issue for that: #541.

In this PR, I add the new volumes and volumeMounts and added examples of how to set up a custom CA bundle in the comments of the CA bundle. 99% of this PR comes from @hawksight's work in #540.

The way I intend users to use custom CA bundles is to first create a config map:

kubectl create configmap cabundle -n venafi \
  --from-file=cabundle=./your/custom/ca/bundle

Then, the user configures its values.yaml with the following:

volumes:
  - name: cabundle
    configMap:
      name: cabundle
      defaultMode: 0644

volumeMounts:
  - name: cabundle
    mountPath: /etc/ssl/certs/cabundle
    subPath: cabundle
    readOnly: true

That's because Venafi Kubernetes Agent will trust any PEM certificate under /etc/ssl/certs.

With venctl, using the same values.yaml as above, it will look like this:

venctl components kubernetes manifest generate \              
  --venafi-kubernetes-agent \
  --venafi-kubernetes-agent-values-files values.yaml \
  --venafi-kubernetes-agent-custom-chart-repository oci://registry.venafi.cloud/charts \
  --venafi-kubernetes-agent-custom-image-registry registry.venafi.cloud/venafi-agent >venafi-agent.yaml

Unit tests

$ helm unittest ./deploy/charts/venafi-kubernetes-agent 
### Chart [ venafi-kubernetes-agent ] ./deploy/charts/venafi-kubernetes-agent

 PASS  test deployment  deploy/charts/venafi-kubernetes-agent/tests/deployment_test.yaml

Charts:      1 passed, 1 total
Test Suites: 1 passed, 1 total
Tests:       6 passed, 6 total
Snapshot:    0 passed, 0 total
Time:        13.530708ms

Manual tests (in Kubernetes)

The first test is about replacing the system bundle and testing with that. The second test is about adding a CA bundle along side the system bundle and checking that it works.

Replacing ca-certificates.crt

I don't use the Helm chart's config.yaml as there is too much noise in the
logs. Instead, I use a bare-bones configmap that doesn't even gather any data
but still connects to the Venafi Cloud API.

mkdir -p mine
cat <<EOF >mine/minimal-config.yaml
cluster_id: "kind-mael"
cluster_description: "Kind cluster on Mael's Aorus home machine"
server: "https://api.venafi.cloud/"
venafi-cloud:
  uploader_id: "no"
  upload_path: "/v1/tlspk/upload/clusterdata"
data-gatherers: []
EOF
kubectl create configmap -n venafi agent-config-minimal --from-file=config.yaml=mine/minimal-config.yaml \
    -oyaml --dry-run=client | kubectl apply -f -

mkdir -p mine
venctl iam service-account agent create --name sa-agent-mael \
  --output secret \
  --output-file mine/agent-credentials.json \
  --api-key $(lpass show glow-in-the-dark.venafi.cloud -p)
kubectl apply -n venafi -f <(jq '.private_key' -r mine/agent-credentials.json)

Now, let's create a configmap with a CA bundle that only contains the Venafi
Cloud certificate:

kubectl create configmap cabundle -n venafi \
  --from-literal=cabundle="$(certigo connect api.venafi.cloud --pem)"

Then, I created mine/ca-bundle.values.yaml:

volumes:
  - name: cabundle
    configMap:
      name: cabundle
      optional: false
      defaultMode: 0644
      items:
        - key: cabundle
          path: ca-certificates.crt

volumeMounts:
  - name: cabundle
    mountPath: /etc/ssl/certs/ca-certificates.crt
    subPath: ca-certificates.crt
    readOnly: true

config:
  configmap:
    name: agent-config-minimal

helm upgrade -i venafi-kubernetes-agent ./deploy/charts/venafi-kubernetes-agent \
  --create-namespace -n venafi --values mine/ca-bundle.values.yaml

In the logs, we can see that the TLS connection is working (although there is
this 404 error that I don't understand):

2024/06/28 11:17:43 Running Agent...
2024/06/28 11:17:43 Posting data to: https://api.venafi.cloud/
2024/06/28 11:17:43 retrying in 1m39.065989069s after error: post to server failed: received response with status code 404. Body:

Just to double check that I got it right, I changed the cabundle configmap to
another bundle that shouldn't work:

kubectl delete configmap cabundle -n venafi
kubectl create configmap cabundle -n venafi --from-literal=cabundle="$(certigo connect google.com --pem)"

As expected, the TLS validation fails:

2024/06/28 11:19:35 Running Agent...
2024/06/28 11:19:35 Posting data to: https://api.venafi.cloud/
2024/06/28 11:19:36 retrying in 43.516197345s after error: post to server failed: Post "https://api.venafi.cloud/api/v1/org/datareadings"
: tls: failed to verify certificate: x509: certificate signed by unknown authority

It proves that the CA bundle is correctly used by the agent.

Without replacing ca-certificates.crt

Go loads all the certs in /etc/ssl/certs, so there is no need to replace /etc/ssl/certs/ca-certificates.crt.

Do the same as above, but replace mine/ca-bundle.values.yaml with the following:

volumes:
  - name: ca-certs-empty
    emptyDir: {}
  - name: cabundle
    configMap:
      name: cabundle

volumeMounts:
  - name: ca-certs-empty
    mountPath: /etc/ssl/certs/
  - name: cabundle
    mountPath: /etc/ssl/certs/cabundle
    subPath: cabundle
    readOnly: true

config:
  configmap:
    name: agent-config-minimal

Don't forget to replace the cabundle configmap with the Venafi cert:

kubectl delete configmap cabundle -n venafi
kubectl create configmap cabundle -n venafi --from-literal=cabundle="$(certigo connect api.venafi.cloud --pem)"

It works as expected:

2024/06/28 14:28:16 Running Agent...
2024/06/28 14:28:16 Posting data to: https://api.venafi.cloud/
2024/06/28 14:28:16 retrying in 31.372528523s after error: post to server failed: received response with status code 404. Body:

Now, let's check that it doesn't default to using ca-certificates.crt by using a different cabundle:

kubectl delete configmap cabundle -n venafi
kubectl create configmap cabundle -n venafi --from-literal=cabundle="$(certigo connect google.com --pem)"
kubectl -n venafi rollout restart deploy venafi-kubernetes-agent

As expected:

│ 2024/06/28 14:32:25 Running Agent...
│ 2024/06/28 14:32:25 Posting data to: https://api.venafi.cloud/
│ 2024/06/28 14:32:25 retrying in 16.288074513s after error: post to server failed: Post "https://api.venafi.cloud/api/v1/org/datareadings"
│ : tls: failed to verify certificate: x509: certificate signed by unknown authority

Checking the actual content of the CA bundle without a shell in the container

I needed to check whether the /etc/ssl/certs/ca-certificates.crt file in the
container actually contained the right certificates. I couldn't get a shell in
the Venafi Agent container, so I had to read the containers' filesystem from the host:

docker exec -it kind-control-plane bash <<'EOF'
CONTAINER_ID=$(crictl ps --name venafi-kubernetes-agent -o json | jq -r ".containers[].id")
PID=$(ctr -n k8s.io t ls | grep $CONTAINER_ID | awk '{print $2}')
cat /proc/$PID/root/etc/ssl/certs/ca-certificates.crt
EOF

Pre-testing in Docker

Before having it work in Kubernetes, I tested this in Docker.

mkdir -p mine
cat <<EOF >mine/minimal-config.yaml
cluster_id: "kind-mael"
cluster_description: "Kind cluster on Mael's machine"
server: "https://api.venafi.cloud/"
venafi-cloud:
  uploader_id: "no"
  upload_path: "/v1/tlspk/upload/clusterdata"
data-gatherers: []
EOF
venctl iam service-account agent create --name sa-agent-mael \
  --output secret \
  --output-file mine/agent-credentials.json \
  --api-key $(lpass show glow-in-the-dark.venafi.cloud -p)
jq -r .private_key mine/agent-credentials.json | yq -r '.data."privatekey.pem"' | base64 -d >mine/private-key.pem
jq -r .client_id mine/agent-credentials.json >mine/client-id
certigo connect api.venafi.cloud --pem >mine/ca-certificates.crt
go build .
docker run -it --rm \
  -v $PWD/preflight:/bin/preflight:ro \
  -v $PWD/mine/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \
  -v $PWD/mine:/mine:ro \
  debian

In the container:

preflight agent -c mine/minimal-config.yaml \
  --client-id /mine/client-id \
  --private-key-path /mine/private-key.pem \
  --venafi-cloud \
  -p 5s

It should work:

2024/06/28 10:35:08 Posting data to: https://api.venafi.cloud/
2024/06/28 10:35:08 retrying in 30.463172694s after error: post to server failed: failed to execute http request to Venafi Control Plane. Request https://api.venafi.cloud/v1/oauth/token/serviceaccount, status code: 400, body: [{"error":"invalid_grant","error_description":"malformed subject claim"}

(OK, that's an error, but it still shows that the TLS connection is working.)

Now, change the ca-certificates.crt to something else:

certigo connect google.com --pem >mine/ca-certificates.crt

You will see:

2024/06/28 10:34:04 retrying in 38.411339922s after error: post to server failed: Post "https://api.venafi.cloud/v1/oauth/token/serviceaccount": tls: failed to verify certificate: x509: certificate signed by unknown authority

[maelvls]

@hawksight Hey, can you take a quick look? All the credit goes to you since I've copied a lot of your work in #540 😅

[maelvls]

@hawksight Do I also need to set Go's SSL_CERT_FILE for clarity, or is that unnecessary?

env:
  - name: SSL_CERT_FILE
    value: /etc/ssl/certs/ca-certificates.crt

I'm thinking that having this will make it more obvious. It's already the path used by Go to look for the system CA bundle. Note that env can't be set at the moment in values.yaml.

[maelvls]

Other question: do I really need to overwrite the container's ca-certificates.crt, or is Go going to load anything that is under /etc/ssl/certs?

Update: Ah... Go does load all certificates under /etc/ssl/certs: https://github.com/golang/go/blob/ea537cca314d9da5365eeefcc375410c76e93b36/src/crypto/x509/root_linux.go#L21, https://github.com/golang/go/blob/ea537cca314d9da5365eeefcc375410c76e93b36/src/crypto/x509/root_unix.go#L61.

I'll undo my complicated /etc/ssl/certs/ca-certificates.crt and do the same thing as you did in #540.

[hawksight]

I'm thinking that having this will make it more obvious. It's already the path used by Go to look for the system CA bundle. Note that env can't be set at the moment in values.yaml.

You are right that it does make it very explicit. There is a SSL_CERT_DIR as well perhaps we could set?
Right now we just rely on Go's defaults. It's worked to date but perhaps we should be explicit?

[hawksight]

I've tested in the jetstsack-agent chart and this looks very similar so happy to approve 👍

[maelvls]

Thanks. I'll change things to match what you have done. My proposition is too complex, I don't need to overwrite the ca-certificates.crt for this to work.

[maelvls]

I've re-tested my changes. It works fine. I'll merge now. Thank you Peter!