[GitHub] Handshake failed: knownhosts: key mismatch

[pkit]

Started getting these errors out of the blue on all clusters.

{"level":"error","ts":"2021-11-16T18:21:07.474Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-system","namespace":"flux-system","error":"unable to clone 'ssh://git@github.com/user/repository', error: ssh: handshake failed: knownhosts: key mismatch"}

Doing find -name known_hosts in the pod produces nothing.
Restarting the pod = same error immediately.
What's going on, where's the known_hosts file?

[stefanprodan]

What's going on, where's the known_hosts file?

The known_hosts file is in the same secret as the SSH key, please see the docs here https://fluxcd.io/docs/components/source/gitrepositories/#ssh-authentication

[stefanprodan]

I'm getting the same error on my cluster:

✗ GitRepository reconciliation failed: 'unable to clone 'ssh://git@github.com/stefanprodan/my-demo-fleet': ssh: handshake failed: knownhosts: key mismatch'

Looks like an issue with GitHub host keys.

[kmannuz]

I am also seeing this error in the last 30 minutes on 3 clusters that had been previously working fine

[kingdonb]

According to: https://github.blog/2021-09-01-improving-git-protocol-security-github/

Today is the day that host keys get rotated at GitHub. There are two new host keys in the blog post, one for ECDSA and another for Ed25519.

[stefanprodan]

Ok so rotating the SSH key fixes it.

Before:

$ k -n flux-system get secret flux-system -o json | jq '.data | map_values(@base64d)'
{
  "identity": "-----BEGIN PRIVATE KEY-----\n",
  "identity.pub": "ecdsa-sha2-nistp384 \n",
  "known_hosts": "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=="
}

After:

{
  "identity": "-----BEGIN PRIVATE KEY-----\n",
  "identity.pub": "ecdsa-sha2-nistp384 \n",
  "known_hosts": "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg="
}

[pkit]

The known_hosts file is in the same secret as the SSH key, please see the docs here https://fluxcd.io/docs/components/source/gitrepositories/#ssh-authentication

Cool, thanks, but I do see the "old" keys when doing keyscan on the nodes.
Somehow only the pods see the "new" ones.
It makes sense though.

[stefanprodan]

GitHub has changed its SSH host keys from DSA to ECDSA!
https://github.blog/2021-09-01-improving-git-protocol-security-github/

To fix the key mismatch error, you have two options:

Update the known_hosts in the flux-system secret with the ecdsa-sha2-nistp25 value:

github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=

Or rotate the SSH keys with flux boostrap like so:

• delete the deploy key secret from your cluster kubectl -n flux-system delete secret flux-system
• rerun flux bootstrap github with the same arguments as before
• Flux will generate the secret with ecdsa-sha2 SSH key and Host key

[pkit]

Updated known_hosts in flux-system secret manually everywhere.
Seems to work now.

[seh]

If you'd like a short program to do it:

#!/usr/bin/env bash

set -e -u -o pipefail

# NB: The Ed25519-format key does not work with Flux.
for secret_name in flux-system repo-2 repo-3; do
  kubectl --namespace=flux-system \
          patch secret "${secret_name}" \
          --patch='
stringData:
  known_hosts: >
    github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg='
done

kubectl --namespace=flux-system rollout restart deployment source-controller
kubectl --namespace=flux-system rollout status deployment/source-controller --watch

[brianpham]

Confirmed. Working for us now as well after deleting the secret and bootstrapping again.

[stefanprodan]

@seh the secret is not mounted inside source-controller, instead the controller reads the secret from Kubernetes API before each Git operations. I don't think you need rollout restart.

[seh]

I was finding that it sits in what appears to be due to a backed-off timer, such that it won't try again for a while after several consecutive failures, but restarting it caused it to try again immediately.

[ellieayla]

Variant on the above script: https://gist.github.com/ellieayla/76352313c4f5939db6d2268fb70b0d48

Then either wait or request each GitRepository to reconcile.

[poteat]

Confirm that we are getting this on our cluster as well suddenly.