BUG: After SA token renewal the register cluster command overwrites the secret, addon-controller looking for re-kubeconfig key

[mahauber]

Problem Description

1. Register cluster initially via sveltosctl

# kubectl pointing to workload cluster
sveltosctl generate kubeconfig --create --expirationSeconds=86400 > ./sveltos-cluster.config

# kubectl pointing to management cluster
sveltosctl register cluster --namespace=$NAMESPACE --cluster=$CLUSTER_NAME --kubeconfig=./sveltos-cluster.config --labels key=value

kubectl patch sveltoscluster $CLUSTER_NAME -n $NAMESPACE --type='merge' --patch '{"spec":{"tokenRequestRenewalOption":{"renewTokenRequestInterval":"1h0m0s","saName":"projectsveltos", "saNamespace":"projectsveltos"}}}'`

2. Wait for a token renewal of the service account -> will update kubeconfig secret with key re-kubeconfig
3. Step 1 again (eg. to update labels) --> sveltosctl will update kubeconfig secret with key kubeconfig
4. Addon controller cannot find the secret key re-kubeconfig (Failure: data section does not contain key: re-kubeconfig)

To solve: edit secret and rename the key kubeconfig to re-kubeconfig
Question: After initial registration should the labels be set via kubectl patch/label and sveltosctl register be avoided? Is there a better way to manage authentication with AKS clusters (workload identity etc.)?

System Information

INSTRUCTIONS: Provide the system and application information below.

CLUSTERAPI VERSION: not used
SVELTOS VERSION: v0.57.2
KUBERNETES VERSION: v1.33.2

Logs

INSTRUCTIONS: Provide any additional information you think would be helpful below. Large files, logs, etc. can be attached to this issue so long as they meet the GitHub attachment guidelines described here: https://help.github.com/en/github/managing-your-work-on-github/file-attachments-on-issues-and-pull-requests

# kubectl describe clustersummary xy --> status
Failure Message:       data section does not contain key: re-kubeconfig                                                                                                                        

[gianlucam76]

Thank you @mahauber for filing this. I'm not sure I got the issue though.

Here is sveltos behavior when token is renewed.
A new entry is added to secret (re-kubeconfig) and the sveltoscluster spec is updated saying the key within the secret with correct kubeconfig is "re-kubeconfig"

Addon controller (and other sveltos controllers), when needs to access managed cluster, fetches the sveltoscluster, gets the key with kubeconfig and then fetches secret with kubeconfig.

Can you please clarify what was the issue you encountered? Thank you so much!

[mahauber]

Thanks for the quick response and explanation @gianlucam76!

Yes, this is what I encountered. My problem is that running the sveltos register command again is overwriting the kubeconfig secret but leaves the sveltoscluster spec as is. The controller is looking for re-kubeconfig but it doesnt exist anymore. I exptected the sveltos register command to be idempotent and can be executed multiple times in a pipeline.

Would it make sense that the command updates the sveltoscluster spec to match "kubeconfig" instead of "re-kubeconfig" if the sveltoscluster object already exists?

If not I would just do a kubectl patch, I´m also fine with that or using plain yaml to register. Anyway really enjoying sveltos!

[gianlucam76]

Thank you @mahauber

Yes what you are suggesting makes sense and sounds like the correct behavior. I will take care of it