-
Notifications
You must be signed in to change notification settings - Fork 1
update the clusteridentifier in the field for helm values! #345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Rupam-It
wants to merge
16
commits into
main
Choose a base branch
from
rupam/cluster_identifier_token
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
3a7fd62
make changes in helm value and env variable and then do make changes …
Rupam-It 1620349
added logic for cluster_identifier
Rupam-It 61abd24
fix(config): resolve CLUSTER_TOKEN conflict and align Helm ownership
Rupam-It 07a4281
feature can teke cluster identifier from secret
Rupam-It 5632e4c
automatic secret creation added
Rupam-It dea5edc
all case are satisfied
Rupam-It 1d61eb6
update the rbac so that it can create new secret
Rupam-It 6159dc2
remove the cluster identifier from the configmap it will only present…
Rupam-It 223037e
make the zxp compatible to use the default cluster identifier rather …
Rupam-It 884c7fd
fixed the gen/ folder sync with the backend
Rupam-It 2ab6d9a
update the role so that it can create a secret
Rupam-It 86de795
Merge branch 'main' into rupam/cluster_identifier_token
Rupam-It a09fd1e
fix and update the cluster identifier
Rupam-It 4e0ef9d
merge main, take generated proto files from main
Rupam-It 01fc922
Add devzero-zxporter-cluster-identity to RBAC resourceNames
Rupam-It e114131
fix the lint and gitter suggestion
Rupam-It File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,198 @@ | ||
| # Cluster Identity: Stable Cluster Identification Across Reinstalls | ||
|
|
||
| ## Overview | ||
|
|
||
| By default, every time you install or reinstall ZXPorter, it calls `CreateClusterToken` on the DevZero platform — which creates a **brand new cluster entry**. This means if you uninstall and reinstall ZXPorter on the same physical cluster, the DevZero dashboard shows it as a different cluster. | ||
|
|
||
| **Cluster Identity** solves this. By giving your cluster a stable identifier, ZXPorter calls `ReattachCluster` instead, which **finds or creates** a cluster by that identifier. Reinstalling ZXPorter always connects back to the same cluster entry in the DevZero platform. | ||
|
|
||
| --- | ||
|
|
||
| ## End-User Guide | ||
|
|
||
| ### When do you need this? | ||
|
|
||
| - You want your cluster to always appear as the same entry in the DevZero dashboard | ||
| - You manage multiple clusters and need stable, human-readable names | ||
|
|
||
| ### Option A: Let Helm manage the Secret (simplest) | ||
|
|
||
| Set `clusterIdentifier` in your `values.yaml`. Helm will auto-create a Kubernetes Secret that persists the identifier. | ||
|
|
||
| ```yaml | ||
| zxporter: | ||
| patToken: "dzu-your-pat-token" | ||
| clusterIdentifier: "prod-us-east" # stable DNS-label style name | ||
| clusterIdentitySecretName: "devzero-zxporter-cluster-identity" # default | ||
| ``` | ||
|
|
||
| Run `helm upgrade --install`: | ||
|
|
||
| ```bash | ||
| helm upgrade --install zxporter ./helm-chart/zxporter \ | ||
| -n devzero-zxporter --create-namespace \ | ||
| -f values.yaml | ||
| ``` | ||
|
|
||
| Helm creates the Secret automatically with `helm.sh/resource-policy: keep`, meaning the Secret **survives `helm uninstall`**. On the next install, ZXPorter reads the identifier from the Secret and reattaches to the same cluster. | ||
|
|
||
| > **Note:** The Secret wins over `clusterIdentifier` in `values.yaml`. Once the Secret exists, you can clear `clusterIdentifier` from your values file — the Secret is the source of truth. | ||
|
|
||
| --- | ||
|
|
||
| ### Option B: Bring your own Secret (production/GitOps) | ||
|
|
||
| Create the Secret manually before installing ZXPorter: | ||
|
|
||
| ```bash | ||
| kubectl create secret generic devzero-zxporter-cluster-identity \ | ||
| -n devzero-zxporter \ | ||
| --from-literal=CLUSTER_IDENTIFIER=prod-us-east | ||
| ``` | ||
|
|
||
| Leave `clusterIdentifier` empty in `values.yaml` and just point to the Secret: | ||
|
|
||
| ```yaml | ||
| zxporter: | ||
| patToken: "dzu-your-pat-token" | ||
| clusterIdentifier: "" | ||
| clusterIdentitySecretName: "devzero-zxporter-cluster-identity" | ||
| ``` | ||
|
|
||
| This is the recommended approach for GitOps workflows where you don't want the identifier hardcoded in values files. | ||
|
|
||
| --- | ||
|
|
||
| ### What happens on each install? | ||
|
|
||
| | Scenario | Behavior | | ||
| |---|---| | ||
| | Secret exists with `CLUSTER_IDENTIFIER` | Reads identifier from Secret → calls `ReattachCluster` → connects to existing cluster | | ||
| | No Secret, `clusterIdentifier` set in values | Uses values.yaml identifier → calls `ReattachCluster` → Helm auto-creates the Secret | | ||
| | No Secret, no `clusterIdentifier` | Calls `CreateClusterToken` → creates a new cluster entry | | ||
| | Secret exists but `CLUSTER_IDENTIFIER` key is missing/empty | Falls back to `clusterIdentifier` in values.yaml; if also empty → creates new cluster | | ||
|
|
||
| --- | ||
|
|
||
| ### What if someone deletes the Secret? | ||
|
|
||
| If the Secret is deleted but a `CLUSTER_IDENTIFIER` is set in `values.yaml`, ZXPorter falls back to that value and still calls `ReattachCluster`. The next `helm upgrade` recreates the Secret. | ||
|
|
||
| If both are gone, ZXPorter creates a new cluster entry on the DevZero platform. You would need to manually reconcile or set `clusterIdentifier` again. | ||
|
|
||
| --- | ||
|
|
||
| ### What if someone deletes the Secret and forgets the identifier? | ||
|
|
||
| Without the identifier, ZXPorter creates a new cluster. To recover: | ||
| 1. Find the cluster identifier from the DevZero dashboard or your previous values file | ||
| 2. Recreate the Secret manually (Option B above) | ||
| 3. Run `helm upgrade --install` to reconnect | ||
|
|
||
| --- | ||
|
|
||
| ## Developer Guide | ||
|
|
||
| ### Architecture | ||
|
|
||
| The cluster identity flow lives in `internal/controller/custom.go`, inside `initializeTelemetryComponents`. | ||
|
|
||
| #### Priority chain (highest to lowest) | ||
|
|
||
| ``` | ||
| 1. Kubernetes Secret (CLUSTER_IDENTITY_SECRET_NAME) | ||
| └─ Key: CLUSTER_IDENTIFIER | ||
| 2. values.yaml / ConfigMap env var | ||
| └─ Key: CLUSTER_IDENTIFIER | ||
| 3. No identifier → CreateClusterToken (new cluster) | ||
| ``` | ||
|
|
||
| #### How the identifier is resolved | ||
|
|
||
| ```go | ||
| // 1. Read secret name from ConfigMap (mounted as file at /etc/zxporter/config/) | ||
| clusterIdentitySecretName := os.Getenv("CLUSTER_IDENTITY_SECRET_NAME") | ||
| if clusterIdentitySecretName == "" { | ||
| // fallback: read from mounted config file | ||
| } | ||
|
|
||
| // 2. If secret name is set, read CLUSTER_IDENTIFIER from that secret | ||
| if clusterIdentitySecretName != "" { | ||
| if identifier := c.readClusterIdentifierFromSecret(ctx, clusterIdentitySecretName); identifier != "" { | ||
| envSpec.Policies.ClusterIdentifier = identifier // secret wins | ||
| } | ||
| } | ||
|
|
||
| // 3. If no stored token, decide which RPC to call | ||
| if envSpec.Policies.ClusterIdentifier != "" { | ||
| token, clusterId, err = tempClient.ReattachCluster(...) | ||
| } else { | ||
| token, clusterId, err = tempClient.ExchangePATForClusterToken(...) | ||
| } | ||
| ``` | ||
|
|
||
| #### `readClusterIdentifierFromSecret` (`custom.go`) | ||
|
|
||
| - Read-only. The operator **never writes** to the identity Secret. | ||
| - Reads `CLUSTER_IDENTIFIER` key from the named Secret in the operator's namespace. | ||
| - Returns empty string on any error (not found, key missing, permission denied) — never blocks startup. | ||
| - Logs a hint if the key is missing, telling the user to set `clusterIdentifier` in values.yaml. | ||
|
|
||
| #### `ReattachCluster` RPC (`internal/transport/dakr_client.go`) | ||
|
|
||
| Calls the `ReattachCluster` endpoint on the DevZero platform with: | ||
| - `pat_token` — the user's Personal Access Token | ||
| - `cluster_identifier` — the stable identifier | ||
| - `cluster_name` — from `kubeContextName` in values.yaml | ||
| - `k8s_provider` — from `k8sProvider` in values.yaml | ||
|
|
||
| The platform does a find-or-create: if a cluster with that identifier already exists under the PAT's account, it returns the existing cluster token. Otherwise it creates a new cluster with that identifier. | ||
|
|
||
| #### Token persistence | ||
|
|
||
| After a successful exchange (via either `ReattachCluster` or `ExchangePATForClusterToken`), the token **and the identifier** are persisted via `persistClusterToken`. This writes to either: | ||
| - **ConfigMap** (`devzero-zxporter-env-config`) — default when `useSecretForToken: false` | ||
| - **Kubernetes Secret** (`devzero-zxporter-token`) — when `useSecretForToken: true` | ||
|
|
||
| On the **next startup**, `tryRecoverStoredClusterToken` finds the stored token and skips the PAT exchange entirely. This means the PAT is only used once per "new cluster" event. | ||
|
|
||
| --- | ||
|
|
||
| ### Helm templates involved | ||
|
|
||
| | File | Purpose | | ||
| |---|---| | ||
| | `templates/secret-cluster-identity.yaml` | Auto-creates the identity Secret when `clusterIdentifier` is set. Annotated with `helm.sh/resource-policy: keep` so it survives `helm uninstall`. | | ||
| | `templates/configmap.yaml` | Includes `CLUSTER_IDENTITY_SECRET_NAME` so the operator knows which Secret to read. Only emits `CLUSTER_TOKEN` when non-empty (avoids Helm field ownership conflicts on upgrade). | | ||
| | `templates/zxporter-rbac.yaml` | Grants `get` on the identity Secret unconditionally (not gated by `useSecretForToken`), so the operator can always read the identifier. | | ||
|
|
||
| --- | ||
|
|
||
| ### RBAC | ||
|
|
||
| The operator needs read access to the cluster identity Secret. This is granted in the `devzero-zxporter-leader-election-role` (namespace-scoped Role): | ||
|
|
||
| ```yaml | ||
| - apiGroups: [""] | ||
| resourceNames: ["devzero-zxporter-cluster-identity"] | ||
| resources: ["secrets"] | ||
| verbs: ["get"] | ||
| ``` | ||
|
|
||
| This is **not** gated by `useSecretForToken` — the identity Secret is separate from the token storage Secret. | ||
|
|
||
| --- | ||
|
|
||
| ### Key design decisions | ||
|
|
||
| **Why a Secret instead of just values.yaml?** | ||
| - values.yaml changes on upgrade. If you accidentally clear `clusterIdentifier`, you'd create a new cluster. | ||
| - Secrets annotated with `helm.sh/resource-policy: keep` survive `helm uninstall`, making them the durable source of truth. | ||
|
|
||
| **Why is the operator read-only on the identity Secret?** | ||
| - The identity Secret is user-managed. The operator reading it doesn't need write access, which follows least-privilege. | ||
| - The token storage Secret (`devzero-zxporter-token`) is separately managed with read/write access. | ||
|
|
||
| **Why ConfigMap-as-volume instead of env vars?** | ||
| - ZXPorter uses `configmap` mounted as a volume at `/etc/zxporter/config/`. Files in that directory reflect ConfigMap changes without pod restarts. | ||
| - `CLUSTER_IDENTITY_SECRET_NAME` is passed via this mechanism. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this.