From 39e3ea6037de912c6c3dce968fca6238c7a33e9a Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 22 Apr 2026 21:44:19 -0500 Subject: [PATCH 01/44] feat: optional NodePool for dedicated psql workloads Add opt-in Karpenter NodePool composed resource. When spec.nodePool.enabled: true, renders a NodePool targeting arm64 spot on r7g.large/r7g.xlarge/m7g.large/m7g.xlarge (memory-optimized Graviton), tainted with psql=true:NoSchedule and labeled workload-type: psql. StackGres (operator/restapi/jobs) and Atlas get nodeSelector + tolerations injected into their Helm values when the NodePool is enabled. Usages pin both releases to be drained before the NodePool is deleted. Adds provider-kubernetes to upbound.yaml for the NodePool Object wrapper. Implements [[tasks/psql-stack-vela-simplyblock]] Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 22 ++++- apis/psqlstacks/definition.yaml | 56 +++++++++++++ examples/psqlstacks/standard.yaml | 2 + functions/render/000-state-init.yaml.gotmpl | 48 +++++++++++ functions/render/010-state-status.yaml.gotmpl | 3 +- functions/render/150-nodepool.yaml.gotmpl | 80 +++++++++++++++++++ .../render/200-stackgres-operator.yaml.gotmpl | 14 ++++ .../render/210-atlas-operator.yaml.gotmpl | 4 + upbound.yaml | 7 +- 9 files changed, 231 insertions(+), 5 deletions(-) create mode 100644 functions/render/150-nodepool.yaml.gotmpl diff --git a/README.md b/README.md index d9a50ac..99fa22c 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,8 @@ PostgreSQL management stack deploying StackGres and Atlas Operator as Helm relea - **StackGres Operator** — Full PostgreSQL lifecycle management with native Citus support for distributed PostgreSQL via `SGShardedCluster` CRDs - **Atlas Operator** — Declarative database schema migrations via `AtlasMigration` and `AtlasSchema` CRDs -- **Usage resource** — Ensures Atlas is deleted before StackGres to prevent orphaned migration state +- **Karpenter NodePool** *(opt-in, `nodePool.enabled: true`)* — Dedicated nodes for database workloads. Default: arm64 spot on `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` (memory-optimized Graviton for cheap, low-contention scheduling). StackGres operator + REST API + jobs and Atlas are pinned here via nodeSelector + tolerations. +- **Usage resources** — Atlas is deleted before StackGres to prevent orphaned migration state; Helm releases are deleted before the NodePool so pods drain cleanly. ## The Journey @@ -40,7 +41,7 @@ spec: ### Stage 2: Team Usage -Add labels for ownership tracking and customize Helm values per operator. +Add labels for ownership tracking, pin operators to a dedicated NodePool, and customize Helm values. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -53,6 +54,8 @@ spec: namespace: stackgres labels: team: platform + nodePool: + enabled: true stackgresOperator: values: deploy: @@ -114,6 +117,15 @@ spec: | `managementPolicies` | string[] | No | `["*"]` | Crossplane management policies | | `helmProviderConfigRef.name` | string | No | `clusterName` | Helm ProviderConfig name | | `helmProviderConfigRef.kind` | enum | No | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` | +| `kubernetesProviderConfigRef.name` | string | No | `clusterName` | Kubernetes ProviderConfig name (for the NodePool Object) | +| `kubernetesProviderConfigRef.kind` | enum | No | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` | +| `nodePool.enabled` | boolean | No | `false` | Create a dedicated Karpenter NodePool and schedule operators on it | +| `nodePool.nodeClassName` | string | No | `default` | EKS NodeClass name | +| `nodePool.limits.cpu` | string | No | `16` | Pool CPU limit | +| `nodePool.limits.memory` | string | No | `64Gi` | Pool memory limit | +| `nodePool.requirements` | array | No | arm64 spot `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` | Karpenter scheduling requirements | +| `nodePool.disruption.consolidationPolicy` | enum | No | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | +| `nodePool.disruption.consolidateAfter` | string | No | `60s` | Consolidation delay | | `stackgresOperator.name` | string | No | `stackgres-operator` | Helm release name | | `stackgresOperator.namespace` | string | No | shared `namespace` | Namespace override | | `stackgresOperator.values` | object | No | — | Helm values merged with chart defaults | @@ -154,9 +166,12 @@ prewarmDevDB: true | Resource | Kind | Purpose | |----------|------|---------| +| `nodepool-psql` | `kubernetes.m.crossplane.io/Object` | Karpenter NodePool (only when `nodePool.enabled: true`) | | `stackgres-operator` | `helm.m.crossplane.io/Release` | StackGres Helm release | | `atlas-operator` | `helm.m.crossplane.io/Release` | Atlas Operator Helm release | -| `usage-sg-atlas` | `protection.crossplane.io/Usage` | Deletion ordering (created once both operators are ready) | +| `usage-sg-atlas` | `protection.crossplane.io/Usage` | Atlas deleted before StackGres | +| `usage-np-stackgres-operator` | `protection.crossplane.io/Usage` | StackGres drained before NodePool is deleted (when NodePool enabled) | +| `usage-np-atlas-operator` | `protection.crossplane.io/Usage` | Atlas drained before NodePool is deleted (when NodePool enabled) | ## Dependencies @@ -164,6 +179,7 @@ prewarmDevDB: true |------|---------|---------| | Function | `crossplane-contrib/function-auto-ready` | `>=v0.6.0` | | Provider | `crossplane-contrib/provider-helm` | `>=v1` | +| Provider | `crossplane-contrib/provider-kubernetes` | `>=v1` (only used when `nodePool.enabled`) | ## Development diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index baac1fe..eae2cc3 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -50,9 +50,65 @@ spec: enum: - ProviderConfig - ClusterProviderConfig + kubernetesProviderConfigRef: + description: Reference to the Kubernetes ProviderConfig (used to create the Karpenter NodePool Object). Defaults to clusterName. + type: object + properties: + name: + description: Name of the Kubernetes ProviderConfig. + type: string + kind: + description: Kind of the ProviderConfig. Defaults to ProviderConfig. + type: string + enum: + - ProviderConfig + - ClusterProviderConfig namespace: description: Shared namespace for all components. Defaults to stackgres. Per-component namespace overrides this. type: string + nodePool: + description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, StackGres operator and Atlas schedule here via nodeSelector + tolerations. + type: object + properties: + enabled: + type: boolean + default: false + nodeClassName: + description: EKS NodeClass to reference. Defaults to "default". + type: string + limits: + type: object + properties: + cpu: + type: string + memory: + type: string + requirements: + description: Karpenter scheduling requirements. Defaults to arm64 spot on r7g/m7g (memory-optimized, general-purpose). + type: array + items: + type: object + properties: + key: + type: string + operator: + type: string + enum: [In, NotIn, Exists, DoesNotExist, Gt, Lt] + values: + type: array + items: + type: string + required: [key, operator] + disruption: + type: object + properties: + consolidationPolicy: + type: string + enum: [WhenEmpty, WhenEmptyOrUnderutilized] + default: WhenEmptyOrUnderutilized + consolidateAfter: + type: string + default: "60s" stackgresOperator: description: Configuration for the StackGres operator component. type: object diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index db1fab6..2907643 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -8,6 +8,8 @@ spec: namespace: stackgres labels: team: platform + nodePool: + enabled: true stackgresOperator: values: deploy: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 6295148..c89d851 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -29,6 +29,41 @@ "kind" ($helmProviderConfigRef.kind | default "ProviderConfig") }} +# Kubernetes provider config (defaults to clusterName) +{{- $k8sProviderConfigRef := $spec.kubernetesProviderConfigRef | default dict }} +{{- $k8sProviderConfigRef = dict + "name" ($k8sProviderConfigRef.name | default $clusterName) + "kind" ($k8sProviderConfigRef.kind | default "ProviderConfig") +}} + +# ============================================================================== +# NodePool (opt-in) — dedicates a pool of nodes for PostgreSQL workloads. +# Defaults: arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). +# ============================================================================== +{{- $nodePoolSpec := $spec.nodePool | default dict }} +{{- $nodePoolEnabled := false }} +{{- if hasKey $nodePoolSpec "enabled" }} + {{- $nodePoolEnabled = $nodePoolSpec.enabled }} +{{- end }} +{{- $nodePoolName := printf "%s-psql" $clusterName }} +{{- $nodePoolNodeClassName := $nodePoolSpec.nodeClassName | default "default" }} +{{- $nodePoolLimits := $nodePoolSpec.limits | default (dict "cpu" "16" "memory" "64Gi") }} +{{- $nodePoolRequirements := $nodePoolSpec.requirements | default (list + (dict "key" "kubernetes.io/arch" "operator" "In" "values" (list "arm64")) + (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "spot")) + (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" (list "r7g.large" "r7g.xlarge" "m7g.large" "m7g.xlarge")) +) }} +{{- $nodePoolDisruption := $nodePoolSpec.disruption | default (dict "consolidationPolicy" "WhenEmptyOrUnderutilized" "consolidateAfter" "60s") }} + +{{- $nodePoolTaintKey := "psql" }} +{{- $nodePoolTaintValue := "true" }} +{{- $nodePoolNodeSelector := dict }} +{{- $nodePoolTolerations := list }} +{{- if $nodePoolEnabled }} + {{- $nodePoolNodeSelector = dict "workload-type" "psql" }} + {{- $nodePoolTolerations = list (dict "key" $nodePoolTaintKey "value" $nodePoolTaintValue "effect" "NoSchedule") }} +{{- end }} + # ============================================================================== # Per-component configuration # ============================================================================== @@ -45,6 +80,19 @@ "managementPolicies" $managementPolicies "labels" $labels "helmProviderConfigRef" $helmProviderConfigRef + "kubernetesProviderConfigRef" $k8sProviderConfigRef + "nodePool" (dict + "enabled" $nodePoolEnabled + "name" $nodePoolName + "nodeClassName" $nodePoolNodeClassName + "limits" $nodePoolLimits + "requirements" $nodePoolRequirements + "disruption" $nodePoolDisruption + "taintKey" $nodePoolTaintKey + "taintValue" $nodePoolTaintValue + "nodeSelector" $nodePoolNodeSelector + "tolerations" $nodePoolTolerations + ) "stackgresOperator" (dict "name" ($sg.name | default "stackgres-operator") "namespace" ($sg.namespace | default $namespace) diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index fbe3ef3..d067313 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "stackgres-operator" "atlas-operator" }} +{{- range $key := list "stackgres-operator" "atlas-operator" "nodepool-psql" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -29,6 +29,7 @@ {{- $state = set $state "observed" (dict "stackgresOperator" (dict "ready" (get $checkReady "stackgres-operator")) "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) + "nodepoolPsql" (dict "ready" (get $checkReady "nodepool-psql")) ) }} # ============================================================================== diff --git a/functions/render/150-nodepool.yaml.gotmpl b/functions/render/150-nodepool.yaml.gotmpl new file mode 100644 index 0000000..b88a9ef --- /dev/null +++ b/functions/render/150-nodepool.yaml.gotmpl @@ -0,0 +1,80 @@ +# code: language=yaml +# +# Karpenter NodePool for dedicated psql workloads. +# StackGres operator + REST API + jobs and the Atlas operator are scheduled +# here via nodeSelector + tolerations. Opt-in via spec.nodePool.enabled. +# + +{{- if $state.nodePool.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-nodepool-psql + annotations: + {{ setResourceNameAnnotation "nodepool-psql" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: karpenter.sh/v1 + kind: NodePool + metadata: + name: {{ $state.nodePool.name }} + spec: + template: + metadata: + labels: + workload-type: psql + spec: + nodeClassRef: + group: eks.amazonaws.com + kind: NodeClass + name: {{ $state.nodePool.nodeClassName }} + taints: + - key: {{ $state.nodePool.taintKey }} + value: {{ $state.nodePool.taintValue | quote }} + effect: NoSchedule + requirements: {{ $state.nodePool.requirements | toJson }} + limits: {{ $state.nodePool.limits | toJson }} + disruption: {{ $state.nodePool.disruption | toJson }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# ============================================================================== +# Usages: protect NodePool from being deleted before the Helm releases. +# If the NodePool is deleted first, operator pods lose their nodes and the +# release cleanup hangs. +# ============================================================================== +{{- $pinnedReleases := dict + "stackgres-operator" "stackgresOperator" + "atlas-operator" "atlasOperator" +}} +{{- range $release, $obsKey := $pinnedReleases }} + {{- $releaseReady := (get (get $state.observed $obsKey | default dict) "ready") }} + {{- if and $state.observed.nodepoolPsql.ready $releaseReady }} +--- +apiVersion: protection.crossplane.io/v1beta1 +kind: Usage +metadata: + name: {{ $state.name }}-delete-{{ $release }}-before-nodepool + annotations: + {{ setResourceNameAnnotation (printf "usage-np-%s" $release) }} + labels: {{ $state.labels | toJson }} +spec: + replayDeletion: true + of: + apiVersion: kubernetes.m.crossplane.io/v1alpha1 + kind: Object + resourceRef: + name: {{ $state.name }}-nodepool-psql + by: + apiVersion: helm.m.crossplane.io/v1beta1 + kind: Release + resourceRef: + name: {{ $release }} + {{- end }} +{{- end }} +{{- end }} diff --git a/functions/render/200-stackgres-operator.yaml.gotmpl b/functions/render/200-stackgres-operator.yaml.gotmpl index fb15bb3..47c1a02 100644 --- a/functions/render/200-stackgres-operator.yaml.gotmpl +++ b/functions/render/200-stackgres-operator.yaml.gotmpl @@ -30,11 +30,25 @@ spec: {{- toYaml $sg.overrideAllValues | nindent 6 }} {{- else }} {{- /* Chart defaults */}} + {{- $operatorDefaults := dict }} + {{- $restapiDefaults := dict }} + {{- $jobsDefaults := dict }} + {{- if $state.nodePool.enabled }} + {{- $_ := set $operatorDefaults "nodeSelector" $state.nodePool.nodeSelector }} + {{- $_ := set $operatorDefaults "tolerations" $state.nodePool.tolerations }} + {{- $_ := set $restapiDefaults "nodeSelector" $state.nodePool.nodeSelector }} + {{- $_ := set $restapiDefaults "tolerations" $state.nodePool.tolerations }} + {{- $_ := set $jobsDefaults "nodeSelector" $state.nodePool.nodeSelector }} + {{- $_ := set $jobsDefaults "tolerations" $state.nodePool.tolerations }} + {{- end }} {{- $chartDefaults := dict "deploy" (dict "operator" true "restapi" true ) + "operator" $operatorDefaults + "restapi" $restapiDefaults + "jobs" $jobsDefaults }} {{- $mergedValues := mergeOverwrite $chartDefaults ($sg.values | default dict) }} values: diff --git a/functions/render/210-atlas-operator.yaml.gotmpl b/functions/render/210-atlas-operator.yaml.gotmpl index 9cbb1dc..1d1a1cf 100644 --- a/functions/render/210-atlas-operator.yaml.gotmpl +++ b/functions/render/210-atlas-operator.yaml.gotmpl @@ -31,6 +31,10 @@ spec: {{- $chartDefaults := dict "prewarmDevDB" true }} + {{- if $state.nodePool.enabled }} + {{- $_ := set $chartDefaults "nodeSelector" $state.nodePool.nodeSelector }} + {{- $_ := set $chartDefaults "tolerations" $state.nodePool.tolerations }} + {{- end }} {{- $mergedValues := mergeOverwrite $chartDefaults ($atlas.values | default dict) }} values: {{- toYaml $mergedValues | nindent 6 }} diff --git a/upbound.yaml b/upbound.yaml index 612c6b7..3f57864 100644 --- a/upbound.yaml +++ b/upbound.yaml @@ -12,8 +12,13 @@ spec: kind: Provider package: xpkg.crossplane.io/crossplane-contrib/provider-helm version: '>=v1' + - apiVersion: pkg.crossplane.io/v1 + kind: Provider + package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes + version: '>=v1' description: PostgreSQL stack deploying StackGres operator (with Citus support) - and Atlas Operator for schema migrations as Helm releases. + and Atlas Operator for schema migrations as Helm releases, with an optional dedicated + Karpenter NodePool for database workloads. license: Apache-2.0 maintainer: Patrick Lee Scott readme: | From 46fe4c900164a62380eb928ded6bb1767426f870 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 22 Apr 2026 22:26:22 -0500 Subject: [PATCH 02/44] feat: StorageClass + ExternalSecrets composed resources - storageClass (default on): creates a gp3 StorageClass backed by the EKS Auto Mode EBS CSI driver (ebs.csi.eks.amazonaws.com). The legacy gp2 in-tree provisioner does not work on EKS Auto Mode. - externalSecrets (opt-in): for each entry in externalSecrets.secrets[], composes a kubernetes.m.crossplane.io/Object wrapping an ESO ExternalSecret that syncs an AWS Secrets Manager value (published via hops secrets sync aws) into a Kubernetes Secret on the target cluster. Requires a ClusterSecretStore (e.g. from SecretStack); defaults to clusterSecretStoreName: hops-aws-secrets-manager. Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + README.md | 17 +++++ apis/psqlstacks/definition.yaml | 59 +++++++++++++++++ examples/psqlstacks/standard.yaml | 8 +++ functions/render/000-state-init.yaml.gotmpl | 64 +++++++++++++++++++ functions/render/120-storageclass.yaml.gotmpl | 34 ++++++++++ .../render/130-external-secrets.yaml.gotmpl | 55 ++++++++++++++++ 7 files changed, 238 insertions(+) create mode 100644 functions/render/120-storageclass.yaml.gotmpl create mode 100644 functions/render/130-external-secrets.yaml.gotmpl diff --git a/.gitignore b/.gitignore index c7d6289..1ef5a9e 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ _output/ # E2E test credentials (never commit secrets) **/aws-creds tests/**/secrets/ +apis/**/configuration.yaml diff --git a/README.md b/README.md index 99fa22c..59dc120 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,8 @@ PostgreSQL management stack deploying StackGres and Atlas Operator as Helm relea - **StackGres Operator** — Full PostgreSQL lifecycle management with native Citus support for distributed PostgreSQL via `SGShardedCluster` CRDs - **Atlas Operator** — Declarative database schema migrations via `AtlasMigration` and `AtlasSchema` CRDs +- **StorageClass** *(on by default, `storageClass.create: true`)* — `gp3` class backed by the EKS Auto Mode EBS CSI driver (`ebs.csi.eks.amazonaws.com`). The legacy `gp2` class on EKS Auto Mode uses a deprecated in-tree provisioner that no longer works. +- **ExternalSecret(s)** *(opt-in, `externalSecrets.enabled: true`)* — For each entry in `externalSecrets.secrets[]`, creates an ESO `ExternalSecret` that syncs an AWS Secrets Manager value into a Kubernetes Secret. Values are published with `hops secrets sync aws`; a `ClusterSecretStore` (e.g. from `SecretStack`) must already exist on the target cluster. - **Karpenter NodePool** *(opt-in, `nodePool.enabled: true`)* — Dedicated nodes for database workloads. Default: arm64 spot on `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` (memory-optimized Graviton for cheap, low-contention scheduling). StackGres operator + REST API + jobs and Atlas are pinned here via nodeSelector + tolerations. - **Usage resources** — Atlas is deleted before StackGres to prevent orphaned migration state; Helm releases are deleted before the NodePool so pods drain cleanly. @@ -126,6 +128,19 @@ spec: | `nodePool.requirements` | array | No | arm64 spot `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` | Karpenter scheduling requirements | | `nodePool.disruption.consolidationPolicy` | enum | No | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | | `nodePool.disruption.consolidateAfter` | string | No | `60s` | Consolidation delay | +| `storageClass.create` | boolean | No | `true` | Create a StorageClass on the target cluster | +| `storageClass.name` | string | No | `gp3` | StorageClass name | +| `storageClass.provisioner` | string | No | `ebs.csi.eks.amazonaws.com` | CSI provisioner | +| `storageClass.parameters` | object | No | `{type: gp3, fsType: ext4}` | Provisioner parameters | +| `storageClass.volumeBindingMode` | enum | No | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | +| `storageClass.allowVolumeExpansion` | boolean | No | `true` | Allow PVC resize | +| `storageClass.reclaimPolicy` | enum | No | `Delete` | `Delete` or `Retain` | +| `externalSecrets.enabled` | boolean | No | `false` | Enable ESO integration | +| `externalSecrets.clusterSecretStoreName` | string | No | `hops-aws-secrets-manager` | Name of the ClusterSecretStore | +| `externalSecrets.refreshInterval` | string | No | `1h` | ESO refresh interval | +| `externalSecrets.secrets[].path` | string | Yes | — | AWS Secrets Manager secret name (what `hops secrets sync aws` writes) | +| `externalSecrets.secrets[].name` | string | No | derived from `path` | Target K8s Secret name | +| `externalSecrets.secrets[].namespace` | string | No | stack `namespace` | Target namespace | | `stackgresOperator.name` | string | No | `stackgres-operator` | Helm release name | | `stackgresOperator.namespace` | string | No | shared `namespace` | Namespace override | | `stackgresOperator.values` | object | No | — | Helm values merged with chart defaults | @@ -166,6 +181,8 @@ prewarmDevDB: true | Resource | Kind | Purpose | |----------|------|---------| +| `storageclass` | `kubernetes.m.crossplane.io/Object` | StorageClass (default `gp3`; when `storageClass.create: true`) | +| `extsecret-` | `kubernetes.m.crossplane.io/Object` | One per `externalSecrets.secrets[]` entry; wraps an ESO `ExternalSecret` | | `nodepool-psql` | `kubernetes.m.crossplane.io/Object` | Karpenter NodePool (only when `nodePool.enabled: true`) | | `stackgres-operator` | `helm.m.crossplane.io/Release` | StackGres Helm release | | `atlas-operator` | `helm.m.crossplane.io/Release` | Atlas Operator Helm release | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index eae2cc3..1cd4927 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -66,6 +66,65 @@ spec: namespace: description: Shared namespace for all components. Defaults to stackgres. Per-component namespace overrides this. type: string + storageClass: + description: "StorageClass for PostgreSQL workloads. Default provisioner is the EKS Auto Mode EBS CSI driver. Disable with create=false if the cluster already provides a suitable StorageClass." + type: object + properties: + create: + type: boolean + default: true + name: + description: StorageClass name. Defaults to "gp3". + type: string + provisioner: + description: CSI provisioner. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). + type: string + parameters: + description: Provisioner parameters. Defaults to gp3 + ext4. + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true + reclaimPolicy: + type: string + enum: [Delete, Retain] + default: Delete + volumeBindingMode: + type: string + enum: [Immediate, WaitForFirstConsumer] + default: WaitForFirstConsumer + allowVolumeExpansion: + type: boolean + default: true + externalSecrets: + description: External Secrets Operator integration. Requires a ClusterSecretStore to already exist on the target cluster (provisioned by SecretStack or equivalent). + type: object + properties: + enabled: + type: boolean + default: false + clusterSecretStoreName: + description: Name of the ClusterSecretStore to read from. Defaults to "hops-aws-secrets-manager". + type: string + refreshInterval: + description: How often ESO should re-pull the secret. Defaults to 1h. + type: string + secrets: + description: List of AWS Secrets Manager paths to sync into Kubernetes Secrets. + type: array + items: + type: object + properties: + path: + description: AWS Secrets Manager secret name (what `hops secrets sync aws` writes). + type: string + name: + description: Target Kubernetes Secret name. Defaults to the last path segment with "/" replaced by "-". + type: string + namespace: + description: Target namespace. Defaults to the stack namespace. + type: string + required: [path] nodePool: description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, StackGres operator and Atlas schedule here via nodeSelector + tolerations. type: object diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index 2907643..bd91434 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -10,6 +10,14 @@ spec: team: platform nodePool: enabled: true + storageClass: + create: true + name: gp3 + externalSecrets: + enabled: true + secrets: + - path: psql/production + name: psql-production stackgresOperator: values: deploy: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index c89d851..222a41c 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -36,6 +36,55 @@ "kind" ($k8sProviderConfigRef.kind | default "ProviderConfig") }} +# ============================================================================== +# StorageClass (opt-in) — creates a gp3 StorageClass on the target cluster. +# EKS Auto Mode uses ebs.csi.eks.amazonaws.com; the in-tree kubernetes.io/aws-ebs +# provisioner (used by the legacy "gp2" class) does not work there. +# ============================================================================== +{{- $sc := $spec.storageClass | default dict }} +{{- $scCreate := true }} +{{- if hasKey $sc "create" }} + {{- $scCreate = $sc.create }} +{{- end }} +{{- $scName := $sc.name | default "gp3" }} +{{- $scProvisioner := $sc.provisioner | default "ebs.csi.eks.amazonaws.com" }} +{{- $scParameters := $sc.parameters | default (dict "type" "gp3" "fsType" "ext4") }} +{{- $scReclaimPolicy := $sc.reclaimPolicy | default "Delete" }} +{{- $scVolumeBindingMode := $sc.volumeBindingMode | default "WaitForFirstConsumer" }} +{{- $scAllowVolumeExpansion := true }} +{{- if hasKey $sc "allowVolumeExpansion" }} + {{- $scAllowVolumeExpansion = $sc.allowVolumeExpansion }} +{{- end }} + +# ============================================================================== +# ExternalSecrets (opt-in) — sync AWS Secrets Manager values into K8s Secrets +# via ESO. Requires a ClusterSecretStore to already exist on the target cluster. +# ============================================================================== +{{- $esoSpec := $spec.externalSecrets | default dict }} +{{- $esoEnabled := false }} +{{- if hasKey $esoSpec "enabled" }} + {{- $esoEnabled = $esoSpec.enabled }} +{{- end }} +{{- $esoStore := $esoSpec.clusterSecretStoreName | default "hops-aws-secrets-manager" }} +{{- $esoRefreshInterval := $esoSpec.refreshInterval | default "1h" }} +{{- $esoInputSecrets := $esoSpec.secrets | default list }} +{{- $esoSecrets := list }} +{{- range $s := $esoInputSecrets }} + {{- $targetName := $s.name }} + {{- if not $targetName }} + {{- /* default K8s Secret name: last path segment, / → - */}} + {{- $parts := splitList "/" $s.path }} + {{- $targetName = index $parts (sub (len $parts) 1) }} + {{- $targetName = replace "_" "-" $targetName }} + {{- end }} + {{- $targetNs := $s.namespace | default $namespace }} + {{- $esoSecrets = append $esoSecrets (dict + "path" $s.path + "name" $targetName + "namespace" $targetNs + ) }} +{{- end }} + # ============================================================================== # NodePool (opt-in) — dedicates a pool of nodes for PostgreSQL workloads. # Defaults: arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). @@ -81,6 +130,21 @@ "labels" $labels "helmProviderConfigRef" $helmProviderConfigRef "kubernetesProviderConfigRef" $k8sProviderConfigRef + "storageClass" (dict + "create" $scCreate + "name" $scName + "provisioner" $scProvisioner + "parameters" $scParameters + "reclaimPolicy" $scReclaimPolicy + "volumeBindingMode" $scVolumeBindingMode + "allowVolumeExpansion" $scAllowVolumeExpansion + ) + "externalSecrets" (dict + "enabled" $esoEnabled + "clusterSecretStoreName" $esoStore + "refreshInterval" $esoRefreshInterval + "secrets" $esoSecrets + ) "nodePool" (dict "enabled" $nodePoolEnabled "name" $nodePoolName diff --git a/functions/render/120-storageclass.yaml.gotmpl b/functions/render/120-storageclass.yaml.gotmpl new file mode 100644 index 0000000..befa049 --- /dev/null +++ b/functions/render/120-storageclass.yaml.gotmpl @@ -0,0 +1,34 @@ +# code: language=yaml +# +# StorageClass for PostgreSQL workloads. +# Opt-in via spec.storageClass.enabled. Defaults to gp3 on the EKS Auto Mode +# EBS CSI driver (ebs.csi.eks.amazonaws.com). +# + +{{- if $state.storageClass.create }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-storageclass + annotations: + {{ setResourceNameAnnotation "storageclass" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: {{ $state.storageClass.name }} + labels: {{ $state.labels | toJson }} + provisioner: {{ $state.storageClass.provisioner }} + parameters: {{ $state.storageClass.parameters | toJson }} + reclaimPolicy: {{ $state.storageClass.reclaimPolicy }} + volumeBindingMode: {{ $state.storageClass.volumeBindingMode }} + allowVolumeExpansion: {{ $state.storageClass.allowVolumeExpansion }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/130-external-secrets.yaml.gotmpl b/functions/render/130-external-secrets.yaml.gotmpl new file mode 100644 index 0000000..deac3f6 --- /dev/null +++ b/functions/render/130-external-secrets.yaml.gotmpl @@ -0,0 +1,55 @@ +# code: language=yaml +# +# ExternalSecret resources: sync AWS Secrets Manager values into K8s Secrets +# on the target cluster via ESO. +# +# Each `spec.externalSecrets.secrets[]` entry becomes one ExternalSecret CR +# wrapped in a kubernetes.m.crossplane.io/Object for Crossplane-managed +# reconciliation. The resulting K8s Secret mirrors every JSON key from the +# AWS Secrets Manager secret (dataFrom.extract) — consumers reference +# specific keys via secretKeyRef on the produced Secret. +# +# Prerequisites: +# - ESO installed on the target cluster +# - A ClusterSecretStore pointing at AWS Secrets Manager (e.g. from SecretStack) +# - `hops secrets sync aws` has published values to the configured paths +# + +{{- $eso := $state.externalSecrets }} + +{{- if $eso.enabled }} +{{- range $idx, $sec := $eso.secrets }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-extsecret-{{ $sec.name }} + annotations: + {{ setResourceNameAnnotation (printf "extsecret-%s" $sec.name) }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: external-secrets.io/v1 + kind: ExternalSecret + metadata: + name: {{ $sec.name }} + namespace: {{ $sec.namespace }} + labels: {{ $state.labels | toJson }} + spec: + refreshInterval: {{ $eso.refreshInterval }} + secretStoreRef: + name: {{ $eso.clusterSecretStoreName }} + kind: ClusterSecretStore + target: + name: {{ $sec.name }} + creationPolicy: Owner + dataFrom: + - extract: + key: {{ $sec.path }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} +{{- end }} From 3a2946f149bdab7021dcefd3b9428bd42e3f1be1 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 22 Apr 2026 22:27:45 -0500 Subject: [PATCH 03/44] fix: default StorageClass name to 'psql' (per-stack naming) Mirrors observe stack where StorageClasses are named loki/prometheus/tempo per-component. Keeps the name specific to the stack so it doesn't collide with cluster-provided defaults. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 6 +++--- apis/psqlstacks/definition.yaml | 2 +- examples/psqlstacks/standard.yaml | 2 +- functions/render/000-state-init.yaml.gotmpl | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 59dc120..ad5ce31 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ PostgreSQL management stack deploying StackGres and Atlas Operator as Helm relea - **StackGres Operator** — Full PostgreSQL lifecycle management with native Citus support for distributed PostgreSQL via `SGShardedCluster` CRDs - **Atlas Operator** — Declarative database schema migrations via `AtlasMigration` and `AtlasSchema` CRDs -- **StorageClass** *(on by default, `storageClass.create: true`)* — `gp3` class backed by the EKS Auto Mode EBS CSI driver (`ebs.csi.eks.amazonaws.com`). The legacy `gp2` class on EKS Auto Mode uses a deprecated in-tree provisioner that no longer works. +- **StorageClass** *(on by default, `storageClass.create: true`)* — `psql` class backed by the EKS Auto Mode EBS CSI driver (`ebs.csi.eks.amazonaws.com`). Name mirrors the per-stack convention used by the observe stack (`loki`/`prometheus`/`tempo`). The legacy `gp2` class on EKS Auto Mode uses a deprecated in-tree provisioner that no longer works. - **ExternalSecret(s)** *(opt-in, `externalSecrets.enabled: true`)* — For each entry in `externalSecrets.secrets[]`, creates an ESO `ExternalSecret` that syncs an AWS Secrets Manager value into a Kubernetes Secret. Values are published with `hops secrets sync aws`; a `ClusterSecretStore` (e.g. from `SecretStack`) must already exist on the target cluster. - **Karpenter NodePool** *(opt-in, `nodePool.enabled: true`)* — Dedicated nodes for database workloads. Default: arm64 spot on `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` (memory-optimized Graviton for cheap, low-contention scheduling). StackGres operator + REST API + jobs and Atlas are pinned here via nodeSelector + tolerations. - **Usage resources** — Atlas is deleted before StackGres to prevent orphaned migration state; Helm releases are deleted before the NodePool so pods drain cleanly. @@ -129,7 +129,7 @@ spec: | `nodePool.disruption.consolidationPolicy` | enum | No | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | | `nodePool.disruption.consolidateAfter` | string | No | `60s` | Consolidation delay | | `storageClass.create` | boolean | No | `true` | Create a StorageClass on the target cluster | -| `storageClass.name` | string | No | `gp3` | StorageClass name | +| `storageClass.name` | string | No | `psql` | StorageClass name (mirrors observe stack's per-stack naming) | | `storageClass.provisioner` | string | No | `ebs.csi.eks.amazonaws.com` | CSI provisioner | | `storageClass.parameters` | object | No | `{type: gp3, fsType: ext4}` | Provisioner parameters | | `storageClass.volumeBindingMode` | enum | No | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | @@ -181,7 +181,7 @@ prewarmDevDB: true | Resource | Kind | Purpose | |----------|------|---------| -| `storageclass` | `kubernetes.m.crossplane.io/Object` | StorageClass (default `gp3`; when `storageClass.create: true`) | +| `storageclass` | `kubernetes.m.crossplane.io/Object` | StorageClass (default name `psql`; when `storageClass.create: true`) | | `extsecret-` | `kubernetes.m.crossplane.io/Object` | One per `externalSecrets.secrets[]` entry; wraps an ESO `ExternalSecret` | | `nodepool-psql` | `kubernetes.m.crossplane.io/Object` | Karpenter NodePool (only when `nodePool.enabled: true`) | | `stackgres-operator` | `helm.m.crossplane.io/Release` | StackGres Helm release | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 1cd4927..8ccdcec 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -74,7 +74,7 @@ spec: type: boolean default: true name: - description: StorageClass name. Defaults to "gp3". + description: StorageClass name. Defaults to "psql" (mirrors per-stack naming used by the observe stack — loki/prometheus/tempo). type: string provisioner: description: CSI provisioner. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index bd91434..1374000 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -12,7 +12,7 @@ spec: enabled: true storageClass: create: true - name: gp3 + name: psql externalSecrets: enabled: true secrets: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 222a41c..968af9d 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -46,7 +46,7 @@ {{- if hasKey $sc "create" }} {{- $scCreate = $sc.create }} {{- end }} -{{- $scName := $sc.name | default "gp3" }} +{{- $scName := $sc.name | default "psql" }} {{- $scProvisioner := $sc.provisioner | default "ebs.csi.eks.amazonaws.com" }} {{- $scParameters := $sc.parameters | default (dict "type" "gp3" "fsType" "ext4") }} {{- $scReclaimPolicy := $sc.reclaimPolicy | default "Delete" }} From 622fae59a4c28fb9ea288c5dd7ab047bb67d02f1 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 22 Apr 2026 23:03:54 -0500 Subject: [PATCH 04/44] =?UTF-8?q?feat:=20externalSecrets.connections[]=20c?= =?UTF-8?q?omposes=20password=20+=20config=20=E2=86=92=20URL?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the generic externalSecrets.secrets[] passthrough with a Postgres-specific connections[] API. The user publishes just a password via hops secrets (single JSON key, default 'password'); the stack combines it with non-secret host/port/database/username/sslmode/namespace and emits a K8s Secret with a ready-to-use 'url' key plus discrete fields. Downstream consumers reference whichever key they need: AtlasSchema.devURLFrom → url SGCluster credentials.users.superuser.password → password applications → url (or discrete fields) Breaking change to the (locally-only) externalSecrets API; redeploy with the new shape. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 16 ++++-- apis/psqlstacks/definition.yaml | 34 ++++++++++--- examples/psqlstacks/standard.yaml | 9 ++-- functions/render/000-state-init.yaml.gotmpl | 33 ++++++------ .../render/130-external-secrets.yaml.gotmpl | 51 ++++++++++++------- 5 files changed, 95 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index ad5ce31..c48a0ec 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ PostgreSQL management stack deploying StackGres and Atlas Operator as Helm relea - **StackGres Operator** — Full PostgreSQL lifecycle management with native Citus support for distributed PostgreSQL via `SGShardedCluster` CRDs - **Atlas Operator** — Declarative database schema migrations via `AtlasMigration` and `AtlasSchema` CRDs - **StorageClass** *(on by default, `storageClass.create: true`)* — `psql` class backed by the EKS Auto Mode EBS CSI driver (`ebs.csi.eks.amazonaws.com`). Name mirrors the per-stack convention used by the observe stack (`loki`/`prometheus`/`tempo`). The legacy `gp2` class on EKS Auto Mode uses a deprecated in-tree provisioner that no longer works. -- **ExternalSecret(s)** *(opt-in, `externalSecrets.enabled: true`)* — For each entry in `externalSecrets.secrets[]`, creates an ESO `ExternalSecret` that syncs an AWS Secrets Manager value into a Kubernetes Secret. Values are published with `hops secrets sync aws`; a `ClusterSecretStore` (e.g. from `SecretStack`) must already exist on the target cluster. +- **Postgres connection Secret(s)** *(opt-in, `externalSecrets.enabled: true`)* — For each entry in `externalSecrets.connections[]`, composes an ESO `ExternalSecret` that pulls a single password from AWS Secrets Manager (published with `hops secrets sync aws`) and templates it with non-secret config (host/port/database/username/sslmode) into a K8s Secret with keys `url`, `host`, `port`, `username`, `database`, `sslmode`, `password`. Consumers reference whichever key they need: `AtlasSchema.devURLFrom` → `url`, `SGCluster.spec.configurations.credentials.users.superuser.password` → `password`, etc. - **Karpenter NodePool** *(opt-in, `nodePool.enabled: true`)* — Dedicated nodes for database workloads. Default: arm64 spot on `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` (memory-optimized Graviton for cheap, low-contention scheduling). StackGres operator + REST API + jobs and Atlas are pinned here via nodeSelector + tolerations. - **Usage resources** — Atlas is deleted before StackGres to prevent orphaned migration state; Helm releases are deleted before the NodePool so pods drain cleanly. @@ -138,9 +138,15 @@ spec: | `externalSecrets.enabled` | boolean | No | `false` | Enable ESO integration | | `externalSecrets.clusterSecretStoreName` | string | No | `hops-aws-secrets-manager` | Name of the ClusterSecretStore | | `externalSecrets.refreshInterval` | string | No | `1h` | ESO refresh interval | -| `externalSecrets.secrets[].path` | string | Yes | — | AWS Secrets Manager secret name (what `hops secrets sync aws` writes) | -| `externalSecrets.secrets[].name` | string | No | derived from `path` | Target K8s Secret name | -| `externalSecrets.secrets[].namespace` | string | No | stack `namespace` | Target namespace | +| `externalSecrets.connections[].name` | string | Yes | — | K8s Secret name (also the connection identifier) | +| `externalSecrets.connections[].namespace` | string | No | stack `namespace` | K8s Secret namespace | +| `externalSecrets.connections[].passwordPath` | string | Yes | — | AWS Secrets Manager secret name containing the password | +| `externalSecrets.connections[].passwordKey` | string | No | `password` | JSON key in the AWS secret holding the password | +| `externalSecrets.connections[].host` | string | Yes | — | Postgres host (e.g. `test-pg.stackgres.svc.cluster.local`) | +| `externalSecrets.connections[].port` | integer | No | `5432` | Postgres port | +| `externalSecrets.connections[].username` | string | No | `postgres` | Postgres username | +| `externalSecrets.connections[].database` | string | Yes | — | Database name | +| `externalSecrets.connections[].sslmode` | string | No | `disable` | sslmode query parameter | | `stackgresOperator.name` | string | No | `stackgres-operator` | Helm release name | | `stackgresOperator.namespace` | string | No | shared `namespace` | Namespace override | | `stackgresOperator.values` | object | No | — | Helm values merged with chart defaults | @@ -182,7 +188,7 @@ prewarmDevDB: true | Resource | Kind | Purpose | |----------|------|---------| | `storageclass` | `kubernetes.m.crossplane.io/Object` | StorageClass (default name `psql`; when `storageClass.create: true`) | -| `extsecret-` | `kubernetes.m.crossplane.io/Object` | One per `externalSecrets.secrets[]` entry; wraps an ESO `ExternalSecret` | +| `connsecret-` | `kubernetes.m.crossplane.io/Object` | One per `externalSecrets.connections[]` entry; wraps an ESO `ExternalSecret` that assembles a connection Secret | | `nodepool-psql` | `kubernetes.m.crossplane.io/Object` | Karpenter NodePool (only when `nodePool.enabled: true`) | | `stackgres-operator` | `helm.m.crossplane.io/Release` | StackGres Helm release | | `atlas-operator` | `helm.m.crossplane.io/Release` | Atlas Operator Helm release | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 8ccdcec..5e39d63 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -97,7 +97,7 @@ spec: type: boolean default: true externalSecrets: - description: External Secrets Operator integration. Requires a ClusterSecretStore to already exist on the target cluster (provisioned by SecretStack or equivalent). + description: External Secrets Operator integration. Pulls a password from AWS Secrets Manager (via hops secrets) and assembles it with non-secret connection config into a Kubernetes Secret containing a ready-to-use Postgres connection URL. Requires a ClusterSecretStore on the target cluster (e.g. from SecretStack). type: object properties: enabled: @@ -109,22 +109,40 @@ spec: refreshInterval: description: How often ESO should re-pull the secret. Defaults to 1h. type: string - secrets: - description: List of AWS Secrets Manager paths to sync into Kubernetes Secrets. + connections: + description: "List of Postgres connection secrets to compose. The user publishes just the password via `hops secrets sync aws`; the stack combines it with host/port/database/ns to emit a K8s Secret with url + discrete fields." type: array items: type: object properties: - path: - description: AWS Secrets Manager secret name (what `hops secrets sync aws` writes). - type: string name: - description: Target Kubernetes Secret name. Defaults to the last path segment with "/" replaced by "-". + description: Target Kubernetes Secret name. Also used as the logical connection identifier. type: string namespace: description: Target namespace. Defaults to the stack namespace. type: string - required: [path] + passwordPath: + description: "AWS Secrets Manager secret name containing the password (what `hops secrets sync aws` writes)." + type: string + passwordKey: + description: JSON key inside the AWS Secrets Manager secret holding the password. Defaults to "password". + type: string + host: + description: "Postgres host (e.g. test-pg.stackgres.svc.cluster.local)." + type: string + port: + description: Postgres port. Defaults to 5432. + type: integer + username: + description: Postgres username. Defaults to "postgres". + type: string + database: + description: Postgres database name to connect to. + type: string + sslmode: + description: sslmode query parameter. Defaults to "disable". + type: string + required: [name, passwordPath, host, database] nodePool: description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, StackGres operator and Atlas schedule here via nodeSelector + tolerations. type: object diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index 1374000..f7de490 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -15,9 +15,12 @@ spec: name: psql externalSecrets: enabled: true - secrets: - - path: psql/production - name: psql-production + connections: + - name: psql-production + namespace: stackgres + passwordPath: psql/production + host: test-pg.stackgres.svc.cluster.local + database: postgres stackgresOperator: values: deploy: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 968af9d..6a35365 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -67,21 +67,24 @@ {{- end }} {{- $esoStore := $esoSpec.clusterSecretStoreName | default "hops-aws-secrets-manager" }} {{- $esoRefreshInterval := $esoSpec.refreshInterval | default "1h" }} -{{- $esoInputSecrets := $esoSpec.secrets | default list }} -{{- $esoSecrets := list }} -{{- range $s := $esoInputSecrets }} - {{- $targetName := $s.name }} - {{- if not $targetName }} - {{- /* default K8s Secret name: last path segment, / → - */}} - {{- $parts := splitList "/" $s.path }} - {{- $targetName = index $parts (sub (len $parts) 1) }} - {{- $targetName = replace "_" "-" $targetName }} - {{- end }} - {{- $targetNs := $s.namespace | default $namespace }} - {{- $esoSecrets = append $esoSecrets (dict - "path" $s.path - "name" $targetName +{{- $esoInputConnections := $esoSpec.connections | default list }} +{{- $esoConnections := list }} +{{- range $c := $esoInputConnections }} + {{- $port := $c.port | default 5432 }} + {{- $username := $c.username | default "postgres" }} + {{- $sslmode := $c.sslmode | default "disable" }} + {{- $passwordKey := $c.passwordKey | default "password" }} + {{- $targetNs := $c.namespace | default $namespace }} + {{- $esoConnections = append $esoConnections (dict + "name" $c.name "namespace" $targetNs + "passwordPath" $c.passwordPath + "passwordKey" $passwordKey + "host" $c.host + "port" $port + "username" $username + "database" $c.database + "sslmode" $sslmode ) }} {{- end }} @@ -143,7 +146,7 @@ "enabled" $esoEnabled "clusterSecretStoreName" $esoStore "refreshInterval" $esoRefreshInterval - "secrets" $esoSecrets + "connections" $esoConnections ) "nodePool" (dict "enabled" $nodePoolEnabled diff --git a/functions/render/130-external-secrets.yaml.gotmpl b/functions/render/130-external-secrets.yaml.gotmpl index deac3f6..81b2c01 100644 --- a/functions/render/130-external-secrets.yaml.gotmpl +++ b/functions/render/130-external-secrets.yaml.gotmpl @@ -1,31 +1,36 @@ # code: language=yaml # -# ExternalSecret resources: sync AWS Secrets Manager values into K8s Secrets -# on the target cluster via ESO. +# ExternalSecret resources: assemble Postgres connection Secrets from a +# password stored in AWS Secrets Manager plus non-secret connection config +# (host, port, database, user, namespace) defined on the PSQLStack spec. # -# Each `spec.externalSecrets.secrets[]` entry becomes one ExternalSecret CR -# wrapped in a kubernetes.m.crossplane.io/Object for Crossplane-managed -# reconciliation. The resulting K8s Secret mirrors every JSON key from the -# AWS Secrets Manager secret (dataFrom.extract) — consumers reference -# specific keys via secretKeyRef on the produced Secret. +# For each `spec.externalSecrets.connections[]` entry the stack composes an +# ESO ExternalSecret whose `target.template` renders these keys in the +# produced Kubernetes Secret: +# - url (postgres://:@:/?sslmode=) +# - host, port, username, database, sslmode, password (discrete fields) +# +# Consumers (AtlasSchema.devURLFrom, SGCluster initialData, applications) +# reference whichever key they need via secretKeyRef. # # Prerequisites: # - ESO installed on the target cluster # - A ClusterSecretStore pointing at AWS Secrets Manager (e.g. from SecretStack) -# - `hops secrets sync aws` has published values to the configured paths +# - `hops secrets sync aws` has published the password to the configured path, +# with the key matching `passwordKey` (default "password"). # {{- $eso := $state.externalSecrets }} {{- if $eso.enabled }} -{{- range $idx, $sec := $eso.secrets }} +{{- range $idx, $conn := $eso.connections }} --- apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object metadata: - name: {{ $state.name }}-extsecret-{{ $sec.name }} + name: {{ $state.name }}-connsecret-{{ $conn.name }} annotations: - {{ setResourceNameAnnotation (printf "extsecret-%s" $sec.name) }} + {{ setResourceNameAnnotation (printf "connsecret-%s" $conn.name) }} labels: {{ $state.labels | toJson }} spec: managementPolicies: {{ $state.managementPolicies | toJson }} @@ -34,8 +39,8 @@ spec: apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: - name: {{ $sec.name }} - namespace: {{ $sec.namespace }} + name: {{ $conn.name }} + namespace: {{ $conn.namespace }} labels: {{ $state.labels | toJson }} spec: refreshInterval: {{ $eso.refreshInterval }} @@ -43,11 +48,23 @@ spec: name: {{ $eso.clusterSecretStoreName }} kind: ClusterSecretStore target: - name: {{ $sec.name }} + name: {{ $conn.name }} creationPolicy: Owner - dataFrom: - - extract: - key: {{ $sec.path }} + template: + type: Opaque + data: + url: "postgres://{{ $conn.username }}:{{ `{{ .password }}` }}@{{ $conn.host }}:{{ $conn.port }}/{{ $conn.database }}?sslmode={{ $conn.sslmode }}" + host: {{ $conn.host | quote }} + port: "{{ $conn.port }}" + username: {{ $conn.username | quote }} + database: {{ $conn.database | quote }} + sslmode: {{ $conn.sslmode | quote }} + password: {{ `"{{ .password }}"` }} + data: + - secretKey: password + remoteRef: + key: {{ $conn.passwordPath }} + property: {{ $conn.passwordKey }} providerConfigRef: name: {{ $state.kubernetesProviderConfigRef.name }} kind: {{ $state.kubernetesProviderConfigRef.kind }} From ff1aec4c1918cea71e56d18bb3cc9a3767f81021 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Thu, 23 Apr 2026 12:05:41 -0500 Subject: [PATCH 05/44] refactor: drop externalSecrets from PSQLStack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Without managing the SGCluster, every field in externalSecrets.connections[] (host/port/database/namespace) is passthrough — no abstraction. Users write ExternalSecret CRs directly (or as a Crossplane Object wrapper in local/) against the ClusterSecretStore provisioned by SecretStack. PSQLStack now = platform only: StackGres + Atlas operators + NodePool + StorageClass. If we later add an instances[] that composes SGClusters, ESO wiring can come back for free since the stack will then know host/port/database/namespace without the user restating them. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 14 ---- apis/psqlstacks/definition.yaml | 47 ------------ examples/psqlstacks/standard.yaml | 8 --- functions/render/000-state-init.yaml.gotmpl | 38 ---------- .../render/130-external-secrets.yaml.gotmpl | 72 ------------------- 5 files changed, 179 deletions(-) delete mode 100644 functions/render/130-external-secrets.yaml.gotmpl diff --git a/README.md b/README.md index c48a0ec..c2e0a7e 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,6 @@ PostgreSQL management stack deploying StackGres and Atlas Operator as Helm relea - **StackGres Operator** — Full PostgreSQL lifecycle management with native Citus support for distributed PostgreSQL via `SGShardedCluster` CRDs - **Atlas Operator** — Declarative database schema migrations via `AtlasMigration` and `AtlasSchema` CRDs - **StorageClass** *(on by default, `storageClass.create: true`)* — `psql` class backed by the EKS Auto Mode EBS CSI driver (`ebs.csi.eks.amazonaws.com`). Name mirrors the per-stack convention used by the observe stack (`loki`/`prometheus`/`tempo`). The legacy `gp2` class on EKS Auto Mode uses a deprecated in-tree provisioner that no longer works. -- **Postgres connection Secret(s)** *(opt-in, `externalSecrets.enabled: true`)* — For each entry in `externalSecrets.connections[]`, composes an ESO `ExternalSecret` that pulls a single password from AWS Secrets Manager (published with `hops secrets sync aws`) and templates it with non-secret config (host/port/database/username/sslmode) into a K8s Secret with keys `url`, `host`, `port`, `username`, `database`, `sslmode`, `password`. Consumers reference whichever key they need: `AtlasSchema.devURLFrom` → `url`, `SGCluster.spec.configurations.credentials.users.superuser.password` → `password`, etc. - **Karpenter NodePool** *(opt-in, `nodePool.enabled: true`)* — Dedicated nodes for database workloads. Default: arm64 spot on `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` (memory-optimized Graviton for cheap, low-contention scheduling). StackGres operator + REST API + jobs and Atlas are pinned here via nodeSelector + tolerations. - **Usage resources** — Atlas is deleted before StackGres to prevent orphaned migration state; Helm releases are deleted before the NodePool so pods drain cleanly. @@ -135,18 +134,6 @@ spec: | `storageClass.volumeBindingMode` | enum | No | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | | `storageClass.allowVolumeExpansion` | boolean | No | `true` | Allow PVC resize | | `storageClass.reclaimPolicy` | enum | No | `Delete` | `Delete` or `Retain` | -| `externalSecrets.enabled` | boolean | No | `false` | Enable ESO integration | -| `externalSecrets.clusterSecretStoreName` | string | No | `hops-aws-secrets-manager` | Name of the ClusterSecretStore | -| `externalSecrets.refreshInterval` | string | No | `1h` | ESO refresh interval | -| `externalSecrets.connections[].name` | string | Yes | — | K8s Secret name (also the connection identifier) | -| `externalSecrets.connections[].namespace` | string | No | stack `namespace` | K8s Secret namespace | -| `externalSecrets.connections[].passwordPath` | string | Yes | — | AWS Secrets Manager secret name containing the password | -| `externalSecrets.connections[].passwordKey` | string | No | `password` | JSON key in the AWS secret holding the password | -| `externalSecrets.connections[].host` | string | Yes | — | Postgres host (e.g. `test-pg.stackgres.svc.cluster.local`) | -| `externalSecrets.connections[].port` | integer | No | `5432` | Postgres port | -| `externalSecrets.connections[].username` | string | No | `postgres` | Postgres username | -| `externalSecrets.connections[].database` | string | Yes | — | Database name | -| `externalSecrets.connections[].sslmode` | string | No | `disable` | sslmode query parameter | | `stackgresOperator.name` | string | No | `stackgres-operator` | Helm release name | | `stackgresOperator.namespace` | string | No | shared `namespace` | Namespace override | | `stackgresOperator.values` | object | No | — | Helm values merged with chart defaults | @@ -188,7 +175,6 @@ prewarmDevDB: true | Resource | Kind | Purpose | |----------|------|---------| | `storageclass` | `kubernetes.m.crossplane.io/Object` | StorageClass (default name `psql`; when `storageClass.create: true`) | -| `connsecret-` | `kubernetes.m.crossplane.io/Object` | One per `externalSecrets.connections[]` entry; wraps an ESO `ExternalSecret` that assembles a connection Secret | | `nodepool-psql` | `kubernetes.m.crossplane.io/Object` | Karpenter NodePool (only when `nodePool.enabled: true`) | | `stackgres-operator` | `helm.m.crossplane.io/Release` | StackGres Helm release | | `atlas-operator` | `helm.m.crossplane.io/Release` | Atlas Operator Helm release | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 5e39d63..e7645f7 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -96,53 +96,6 @@ spec: allowVolumeExpansion: type: boolean default: true - externalSecrets: - description: External Secrets Operator integration. Pulls a password from AWS Secrets Manager (via hops secrets) and assembles it with non-secret connection config into a Kubernetes Secret containing a ready-to-use Postgres connection URL. Requires a ClusterSecretStore on the target cluster (e.g. from SecretStack). - type: object - properties: - enabled: - type: boolean - default: false - clusterSecretStoreName: - description: Name of the ClusterSecretStore to read from. Defaults to "hops-aws-secrets-manager". - type: string - refreshInterval: - description: How often ESO should re-pull the secret. Defaults to 1h. - type: string - connections: - description: "List of Postgres connection secrets to compose. The user publishes just the password via `hops secrets sync aws`; the stack combines it with host/port/database/ns to emit a K8s Secret with url + discrete fields." - type: array - items: - type: object - properties: - name: - description: Target Kubernetes Secret name. Also used as the logical connection identifier. - type: string - namespace: - description: Target namespace. Defaults to the stack namespace. - type: string - passwordPath: - description: "AWS Secrets Manager secret name containing the password (what `hops secrets sync aws` writes)." - type: string - passwordKey: - description: JSON key inside the AWS Secrets Manager secret holding the password. Defaults to "password". - type: string - host: - description: "Postgres host (e.g. test-pg.stackgres.svc.cluster.local)." - type: string - port: - description: Postgres port. Defaults to 5432. - type: integer - username: - description: Postgres username. Defaults to "postgres". - type: string - database: - description: Postgres database name to connect to. - type: string - sslmode: - description: sslmode query parameter. Defaults to "disable". - type: string - required: [name, passwordPath, host, database] nodePool: description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, StackGres operator and Atlas schedule here via nodeSelector + tolerations. type: object diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index f7de490..b33bf8e 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -13,14 +13,6 @@ spec: storageClass: create: true name: psql - externalSecrets: - enabled: true - connections: - - name: psql-production - namespace: stackgres - passwordPath: psql/production - host: test-pg.stackgres.svc.cluster.local - database: postgres stackgresOperator: values: deploy: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 6a35365..0bbe13f 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -56,38 +56,6 @@ {{- $scAllowVolumeExpansion = $sc.allowVolumeExpansion }} {{- end }} -# ============================================================================== -# ExternalSecrets (opt-in) — sync AWS Secrets Manager values into K8s Secrets -# via ESO. Requires a ClusterSecretStore to already exist on the target cluster. -# ============================================================================== -{{- $esoSpec := $spec.externalSecrets | default dict }} -{{- $esoEnabled := false }} -{{- if hasKey $esoSpec "enabled" }} - {{- $esoEnabled = $esoSpec.enabled }} -{{- end }} -{{- $esoStore := $esoSpec.clusterSecretStoreName | default "hops-aws-secrets-manager" }} -{{- $esoRefreshInterval := $esoSpec.refreshInterval | default "1h" }} -{{- $esoInputConnections := $esoSpec.connections | default list }} -{{- $esoConnections := list }} -{{- range $c := $esoInputConnections }} - {{- $port := $c.port | default 5432 }} - {{- $username := $c.username | default "postgres" }} - {{- $sslmode := $c.sslmode | default "disable" }} - {{- $passwordKey := $c.passwordKey | default "password" }} - {{- $targetNs := $c.namespace | default $namespace }} - {{- $esoConnections = append $esoConnections (dict - "name" $c.name - "namespace" $targetNs - "passwordPath" $c.passwordPath - "passwordKey" $passwordKey - "host" $c.host - "port" $port - "username" $username - "database" $c.database - "sslmode" $sslmode - ) }} -{{- end }} - # ============================================================================== # NodePool (opt-in) — dedicates a pool of nodes for PostgreSQL workloads. # Defaults: arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). @@ -142,12 +110,6 @@ "volumeBindingMode" $scVolumeBindingMode "allowVolumeExpansion" $scAllowVolumeExpansion ) - "externalSecrets" (dict - "enabled" $esoEnabled - "clusterSecretStoreName" $esoStore - "refreshInterval" $esoRefreshInterval - "connections" $esoConnections - ) "nodePool" (dict "enabled" $nodePoolEnabled "name" $nodePoolName diff --git a/functions/render/130-external-secrets.yaml.gotmpl b/functions/render/130-external-secrets.yaml.gotmpl deleted file mode 100644 index 81b2c01..0000000 --- a/functions/render/130-external-secrets.yaml.gotmpl +++ /dev/null @@ -1,72 +0,0 @@ -# code: language=yaml -# -# ExternalSecret resources: assemble Postgres connection Secrets from a -# password stored in AWS Secrets Manager plus non-secret connection config -# (host, port, database, user, namespace) defined on the PSQLStack spec. -# -# For each `spec.externalSecrets.connections[]` entry the stack composes an -# ESO ExternalSecret whose `target.template` renders these keys in the -# produced Kubernetes Secret: -# - url (postgres://:@:/?sslmode=) -# - host, port, username, database, sslmode, password (discrete fields) -# -# Consumers (AtlasSchema.devURLFrom, SGCluster initialData, applications) -# reference whichever key they need via secretKeyRef. -# -# Prerequisites: -# - ESO installed on the target cluster -# - A ClusterSecretStore pointing at AWS Secrets Manager (e.g. from SecretStack) -# - `hops secrets sync aws` has published the password to the configured path, -# with the key matching `passwordKey` (default "password"). -# - -{{- $eso := $state.externalSecrets }} - -{{- if $eso.enabled }} -{{- range $idx, $conn := $eso.connections }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-connsecret-{{ $conn.name }} - annotations: - {{ setResourceNameAnnotation (printf "connsecret-%s" $conn.name) }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: external-secrets.io/v1 - kind: ExternalSecret - metadata: - name: {{ $conn.name }} - namespace: {{ $conn.namespace }} - labels: {{ $state.labels | toJson }} - spec: - refreshInterval: {{ $eso.refreshInterval }} - secretStoreRef: - name: {{ $eso.clusterSecretStoreName }} - kind: ClusterSecretStore - target: - name: {{ $conn.name }} - creationPolicy: Owner - template: - type: Opaque - data: - url: "postgres://{{ $conn.username }}:{{ `{{ .password }}` }}@{{ $conn.host }}:{{ $conn.port }}/{{ $conn.database }}?sslmode={{ $conn.sslmode }}" - host: {{ $conn.host | quote }} - port: "{{ $conn.port }}" - username: {{ $conn.username | quote }} - database: {{ $conn.database | quote }} - sslmode: {{ $conn.sslmode | quote }} - password: {{ `"{{ .password }}"` }} - data: - - secretKey: password - remoteRef: - key: {{ $conn.passwordPath }} - property: {{ $conn.passwordKey }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} -{{- end }} From 73585668c1b57b9e34f63022d217b18071528eb3 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Thu, 23 Apr 2026 20:10:10 -0500 Subject: [PATCH 06/44] feat: swap StackGres operator for CloudNativePG (phase 1 of CNPG pivot) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites PSQLStack schema to remove stackgresOperator, add cnpg and scaleToZeroPlugin blocks. Default namespace: cnpg-system. CNPG 1.29 (chart 0.27.1) replaces StackGres 1.18 as the operator. Atlas operator renumbered 210 → 220 to make room for the scale-to-zero plugin install (added in a later phase). Storage (psql StorageClass on EBS gp3) and NodePool blocks preserved unchanged — phases 2 and 3 will rewrite them for the three-profile (mayastor / lvm / ebs) storage model and the branches/primary NodePool split with hugepages + nvme-tcp pre-configured. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- apis/psqlstacks/definition.yaml | 41 ++++++--- functions/render/000-state-init.yaml.gotmpl | 34 ++++--- functions/render/010-state-status.yaml.gotmpl | 5 +- .../render/200-cnpg-operator.yaml.gotmpl | 77 ++++++++++++++++ .../render/200-stackgres-operator.yaml.gotmpl | 88 ------------------- ....gotmpl => 220-atlas-operator.yaml.gotmpl} | 0 upbound.yaml | 19 ++-- 7 files changed, 144 insertions(+), 120 deletions(-) create mode 100644 functions/render/200-cnpg-operator.yaml.gotmpl delete mode 100644 functions/render/200-stackgres-operator.yaml.gotmpl rename functions/render/{210-atlas-operator.yaml.gotmpl => 220-atlas-operator.yaml.gotmpl} (100%) diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index e7645f7..099c6ce 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -14,7 +14,7 @@ spec: served: true schema: openAPIV3Schema: - description: PSQLStack deploys the StackGres operator (with Citus support) and Atlas Operator for PostgreSQL database management and schema migrations. + description: PSQLStack deploys CloudNativePG, the cnpg-i-scale-to-zero plugin, and the Atlas Operator. It's the platform layer — per-app serving clusters live in PSQLCluster; ephemeral forks live in PSQLBranch. type: object properties: spec: @@ -51,7 +51,7 @@ spec: - ProviderConfig - ClusterProviderConfig kubernetesProviderConfigRef: - description: Reference to the Kubernetes ProviderConfig (used to create the Karpenter NodePool Object). Defaults to clusterName. + description: Reference to the Kubernetes ProviderConfig (used to create the Karpenter NodePool Object and to apply the scale-to-zero plugin manifest). Defaults to clusterName. type: object properties: name: @@ -64,17 +64,17 @@ spec: - ProviderConfig - ClusterProviderConfig namespace: - description: Shared namespace for all components. Defaults to stackgres. Per-component namespace overrides this. + description: Shared namespace for CNPG operator, scale-to-zero plugin, and Atlas operator. Defaults to cnpg-system. Per-component namespace overrides this. type: string storageClass: - description: "StorageClass for PostgreSQL workloads. Default provisioner is the EKS Auto Mode EBS CSI driver. Disable with create=false if the cluster already provides a suitable StorageClass." + description: "StorageClass for PostgreSQL workloads. Default provisioner is the EKS Auto Mode EBS CSI driver. Disable with create=false if the cluster already provides a suitable StorageClass. (Phase 2 of the CNPG pivot will replace this with a three-profile storage block.)" type: object properties: create: type: boolean default: true name: - description: StorageClass name. Defaults to "psql" (mirrors per-stack naming used by the observe stack — loki/prometheus/tempo). + description: StorageClass name. Defaults to "psql". type: string provisioner: description: CSI provisioner. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). @@ -97,7 +97,7 @@ spec: type: boolean default: true nodePool: - description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, StackGres operator and Atlas schedule here via nodeSelector + tolerations. + description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, CNPG, the scale-to-zero plugin, and Atlas schedule here via nodeSelector + tolerations. (Phase 3 of the CNPG pivot will split this into branches / primary sub-pools with hugepages + nvme-tcp pre-configured.) type: object properties: enabled: @@ -114,7 +114,7 @@ spec: memory: type: string requirements: - description: Karpenter scheduling requirements. Defaults to arm64 spot on r7g/m7g (memory-optimized, general-purpose). + description: Karpenter scheduling requirements. Defaults to arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). type: array items: type: object @@ -139,16 +139,19 @@ spec: consolidateAfter: type: string default: "60s" - stackgresOperator: - description: Configuration for the StackGres operator component. + cnpg: + description: Configuration for the CloudNativePG operator Helm release. type: object properties: name: - description: Release name. Defaults to stackgres-operator. + description: Helm release name. Defaults to cloudnative-pg. type: string namespace: description: Namespace override for this component. type: string + chartVersion: + description: CNPG Helm chart version. Defaults to 0.27.1 (tracks CNPG 1.29.x). + type: string values: description: Helm values merged with chart defaults. type: object @@ -157,8 +160,22 @@ spec: description: Helm values that replace all defaults for this component. type: object x-kubernetes-preserve-unknown-fields: true + scaleToZeroPlugin: + description: "Configuration for the cnpg-i-scale-to-zero plugin (github.com/xataio/cnpg-i-scale-to-zero). Installed via upstream release manifest. Zero cost when no PSQLCluster opts into scale-to-zero." + type: object + properties: + enabled: + description: Default true — the plugin is zero-cost when unused and per-cluster opt-in is cheap. + type: boolean + default: true + version: + description: Plugin release tag. The manifest URL pins to this. Defaults to v0.1.7. + type: string + namespace: + description: Namespace override for the plugin install. Defaults to cnpg-system (plugin manifest creates the namespace if missing). + type: string atlasOperator: - description: Configuration for the Atlas Operator component (database schema migrations). + description: Configuration for the Atlas Operator component (declarative database schema migrations). type: object properties: name: @@ -182,7 +199,7 @@ spec: type: object properties: ready: - description: Overall readiness of the PostgreSQL stack. + description: Overall readiness of the stack (CNPG + scale-to-zero plugin + Atlas). type: boolean required: - spec diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 0bbe13f..c92c149 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -12,7 +12,7 @@ # ============================================================================== {{- $name := $metadata.name | default "psql" }} {{- $clusterName := $spec.clusterName | default $name }} -{{- $namespace := $spec.namespace | default "stackgres" }} +{{- $namespace := $spec.namespace | default "cnpg-system" }} {{- $managementPolicies := $spec.managementPolicies | default (list "*") }} # Labels @@ -37,9 +37,10 @@ }} # ============================================================================== -# StorageClass (opt-in) — creates a gp3 StorageClass on the target cluster. -# EKS Auto Mode uses ebs.csi.eks.amazonaws.com; the in-tree kubernetes.io/aws-ebs -# provisioner (used by the legacy "gp2" class) does not work there. +# StorageClass (opt-in, default on) — creates a gp3 StorageClass on the target +# cluster. EKS Auto Mode uses ebs.csi.eks.amazonaws.com. +# Phase 2 of the CNPG pivot replaces this block with a three-profile model +# (mayastor / lvm / ebs). # ============================================================================== {{- $sc := $spec.storageClass | default dict }} {{- $scCreate := true }} @@ -59,6 +60,8 @@ # ============================================================================== # NodePool (opt-in) — dedicates a pool of nodes for PostgreSQL workloads. # Defaults: arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). +# Phase 3 of the CNPG pivot splits this into branches / primary sub-pools with +# hugepages + nvme-tcp pre-configured for Mayastor. # ============================================================================== {{- $nodePoolSpec := $spec.nodePool | default dict }} {{- $nodePoolEnabled := false }} @@ -87,7 +90,12 @@ # ============================================================================== # Per-component configuration # ============================================================================== -{{- $sg := $spec.stackgresOperator | default dict }} +{{- $cnpg := $spec.cnpg | default dict }} +{{- $s2z := $spec.scaleToZeroPlugin | default dict }} +{{- $s2zEnabled := true }} +{{- if hasKey $s2z "enabled" }} + {{- $s2zEnabled = $s2z.enabled }} +{{- end }} {{- $atlas := $spec.atlasOperator | default dict }} # ============================================================================== @@ -122,11 +130,17 @@ "nodeSelector" $nodePoolNodeSelector "tolerations" $nodePoolTolerations ) - "stackgresOperator" (dict - "name" ($sg.name | default "stackgres-operator") - "namespace" ($sg.namespace | default $namespace) - "values" ($sg.values | default dict) - "overrideAllValues" ($sg.overrideAllValues | default dict) + "cnpg" (dict + "name" ($cnpg.name | default "cloudnative-pg") + "namespace" ($cnpg.namespace | default $namespace) + "chartVersion" ($cnpg.chartVersion | default "0.27.1") + "values" ($cnpg.values | default dict) + "overrideAllValues" ($cnpg.overrideAllValues | default dict) + ) + "scaleToZeroPlugin" (dict + "enabled" $s2zEnabled + "version" ($s2z.version | default "v0.1.7") + "namespace" ($s2z.namespace | default $namespace) ) "atlasOperator" (dict "name" ($atlas.name | default "atlas-operator") diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index d067313..3a1abaa 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "stackgres-operator" "atlas-operator" "nodepool-psql" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-psql" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -27,7 +27,8 @@ # Set observed state # ============================================================================== {{- $state = set $state "observed" (dict - "stackgresOperator" (dict "ready" (get $checkReady "stackgres-operator")) + "cnpg" (dict "ready" (get $checkReady "cnpg-operator")) + "scaleToZeroPlugin" (dict "ready" (get $checkReady "cnpg-scale-to-zero")) "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) "nodepoolPsql" (dict "ready" (get $checkReady "nodepool-psql")) ) }} diff --git a/functions/render/200-cnpg-operator.yaml.gotmpl b/functions/render/200-cnpg-operator.yaml.gotmpl new file mode 100644 index 0000000..98e8f7c --- /dev/null +++ b/functions/render/200-cnpg-operator.yaml.gotmpl @@ -0,0 +1,77 @@ +# code: language=yaml +# +# Helm Release: cloudnative-pg operator +# +# CloudNativePG is a CNCF PostgreSQL operator for Kubernetes covering the +# full lifecycle of PostgreSQL clusters (HA, streaming replication, +# VolumeSnapshot backups, Barman cloud, pg_basebackup clones, CNPG-I +# plugin protocol for extensions like scale-to-zero). +# +# Chart: https://cloudnative-pg.github.io/charts (chart name: cloudnative-pg) +# Upstream: https://github.com/cloudnative-pg/cloudnative-pg +# + +{{- $cnpg := $state.cnpg }} + +--- +apiVersion: helm.m.crossplane.io/v1beta1 +kind: Release +metadata: + name: {{ $cnpg.name }} + annotations: + {{ setResourceNameAnnotation "cnpg-operator" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + chart: + name: cloudnative-pg + repository: https://cloudnative-pg.github.io/charts + version: {{ $cnpg.chartVersion | quote }} + namespace: {{ $cnpg.namespace }} + {{- if $cnpg.overrideAllValues }} + values: + {{- toYaml $cnpg.overrideAllValues | nindent 6 }} + {{- else }} + {{- /* Chart defaults */}} + {{- $chartDefaults := dict }} + {{- if $state.nodePool.enabled }} + {{- $_ := set $chartDefaults "nodeSelector" $state.nodePool.nodeSelector }} + {{- $_ := set $chartDefaults "tolerations" $state.nodePool.tolerations }} + {{- end }} + {{- $mergedValues := mergeOverwrite $chartDefaults ($cnpg.values | default dict) }} + values: + {{- toYaml $mergedValues | nindent 6 }} + {{- end }} + rollbackLimit: 3 + providerConfigRef: + name: {{ $state.helmProviderConfigRef.name }} + kind: {{ $state.helmProviderConfigRef.kind }} + +# ============================================================================== +# Usage: Protect cnpg-operator from deletion until atlas-operator is deleted. +# Atlas manages schema migrations against CNPG-managed databases; deleting CNPG +# first would orphan Atlas's migration state. +# ============================================================================== +{{- if and $state.observed.cnpg.ready $state.observed.atlasOperator.ready }} +--- +apiVersion: protection.crossplane.io/v1beta1 +kind: Usage +metadata: + name: {{ $state.name }}-delete-atlas-operator-before-cnpg + annotations: + {{ setResourceNameAnnotation "usage-cnpg-atlas" }} + labels: {{ $state.labels | toJson }} +spec: + replayDeletion: true + of: + apiVersion: helm.m.crossplane.io/v1beta1 + kind: Release + resourceRef: + name: {{ $cnpg.name }} + by: + apiVersion: helm.m.crossplane.io/v1beta1 + kind: Release + resourceRef: + name: {{ $state.atlasOperator.name }} +{{- end }} diff --git a/functions/render/200-stackgres-operator.yaml.gotmpl b/functions/render/200-stackgres-operator.yaml.gotmpl deleted file mode 100644 index 47c1a02..0000000 --- a/functions/render/200-stackgres-operator.yaml.gotmpl +++ /dev/null @@ -1,88 +0,0 @@ -# code: language=yaml -# -# Helm Release: stackgres-operator -# -# StackGres provides a full PostgreSQL platform on Kubernetes with native -# Citus support via SGShardedCluster CRDs. -# - -{{- $sg := $state.stackgresOperator }} - ---- -apiVersion: helm.m.crossplane.io/v1beta1 -kind: Release -metadata: - name: {{ $sg.name }} - annotations: - {{ setResourceNameAnnotation "stackgres-operator" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - chart: - name: stackgres-operator - repository: https://stackgres.io/downloads/stackgres-k8s/stackgres/helm/ - url: https://stackgres.io/downloads/stackgres-k8s/stackgres/1.18.4/helm/stackgres-operator.tgz - version: 1.18.4 - namespace: {{ $sg.namespace }} - {{- if $sg.overrideAllValues }} - values: - {{- toYaml $sg.overrideAllValues | nindent 6 }} - {{- else }} - {{- /* Chart defaults */}} - {{- $operatorDefaults := dict }} - {{- $restapiDefaults := dict }} - {{- $jobsDefaults := dict }} - {{- if $state.nodePool.enabled }} - {{- $_ := set $operatorDefaults "nodeSelector" $state.nodePool.nodeSelector }} - {{- $_ := set $operatorDefaults "tolerations" $state.nodePool.tolerations }} - {{- $_ := set $restapiDefaults "nodeSelector" $state.nodePool.nodeSelector }} - {{- $_ := set $restapiDefaults "tolerations" $state.nodePool.tolerations }} - {{- $_ := set $jobsDefaults "nodeSelector" $state.nodePool.nodeSelector }} - {{- $_ := set $jobsDefaults "tolerations" $state.nodePool.tolerations }} - {{- end }} - {{- $chartDefaults := dict - "deploy" (dict - "operator" true - "restapi" true - ) - "operator" $operatorDefaults - "restapi" $restapiDefaults - "jobs" $jobsDefaults - }} - {{- $mergedValues := mergeOverwrite $chartDefaults ($sg.values | default dict) }} - values: - {{- toYaml $mergedValues | nindent 6 }} - {{- end }} - rollbackLimit: 3 - providerConfigRef: - name: {{ $state.helmProviderConfigRef.name }} - kind: {{ $state.helmProviderConfigRef.kind }} - -# ============================================================================== -# Usage: Protect stackgres-operator from deletion until atlas-operator is deleted -# ============================================================================== -# Atlas manages schema migrations against StackGres-managed databases. -# Delete Atlas operator first to avoid orphaned migration state. -{{- if and $state.observed.stackgresOperator.ready $state.observed.atlasOperator.ready }} ---- -apiVersion: protection.crossplane.io/v1beta1 -kind: Usage -metadata: - name: {{ $state.name }}-delete-atlas-operator-before-stackgres-operator - annotations: - {{ setResourceNameAnnotation "usage-sg-atlas" }} - labels: {{ $state.labels | toJson }} -spec: - replayDeletion: true - of: - apiVersion: helm.m.crossplane.io/v1beta1 - kind: Release - resourceRef: - name: {{ $sg.name }} - by: - apiVersion: helm.m.crossplane.io/v1beta1 - kind: Release - resourceRef: - name: {{ $state.atlasOperator.name }} -{{- end }} diff --git a/functions/render/210-atlas-operator.yaml.gotmpl b/functions/render/220-atlas-operator.yaml.gotmpl similarity index 100% rename from functions/render/210-atlas-operator.yaml.gotmpl rename to functions/render/220-atlas-operator.yaml.gotmpl diff --git a/upbound.yaml b/upbound.yaml index 3f57864..9ff08d9 100644 --- a/upbound.yaml +++ b/upbound.yaml @@ -16,19 +16,22 @@ spec: kind: Provider package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes version: '>=v1' - description: PostgreSQL stack deploying StackGres operator (with Citus support) - and Atlas Operator for schema migrations as Helm releases, with an optional dedicated - Karpenter NodePool for database workloads. + description: PostgreSQL stack deploying CloudNativePG, the cnpg-i-scale-to-zero + plugin, and the Atlas Operator as Helm releases, with an optional dedicated Karpenter + NodePool. Platform layer — per-app serving clusters live in PSQLCluster. license: Apache-2.0 maintainer: Patrick Lee Scott readme: | # psql-stack - Deploys a PostgreSQL management stack with correct deletion ordering. - Composes StackGres operator and Atlas Operator as direct Helm releases. + Deploys a PostgreSQL platform stack with correct deletion ordering. + Composes CloudNativePG, the cnpg-i-scale-to-zero plugin, and the Atlas + Operator as direct Helm releases / applied manifests. - StackGres provides full PostgreSQL lifecycle management with native Citus - support via SGShardedCluster CRDs. Atlas Operator provides declarative - database schema migrations via AtlasMigration and AtlasSchema CRDs. + CloudNativePG is a CNCF operator that manages the full lifecycle of + PostgreSQL clusters on Kubernetes. The cnpg-i-scale-to-zero plugin + (github.com/xataio/cnpg-i-scale-to-zero) automatically hibernates + inactive clusters. Atlas Operator provides declarative database schema + migrations via AtlasMigration and AtlasSchema CRDs. repository: ghcr.io/hops-ops/psql-stack source: github.com/hops-ops/psql-stack From 8debdb63b6a0762c71344de3a9a15b028371078c Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 24 Apr 2026 19:15:26 -0500 Subject: [PATCH 07/44] =?UTF-8?q?feat:=20three-profile=20storage=20layer?= =?UTF-8?q?=20=E2=80=94=20mayastor=20+=20lvm=20+=20ebs=20(phase=202)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the single storageClass block with a storage block exposing three independent profiles: - storage.mayastor (replicated NVMe-oF via OpenEBS Mayastor) — enterprise default for primary serving clusters; CoW + HA across N replicas. Default enabled=false until phase 3 lands the NodePool with hugepages + nvme-tcp. - storage.lvm (single-node CoW via OpenEBS LVM LocalPV) — branches and dev clusters. Default enabled=false until phase 3 lands NodePool LVM volume groups. - storage.ebs (EBS gp3 via EKS Auto Mode CSI) — durable fallback, no CoW; always-on default. Render templates: - 120-storageclass.yaml.gotmpl deleted (was the single 'psql' SC) - 160-openebs-lvm.yaml.gotmpl: Helm release for OpenEBS LVM LocalPV - 165-openebs-mayastor.yaml.gotmpl: Helm release for OpenEBS Mayastor - 170-storageclass-mayastor.yaml.gotmpl: psql-mayastor SC + VolumeSnapshotClass - 175-storageclass-lvm.yaml.gotmpl: psql-lvm SC + VolumeSnapshotClass - 180-storageclass-ebs.yaml.gotmpl: psql-ebs SC (renamed from 'psql') state-init defaults the three profile blocks; state-status observes the new resource keys. Mayastor + LVM Helm releases + their StorageClasses are gated on storage.{mayastor,lvm}.enabled — only EBS materializes by default until phase 3 unblocks the others. standard.yaml example patched to drop the removed storageClass and stackgresOperator fields (full example rewrite is phase 5). Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- apis/psqlstacks/definition.yaml | 135 ++++++++++++++---- examples/psqlstacks/standard.yaml | 15 +- functions/render/000-state-init.yaml.gotmpl | 87 +++++++---- functions/render/010-state-status.yaml.gotmpl | 7 +- functions/render/120-storageclass.yaml.gotmpl | 34 ----- functions/render/160-openebs-lvm.yaml.gotmpl | 56 ++++++++ .../render/165-openebs-mayastor.yaml.gotmpl | 55 +++++++ .../170-storageclass-mayastor.yaml.gotmpl | 64 +++++++++ .../render/175-storageclass-lvm.yaml.gotmpl | 65 +++++++++ .../render/180-storageclass-ebs.yaml.gotmpl | 39 +++++ 10 files changed, 463 insertions(+), 94 deletions(-) delete mode 100644 functions/render/120-storageclass.yaml.gotmpl create mode 100644 functions/render/160-openebs-lvm.yaml.gotmpl create mode 100644 functions/render/165-openebs-mayastor.yaml.gotmpl create mode 100644 functions/render/170-storageclass-mayastor.yaml.gotmpl create mode 100644 functions/render/175-storageclass-lvm.yaml.gotmpl create mode 100644 functions/render/180-storageclass-ebs.yaml.gotmpl diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 099c6ce..f46c9dc 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -66,36 +66,117 @@ spec: namespace: description: Shared namespace for CNPG operator, scale-to-zero plugin, and Atlas operator. Defaults to cnpg-system. Per-component namespace overrides this. type: string - storageClass: - description: "StorageClass for PostgreSQL workloads. Default provisioner is the EKS Auto Mode EBS CSI driver. Disable with create=false if the cluster already provides a suitable StorageClass. (Phase 2 of the CNPG pivot will replace this with a three-profile storage block.)" + storage: + description: "Three-profile storage layer. mayastor = replicated NVMe-oF (CoW + HA), lvm = single-node LVM thin clones (CoW, cheap), ebs = EBS gp3 (durable, no CoW). Each profile is independently toggleable. Mayastor + LVM require node-side prerequisites (hugepages, nvme-tcp module, NVMe instance-store devices, LVM volume groups) — these are provided by the NodePool config in phase 3 of the pivot. Until phase 3 lands, leave mayastor.enabled and lvm.enabled at false on clusters that don't already have those prereqs." type: object properties: - create: - type: boolean - default: true - name: - description: StorageClass name. Defaults to "psql". - type: string - provisioner: - description: CSI provisioner. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). - type: string - parameters: - description: Provisioner parameters. Defaults to gp3 + ext4. + mayastor: + description: OpenEBS Mayastor (Replicated NVMe-oF) — enterprise default for primary serving clusters. Provides CoW snapshots + HA replication across NVMe nodes. type: object - additionalProperties: - type: string - x-kubernetes-preserve-unknown-fields: true - reclaimPolicy: - type: string - enum: [Delete, Retain] - default: Delete - volumeBindingMode: - type: string - enum: [Immediate, WaitForFirstConsumer] - default: WaitForFirstConsumer - allowVolumeExpansion: - type: boolean - default: true + properties: + enabled: + type: boolean + default: false + chartVersion: + description: Mayastor Helm chart version. Defaults to 2.10.0. + type: string + namespace: + description: Namespace override for the Mayastor install. Defaults to mayastor. + type: string + storageClassName: + description: StorageClass name. Defaults to psql-mayastor. + type: string + replicationFactor: + description: Number of replicas per volume. Defaults to 3 (quorum-safe HA). + type: integer + default: 3 + thin: + description: Thin-provision volumes (CoW). Defaults to true. + type: boolean + default: true + reclaimPolicy: + type: string + enum: [Delete, Retain] + default: Delete + volumeBindingMode: + type: string + enum: [Immediate, WaitForFirstConsumer] + default: WaitForFirstConsumer + allowVolumeExpansion: + type: boolean + default: true + values: + type: object + x-kubernetes-preserve-unknown-fields: true + overrideAllValues: + type: object + x-kubernetes-preserve-unknown-fields: true + lvm: + description: OpenEBS LVM LocalPV — single-node CoW via LVM thin volumes. Cheaper than Mayastor; appropriate for branches and dev clusters. + type: object + properties: + enabled: + type: boolean + default: false + chartVersion: + description: lvm-localpv Helm chart version. Defaults to 1.7.0. + type: string + namespace: + description: Namespace override. Defaults to openebs. + type: string + storageClassName: + description: StorageClass name. Defaults to psql-lvm. + type: string + volumeGroup: + description: LVM Volume Group name configured on each NVMe node. Defaults to psql-vg. + type: string + reclaimPolicy: + type: string + enum: [Delete, Retain] + default: Delete + volumeBindingMode: + type: string + enum: [Immediate, WaitForFirstConsumer] + default: WaitForFirstConsumer + allowVolumeExpansion: + type: boolean + default: true + values: + type: object + x-kubernetes-preserve-unknown-fields: true + overrideAllValues: + type: object + x-kubernetes-preserve-unknown-fields: true + ebs: + description: EBS gp3 via the EKS Auto Mode CSI driver. Durable but no CoW; pg_basebackup-only branching. Always-on fallback profile. + type: object + properties: + enabled: + type: boolean + default: true + storageClassName: + description: StorageClass name. Defaults to psql-ebs. + type: string + provisioner: + description: CSI provisioner. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). + type: string + parameters: + description: Provisioner parameters. Defaults to gp3 + ext4. + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true + reclaimPolicy: + type: string + enum: [Delete, Retain] + default: Delete + volumeBindingMode: + type: string + enum: [Immediate, WaitForFirstConsumer] + default: WaitForFirstConsumer + allowVolumeExpansion: + type: boolean + default: true nodePool: description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, CNPG, the scale-to-zero plugin, and Atlas schedule here via nodeSelector + tolerations. (Phase 3 of the CNPG pivot will split this into branches / primary sub-pools with hugepages + nvme-tcp pre-configured.) type: object diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index b33bf8e..c443fdb 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -5,18 +5,17 @@ metadata: namespace: default spec: clusterName: production-cluster - namespace: stackgres + namespace: cnpg-system labels: team: platform nodePool: enabled: true - storageClass: - create: true - name: psql - stackgresOperator: - values: - deploy: - restapi: true + storage: + ebs: + enabled: true + storageClassName: psql-ebs + cnpg: + values: {} atlasOperator: values: prewarmDevDB: true diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index c92c149..6ef851b 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -37,25 +37,68 @@ }} # ============================================================================== -# StorageClass (opt-in, default on) — creates a gp3 StorageClass on the target -# cluster. EKS Auto Mode uses ebs.csi.eks.amazonaws.com. -# Phase 2 of the CNPG pivot replaces this block with a three-profile model -# (mayastor / lvm / ebs). +# Storage — three-profile model (mayastor / lvm / ebs) +# Each profile is independently toggleable. Mayastor and LVM require node-side +# prerequisites (hugepages, nvme-tcp module, NVMe instance-store devices, +# LVM volume groups) provided by phase 3's NodePool config — keep them +# disabled until those prereqs land. EBS is always-safe as a fallback. # ============================================================================== -{{- $sc := $spec.storageClass | default dict }} -{{- $scCreate := true }} -{{- if hasKey $sc "create" }} - {{- $scCreate = $sc.create }} +{{- $storage := $spec.storage | default dict }} + +# Mayastor — replicated NVMe-oF (HA + CoW) +{{- $mayastorSpec := $storage.mayastor | default dict }} +{{- $mayastorEnabled := false }} +{{- if hasKey $mayastorSpec "enabled" }} + {{- $mayastorEnabled = $mayastorSpec.enabled }} {{- end }} -{{- $scName := $sc.name | default "psql" }} -{{- $scProvisioner := $sc.provisioner | default "ebs.csi.eks.amazonaws.com" }} -{{- $scParameters := $sc.parameters | default (dict "type" "gp3" "fsType" "ext4") }} -{{- $scReclaimPolicy := $sc.reclaimPolicy | default "Delete" }} -{{- $scVolumeBindingMode := $sc.volumeBindingMode | default "WaitForFirstConsumer" }} -{{- $scAllowVolumeExpansion := true }} -{{- if hasKey $sc "allowVolumeExpansion" }} - {{- $scAllowVolumeExpansion = $sc.allowVolumeExpansion }} +{{- $mayastor := dict + "enabled" $mayastorEnabled + "chartVersion" ($mayastorSpec.chartVersion | default "2.10.0") + "namespace" ($mayastorSpec.namespace | default "mayastor") + "storageClassName" ($mayastorSpec.storageClassName | default "psql-mayastor") + "replicationFactor" ($mayastorSpec.replicationFactor | default 3) + "thin" (or (eq (toString ($mayastorSpec.thin | default true)) "true") false) + "reclaimPolicy" ($mayastorSpec.reclaimPolicy | default "Delete") + "volumeBindingMode" ($mayastorSpec.volumeBindingMode | default "WaitForFirstConsumer") + "allowVolumeExpansion" (or (eq (toString ($mayastorSpec.allowVolumeExpansion | default true)) "true") false) + "values" ($mayastorSpec.values | default dict) + "overrideAllValues" ($mayastorSpec.overrideAllValues | default dict) +}} + +# LVM LocalPV — single-node CoW via LVM thin clones +{{- $lvmSpec := $storage.lvm | default dict }} +{{- $lvmEnabled := false }} +{{- if hasKey $lvmSpec "enabled" }} + {{- $lvmEnabled = $lvmSpec.enabled }} {{- end }} +{{- $lvm := dict + "enabled" $lvmEnabled + "chartVersion" ($lvmSpec.chartVersion | default "1.7.0") + "namespace" ($lvmSpec.namespace | default "openebs") + "storageClassName" ($lvmSpec.storageClassName | default "psql-lvm") + "volumeGroup" ($lvmSpec.volumeGroup | default "psql-vg") + "reclaimPolicy" ($lvmSpec.reclaimPolicy | default "Delete") + "volumeBindingMode" ($lvmSpec.volumeBindingMode | default "WaitForFirstConsumer") + "allowVolumeExpansion" (or (eq (toString ($lvmSpec.allowVolumeExpansion | default true)) "true") false) + "values" ($lvmSpec.values | default dict) + "overrideAllValues" ($lvmSpec.overrideAllValues | default dict) +}} + +# EBS gp3 (EKS Auto Mode CSI) — durable fallback, default on +{{- $ebsSpec := $storage.ebs | default dict }} +{{- $ebsEnabled := true }} +{{- if hasKey $ebsSpec "enabled" }} + {{- $ebsEnabled = $ebsSpec.enabled }} +{{- end }} +{{- $ebs := dict + "enabled" $ebsEnabled + "storageClassName" ($ebsSpec.storageClassName | default "psql-ebs") + "provisioner" ($ebsSpec.provisioner | default "ebs.csi.eks.amazonaws.com") + "parameters" ($ebsSpec.parameters | default (dict "type" "gp3" "fsType" "ext4")) + "reclaimPolicy" ($ebsSpec.reclaimPolicy | default "Delete") + "volumeBindingMode" ($ebsSpec.volumeBindingMode | default "WaitForFirstConsumer") + "allowVolumeExpansion" (or (eq (toString ($ebsSpec.allowVolumeExpansion | default true)) "true") false) +}} # ============================================================================== # NodePool (opt-in) — dedicates a pool of nodes for PostgreSQL workloads. @@ -109,14 +152,10 @@ "labels" $labels "helmProviderConfigRef" $helmProviderConfigRef "kubernetesProviderConfigRef" $k8sProviderConfigRef - "storageClass" (dict - "create" $scCreate - "name" $scName - "provisioner" $scProvisioner - "parameters" $scParameters - "reclaimPolicy" $scReclaimPolicy - "volumeBindingMode" $scVolumeBindingMode - "allowVolumeExpansion" $scAllowVolumeExpansion + "storage" (dict + "mayastor" $mayastor + "lvm" $lvm + "ebs" $ebs ) "nodePool" (dict "enabled" $nodePoolEnabled diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index 3a1abaa..8f9b216 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-psql" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-psql" "openebs-lvm" "openebs-mayastor" "storageclass-mayastor" "storageclass-lvm" "storageclass-ebs" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -31,6 +31,11 @@ "scaleToZeroPlugin" (dict "ready" (get $checkReady "cnpg-scale-to-zero")) "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) "nodepoolPsql" (dict "ready" (get $checkReady "nodepool-psql")) + "openebsLvm" (dict "ready" (get $checkReady "openebs-lvm")) + "openebsMayastor" (dict "ready" (get $checkReady "openebs-mayastor")) + "storageClassMayastor" (dict "ready" (get $checkReady "storageclass-mayastor")) + "storageClassLvm" (dict "ready" (get $checkReady "storageclass-lvm")) + "storageClassEbs" (dict "ready" (get $checkReady "storageclass-ebs")) ) }} # ============================================================================== diff --git a/functions/render/120-storageclass.yaml.gotmpl b/functions/render/120-storageclass.yaml.gotmpl deleted file mode 100644 index befa049..0000000 --- a/functions/render/120-storageclass.yaml.gotmpl +++ /dev/null @@ -1,34 +0,0 @@ -# code: language=yaml -# -# StorageClass for PostgreSQL workloads. -# Opt-in via spec.storageClass.enabled. Defaults to gp3 on the EKS Auto Mode -# EBS CSI driver (ebs.csi.eks.amazonaws.com). -# - -{{- if $state.storageClass.create }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-storageclass - annotations: - {{ setResourceNameAnnotation "storageclass" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: storage.k8s.io/v1 - kind: StorageClass - metadata: - name: {{ $state.storageClass.name }} - labels: {{ $state.labels | toJson }} - provisioner: {{ $state.storageClass.provisioner }} - parameters: {{ $state.storageClass.parameters | toJson }} - reclaimPolicy: {{ $state.storageClass.reclaimPolicy }} - volumeBindingMode: {{ $state.storageClass.volumeBindingMode }} - allowVolumeExpansion: {{ $state.storageClass.allowVolumeExpansion }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/160-openebs-lvm.yaml.gotmpl b/functions/render/160-openebs-lvm.yaml.gotmpl new file mode 100644 index 0000000..55e861d --- /dev/null +++ b/functions/render/160-openebs-lvm.yaml.gotmpl @@ -0,0 +1,56 @@ +# code: language=yaml +# +# Helm Release: OpenEBS LVM LocalPV +# +# Provides single-node CoW storage via LVM thin volumes. Cheaper than Mayastor +# but no replication — suitable for branches and dev clusters where node loss +# is acceptable. Requires an LVM Volume Group ($state.storage.lvm.volumeGroup) +# pre-configured on each NVMe node — phase 3 of the pivot wires this up via +# Karpenter NodePool userData. +# +# Chart: https://openebs.github.io/lvm-localpv (chart name: lvm-localpv) +# Upstream: https://github.com/openebs/lvm-localpv +# + +{{- $lvm := $state.storage.lvm }} +{{- if $lvm.enabled }} +--- +apiVersion: helm.m.crossplane.io/v1beta1 +kind: Release +metadata: + name: openebs-lvm + annotations: + {{ setResourceNameAnnotation "openebs-lvm" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + chart: + name: lvm-localpv + repository: https://openebs.github.io/lvm-localpv + version: {{ $lvm.chartVersion | quote }} + namespace: {{ $lvm.namespace }} + {{- if $lvm.overrideAllValues }} + values: + {{- toYaml $lvm.overrideAllValues | nindent 6 }} + {{- else }} + {{- $chartDefaults := dict }} + {{- if $state.nodePool.enabled }} + {{- $_ := set $chartDefaults "lvmNode" (dict + "nodeSelector" $state.nodePool.nodeSelector + "tolerations" $state.nodePool.tolerations + ) }} + {{- $_ := set $chartDefaults "lvmController" (dict + "nodeSelector" $state.nodePool.nodeSelector + "tolerations" $state.nodePool.tolerations + ) }} + {{- end }} + {{- $mergedValues := mergeOverwrite $chartDefaults ($lvm.values | default dict) }} + values: + {{- toYaml $mergedValues | nindent 6 }} + {{- end }} + rollbackLimit: 3 + providerConfigRef: + name: {{ $state.helmProviderConfigRef.name }} + kind: {{ $state.helmProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/165-openebs-mayastor.yaml.gotmpl b/functions/render/165-openebs-mayastor.yaml.gotmpl new file mode 100644 index 0000000..6fdcba5 --- /dev/null +++ b/functions/render/165-openebs-mayastor.yaml.gotmpl @@ -0,0 +1,55 @@ +# code: language=yaml +# +# Helm Release: OpenEBS Mayastor (Replicated NVMe-oF) +# +# Provides replicated CoW storage via NVMe-over-TCP. Enterprise default for +# primary serving clusters — HA across N nodes (replicationFactor) plus +# instant CoW snapshots for branching. +# +# Node-side prerequisites (provided by phase 3's NodePool config): +# - 2MiB hugepages (`vm.nr_hugepages=1024` minimum) +# - `nvme-tcp` kernel module loaded +# - Local NVMe instance-store devices for DiskPool allocation +# +# Chart: https://openebs.github.io/mayastor-extensions (chart name: mayastor) +# Upstream: https://github.com/openebs/mayastor +# + +{{- $mayastor := $state.storage.mayastor }} +{{- if $mayastor.enabled }} +--- +apiVersion: helm.m.crossplane.io/v1beta1 +kind: Release +metadata: + name: openebs-mayastor + annotations: + {{ setResourceNameAnnotation "openebs-mayastor" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + chart: + name: mayastor + repository: https://openebs.github.io/mayastor-extensions + version: {{ $mayastor.chartVersion | quote }} + namespace: {{ $mayastor.namespace }} + {{- if $mayastor.overrideAllValues }} + values: + {{- toYaml $mayastor.overrideAllValues | nindent 6 }} + {{- else }} + {{- $chartDefaults := dict }} + {{- if $state.nodePool.enabled }} + {{- $_ := set $chartDefaults "io_engine" (dict + "nodeSelector" $state.nodePool.nodeSelector + "tolerations" $state.nodePool.tolerations + ) }} + {{- end }} + {{- $mergedValues := mergeOverwrite $chartDefaults ($mayastor.values | default dict) }} + values: + {{- toYaml $mergedValues | nindent 6 }} + {{- end }} + rollbackLimit: 3 + providerConfigRef: + name: {{ $state.helmProviderConfigRef.name }} + kind: {{ $state.helmProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/170-storageclass-mayastor.yaml.gotmpl b/functions/render/170-storageclass-mayastor.yaml.gotmpl new file mode 100644 index 0000000..b52097d --- /dev/null +++ b/functions/render/170-storageclass-mayastor.yaml.gotmpl @@ -0,0 +1,64 @@ +# code: language=yaml +# +# StorageClass + VolumeSnapshotClass: psql-mayastor (replicated CoW) +# +# Created only when storage.mayastor.enabled. Enterprise default for primary +# PSQLClusters — replicated NVMe-oF + thin-provisioned + CSI snapshots for +# instant CoW branching via PSQLBranch. +# + +{{- $mayastor := $state.storage.mayastor }} +{{- if $mayastor.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-storageclass-mayastor + annotations: + {{ setResourceNameAnnotation "storageclass-mayastor" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: {{ $mayastor.storageClassName }} + labels: {{ $state.labels | toJson }} + provisioner: io.openebs.csi-mayastor + parameters: + repl: "{{ $mayastor.replicationFactor }}" + protocol: nvmf + thin: "{{ $mayastor.thin }}" + ioTimeout: "60" + fsType: ext4 + reclaimPolicy: {{ $mayastor.reclaimPolicy }} + volumeBindingMode: {{ $mayastor.volumeBindingMode }} + allowVolumeExpansion: {{ $mayastor.allowVolumeExpansion }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-volumesnapshotclass-mayastor + annotations: + {{ setResourceNameAnnotation "volumesnapshotclass-mayastor" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshotClass + metadata: + name: {{ $mayastor.storageClassName }} + labels: {{ $state.labels | toJson }} + driver: io.openebs.csi-mayastor + deletionPolicy: Delete + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/175-storageclass-lvm.yaml.gotmpl b/functions/render/175-storageclass-lvm.yaml.gotmpl new file mode 100644 index 0000000..f1de387 --- /dev/null +++ b/functions/render/175-storageclass-lvm.yaml.gotmpl @@ -0,0 +1,65 @@ +# code: language=yaml +# +# StorageClass + VolumeSnapshotClass: psql-lvm (single-node CoW) +# +# Created only when storage.lvm.enabled. LVM thin volumes provide CoW on a +# single node; suitable for branches and dev clusters where node-loss is +# tolerable. References the volume group set up by phase 3's NodePool config. +# + +{{- $lvm := $state.storage.lvm }} +{{- if $lvm.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-storageclass-lvm + annotations: + {{ setResourceNameAnnotation "storageclass-lvm" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: {{ $lvm.storageClassName }} + labels: {{ $state.labels | toJson }} + provisioner: local.csi.openebs.io + parameters: + storage: lvm + volgroup: {{ $lvm.volumeGroup | quote }} + thinProvision: "yes" + fsType: ext4 + reclaimPolicy: {{ $lvm.reclaimPolicy }} + volumeBindingMode: {{ $lvm.volumeBindingMode }} + allowVolumeExpansion: {{ $lvm.allowVolumeExpansion }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-volumesnapshotclass-lvm + annotations: + {{ setResourceNameAnnotation "volumesnapshotclass-lvm" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshotClass + metadata: + name: {{ $lvm.storageClassName }} + labels: {{ $state.labels | toJson }} + driver: local.csi.openebs.io + deletionPolicy: Delete + parameters: + snapSize: "1073741824" + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/180-storageclass-ebs.yaml.gotmpl b/functions/render/180-storageclass-ebs.yaml.gotmpl new file mode 100644 index 0000000..c167f78 --- /dev/null +++ b/functions/render/180-storageclass-ebs.yaml.gotmpl @@ -0,0 +1,39 @@ +# code: language=yaml +# +# StorageClass: psql-ebs (durable EBS gp3, no CoW) +# +# Always-on fallback profile. Uses the EKS Auto Mode CSI driver +# (ebs.csi.eks.amazonaws.com). Branching against this profile uses +# pg_basebackup (no VolumeSnapshotClass — EBS snapshots are not CoW +# at restore, so they don't give the instant-fork semantics PSQLBranch +# expects from a snapshot path). +# + +{{- $ebs := $state.storage.ebs }} +{{- if $ebs.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-storageclass-ebs + annotations: + {{ setResourceNameAnnotation "storageclass-ebs" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: {{ $ebs.storageClassName }} + labels: {{ $state.labels | toJson }} + provisioner: {{ $ebs.provisioner }} + parameters: {{ $ebs.parameters | toJson }} + reclaimPolicy: {{ $ebs.reclaimPolicy }} + volumeBindingMode: {{ $ebs.volumeBindingMode }} + allowVolumeExpansion: {{ $ebs.allowVolumeExpansion }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} From cc258eb1f01f5a55c448a1c6adb00957263a2a68 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 24 Apr 2026 19:34:15 -0500 Subject: [PATCH 08/44] feat: split NodePool into branches + primary sub-pools (phase 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the single Karpenter NodePool with two sub-pools targeting NVMe arm64 instance-store nodes (i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge): - nodePool.branches: spot — for ephemeral PSQLBranch workloads. Spot is acceptable since branches are reproducible. - nodePool.primary: on-demand — for PSQLCluster primaries and operators (CNPG, Atlas, scale-to-zero). Spot preemption would lose a Mayastor replica, so on-demand is the right default for serving workloads. Each sub-pool has its own labels (sub-pool=branches | sub-pool=primary) and matching taints so workloads can target one specifically. Operators ride the primary sub-pool via nodeSelector + tolerations injected from state-init. Render templates: - 150-nodepool.yaml.gotmpl deleted - 140-nodepool-branches.yaml.gotmpl: spot sub-pool - 145-nodepool-primary.yaml.gotmpl: on-demand sub-pool + Usage protection for CNPG/Atlas Releases against premature NodePool deletion. state-init defaults the new sub-pool blocks; state-status observes both. nodePool.enabled stays default-false — existing claims unchanged. Out of scope for this commit: node-side prep for Mayastor + LVM (hugepages, nvme-tcp module, LVM volume group on instance-store NVMe). That's phase 3b (a separate concern that needs careful image / runtime choices) — without it, mayastor.enabled and lvm.enabled won't bind PVCs. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- apis/psqlstacks/definition.yaml | 89 +++++++++++++----- functions/render/000-state-init.yaml.gotmpl | 74 +++++++++++---- functions/render/010-state-status.yaml.gotmpl | 5 +- .../render/140-nodepool-branches.yaml.gotmpl | 53 +++++++++++ .../render/145-nodepool-primary.yaml.gotmpl | 93 +++++++++++++++++++ functions/render/150-nodepool.yaml.gotmpl | 80 ---------------- 6 files changed, 268 insertions(+), 126 deletions(-) create mode 100644 functions/render/140-nodepool-branches.yaml.gotmpl create mode 100644 functions/render/145-nodepool-primary.yaml.gotmpl delete mode 100644 functions/render/150-nodepool.yaml.gotmpl diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index f46c9dc..ae9d840 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -178,39 +178,18 @@ spec: type: boolean default: true nodePool: - description: Optional dedicated Karpenter NodePool for PostgreSQL workloads. When enabled, CNPG, the scale-to-zero plugin, and Atlas schedule here via nodeSelector + tolerations. (Phase 3 of the CNPG pivot will split this into branches / primary sub-pools with hugepages + nvme-tcp pre-configured.) + description: "Dedicated Karpenter NodePools for PostgreSQL workloads. Two sub-pools — `branches` (spot arm64 NVMe, for ephemeral PSQLBranch workloads) and `primary` (on-demand arm64 NVMe, for replicated-cow PSQLCluster primaries where spot preemption would lose a Mayastor replica). Defaults to disabled — node-side prep for Mayastor/LVM (hugepages, nvme-tcp, LVM volume groups) is a separate concern handled in phase 3b. CNPG / scale-to-zero / Atlas schedule on the primary pool when enabled." type: object properties: enabled: + description: Master toggle. When false, neither sub-pool is created. When true, each sub-pool's `enabled` controls individual creation. type: boolean default: false nodeClassName: - description: EKS NodeClass to reference. Defaults to "default". + description: EKS NodeClass shared by both sub-pools. Defaults to "default". type: string - limits: - type: object - properties: - cpu: - type: string - memory: - type: string - requirements: - description: Karpenter scheduling requirements. Defaults to arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). - type: array - items: - type: object - properties: - key: - type: string - operator: - type: string - enum: [In, NotIn, Exists, DoesNotExist, Gt, Lt] - values: - type: array - items: - type: string - required: [key, operator] disruption: + description: Karpenter disruption policy applied to both sub-pools. type: object properties: consolidationPolicy: @@ -220,6 +199,66 @@ spec: consolidateAfter: type: string default: "60s" + branches: + description: Spot arm64 NVMe sub-pool for branches and dev workloads. Cost-optimized; spot preemption is acceptable because branches are ephemeral. + type: object + properties: + enabled: + type: boolean + default: true + limits: + type: object + properties: + cpu: + type: string + memory: + type: string + requirements: + description: Karpenter scheduling requirements. Defaults to arm64 spot on i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge (NVMe instance-store, Graviton). + type: array + items: + type: object + properties: + key: + type: string + operator: + type: string + enum: [In, NotIn, Exists, DoesNotExist, Gt, Lt] + values: + type: array + items: + type: string + required: [key, operator] + primary: + description: On-demand arm64 NVMe sub-pool for replicated-cow PSQLCluster primaries. Spot preemption would lose a Mayastor replica, so on-demand is the right default for serving workloads. + type: object + properties: + enabled: + type: boolean + default: true + limits: + type: object + properties: + cpu: + type: string + memory: + type: string + requirements: + description: Karpenter scheduling requirements. Defaults to arm64 on-demand on i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge (NVMe instance-store, Graviton). + type: array + items: + type: object + properties: + key: + type: string + operator: + type: string + enum: [In, NotIn, Exists, DoesNotExist, Gt, Lt] + values: + type: array + items: + type: string + required: [key, operator] cnpg: description: Configuration for the CloudNativePG operator Helm release. type: object diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 6ef851b..ac925e1 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -101,33 +101,62 @@ }} # ============================================================================== -# NodePool (opt-in) — dedicates a pool of nodes for PostgreSQL workloads. -# Defaults: arm64 spot on r7g/m7g (memory-optimized + general-purpose Graviton). -# Phase 3 of the CNPG pivot splits this into branches / primary sub-pools with -# hugepages + nvme-tcp pre-configured for Mayastor. +# NodePool — branches + primary sub-pools (NVMe arm64 Graviton) +# Defaults to enabled=false; existing claims unchanged. When enabled, both +# sub-pools materialize unless individually disabled. +# +# Common scheduling: nodeSelector workload-type=psql, taint psql=true:NoSchedule. +# Each pool adds a sub-label (sub-pool=branches | sub-pool=primary) and matching +# taint (sub-pool=branches:NoSchedule | sub-pool=primary:NoSchedule) so workloads +# can target one pool specifically. # ============================================================================== {{- $nodePoolSpec := $spec.nodePool | default dict }} {{- $nodePoolEnabled := false }} {{- if hasKey $nodePoolSpec "enabled" }} {{- $nodePoolEnabled = $nodePoolSpec.enabled }} {{- end }} -{{- $nodePoolName := printf "%s-psql" $clusterName }} {{- $nodePoolNodeClassName := $nodePoolSpec.nodeClassName | default "default" }} -{{- $nodePoolLimits := $nodePoolSpec.limits | default (dict "cpu" "16" "memory" "64Gi") }} -{{- $nodePoolRequirements := $nodePoolSpec.requirements | default (list +{{- $nodePoolDisruption := $nodePoolSpec.disruption | default (dict "consolidationPolicy" "WhenEmptyOrUnderutilized" "consolidateAfter" "60s") }} + +# Default NVMe arm64 instance types +{{- $nvmeInstanceTypes := list "i4g.2xlarge" "i4g.4xlarge" "im4gn.2xlarge" }} + +# branches sub-pool (spot) +{{- $branchesSpec := $nodePoolSpec.branches | default dict }} +{{- $branchesEnabled := true }} +{{- if hasKey $branchesSpec "enabled" }} + {{- $branchesEnabled = $branchesSpec.enabled }} +{{- end }} +{{- $branchesLimits := $branchesSpec.limits | default (dict "cpu" "32" "memory" "128Gi") }} +{{- $branchesRequirements := $branchesSpec.requirements | default (list (dict "key" "kubernetes.io/arch" "operator" "In" "values" (list "arm64")) (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "spot")) - (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" (list "r7g.large" "r7g.xlarge" "m7g.large" "m7g.xlarge")) + (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) +) }} + +# primary sub-pool (on-demand) +{{- $primarySpec := $nodePoolSpec.primary | default dict }} +{{- $primaryEnabled := true }} +{{- if hasKey $primarySpec "enabled" }} + {{- $primaryEnabled = $primarySpec.enabled }} +{{- end }} +{{- $primaryLimits := $primarySpec.limits | default (dict "cpu" "32" "memory" "128Gi") }} +{{- $primaryRequirements := $primarySpec.requirements | default (list + (dict "key" "kubernetes.io/arch" "operator" "In" "values" (list "arm64")) + (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "on-demand")) + (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) ) }} -{{- $nodePoolDisruption := $nodePoolSpec.disruption | default (dict "consolidationPolicy" "WhenEmptyOrUnderutilized" "consolidateAfter" "60s") }} -{{- $nodePoolTaintKey := "psql" }} -{{- $nodePoolTaintValue := "true" }} +# Operator scheduling — operators (CNPG, Atlas, scale-to-zero) ride the primary +# pool when enabled (more reliable than spot for control-plane components) {{- $nodePoolNodeSelector := dict }} {{- $nodePoolTolerations := list }} -{{- if $nodePoolEnabled }} - {{- $nodePoolNodeSelector = dict "workload-type" "psql" }} - {{- $nodePoolTolerations = list (dict "key" $nodePoolTaintKey "value" $nodePoolTaintValue "effect" "NoSchedule") }} +{{- if and $nodePoolEnabled $primaryEnabled }} + {{- $nodePoolNodeSelector = dict "workload-type" "psql" "sub-pool" "primary" }} + {{- $nodePoolTolerations = list + (dict "key" "psql" "value" "true" "effect" "NoSchedule") + (dict "key" "sub-pool" "value" "primary" "effect" "NoSchedule") + }} {{- end }} # ============================================================================== @@ -159,15 +188,22 @@ ) "nodePool" (dict "enabled" $nodePoolEnabled - "name" $nodePoolName "nodeClassName" $nodePoolNodeClassName - "limits" $nodePoolLimits - "requirements" $nodePoolRequirements "disruption" $nodePoolDisruption - "taintKey" $nodePoolTaintKey - "taintValue" $nodePoolTaintValue "nodeSelector" $nodePoolNodeSelector "tolerations" $nodePoolTolerations + "branches" (dict + "enabled" $branchesEnabled + "name" (printf "%s-psql-branches" $clusterName) + "limits" $branchesLimits + "requirements" $branchesRequirements + ) + "primary" (dict + "enabled" $primaryEnabled + "name" (printf "%s-psql-primary" $clusterName) + "limits" $primaryLimits + "requirements" $primaryRequirements + ) ) "cnpg" (dict "name" ($cnpg.name | default "cloudnative-pg") diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index 8f9b216..ade157b 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-psql" "openebs-lvm" "openebs-mayastor" "storageclass-mayastor" "storageclass-lvm" "storageclass-ebs" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "openebs-lvm" "openebs-mayastor" "storageclass-mayastor" "storageclass-lvm" "storageclass-ebs" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -30,7 +30,8 @@ "cnpg" (dict "ready" (get $checkReady "cnpg-operator")) "scaleToZeroPlugin" (dict "ready" (get $checkReady "cnpg-scale-to-zero")) "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) - "nodepoolPsql" (dict "ready" (get $checkReady "nodepool-psql")) + "nodepoolBranches" (dict "ready" (get $checkReady "nodepool-branches")) + "nodepoolPrimary" (dict "ready" (get $checkReady "nodepool-primary")) "openebsLvm" (dict "ready" (get $checkReady "openebs-lvm")) "openebsMayastor" (dict "ready" (get $checkReady "openebs-mayastor")) "storageClassMayastor" (dict "ready" (get $checkReady "storageclass-mayastor")) diff --git a/functions/render/140-nodepool-branches.yaml.gotmpl b/functions/render/140-nodepool-branches.yaml.gotmpl new file mode 100644 index 0000000..3f8f49a --- /dev/null +++ b/functions/render/140-nodepool-branches.yaml.gotmpl @@ -0,0 +1,53 @@ +# code: language=yaml +# +# Karpenter NodePool: branches sub-pool (spot arm64 NVMe) +# +# Targets ephemeral PSQLBranch workloads. Spot capacity is acceptable here +# because branches are designed to be reproducible from their parent. +# +# Labels: workload-type=psql, sub-pool=branches +# Taints: psql=true:NoSchedule, sub-pool=branches:NoSchedule +# + +{{- if and $state.nodePool.enabled $state.nodePool.branches.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-nodepool-branches + annotations: + {{ setResourceNameAnnotation "nodepool-branches" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: karpenter.sh/v1 + kind: NodePool + metadata: + name: {{ $state.nodePool.branches.name }} + spec: + template: + metadata: + labels: + workload-type: psql + sub-pool: branches + spec: + nodeClassRef: + group: eks.amazonaws.com + kind: NodeClass + name: {{ $state.nodePool.nodeClassName }} + taints: + - key: psql + value: "true" + effect: NoSchedule + - key: sub-pool + value: branches + effect: NoSchedule + requirements: {{ $state.nodePool.branches.requirements | toJson }} + limits: {{ $state.nodePool.branches.limits | toJson }} + disruption: {{ $state.nodePool.disruption | toJson }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/145-nodepool-primary.yaml.gotmpl b/functions/render/145-nodepool-primary.yaml.gotmpl new file mode 100644 index 0000000..3a55306 --- /dev/null +++ b/functions/render/145-nodepool-primary.yaml.gotmpl @@ -0,0 +1,93 @@ +# code: language=yaml +# +# Karpenter NodePool: primary sub-pool (on-demand arm64 NVMe) +# +# Targets PSQLCluster primary serving DBs and the operators (CNPG, +# scale-to-zero, Atlas). On-demand because spot preemption of a Mayastor +# storage node would lose a replica — unacceptable for production data. +# +# Labels: workload-type=psql, sub-pool=primary +# Taints: psql=true:NoSchedule, sub-pool=primary:NoSchedule +# +# CNPG / Atlas / scale-to-zero schedule here via the nodeSelector + +# tolerations injected by their respective Helm release templates +# (see $state.nodePool.nodeSelector / .tolerations in state-init). +# + +{{- if and $state.nodePool.enabled $state.nodePool.primary.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-nodepool-primary + annotations: + {{ setResourceNameAnnotation "nodepool-primary" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: karpenter.sh/v1 + kind: NodePool + metadata: + name: {{ $state.nodePool.primary.name }} + spec: + template: + metadata: + labels: + workload-type: psql + sub-pool: primary + spec: + nodeClassRef: + group: eks.amazonaws.com + kind: NodeClass + name: {{ $state.nodePool.nodeClassName }} + taints: + - key: psql + value: "true" + effect: NoSchedule + - key: sub-pool + value: primary + effect: NoSchedule + requirements: {{ $state.nodePool.primary.requirements | toJson }} + limits: {{ $state.nodePool.primary.limits | toJson }} + disruption: {{ $state.nodePool.disruption | toJson }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# ============================================================================== +# Usages: protect the primary NodePool from being deleted before the Helm +# releases that depend on it (CNPG operator, Atlas operator). If the NodePool +# vanishes first, operator pods lose their nodes and Helm cleanup hangs. +# ============================================================================== +{{- $pinnedReleases := dict + "cloudnative-pg" "cnpg" + "atlas-operator" "atlasOperator" +}} +{{- range $release, $obsKey := $pinnedReleases }} + {{- $releaseReady := (get (get $state.observed $obsKey | default dict) "ready") }} + {{- if and $state.observed.nodepoolPrimary.ready $releaseReady }} +--- +apiVersion: protection.crossplane.io/v1beta1 +kind: Usage +metadata: + name: {{ $state.name }}-delete-{{ $release }}-before-nodepool-primary + annotations: + {{ setResourceNameAnnotation (printf "usage-np-primary-%s" $release) }} + labels: {{ $state.labels | toJson }} +spec: + replayDeletion: true + of: + apiVersion: kubernetes.m.crossplane.io/v1alpha1 + kind: Object + resourceRef: + name: {{ $state.name }}-nodepool-primary + by: + apiVersion: helm.m.crossplane.io/v1beta1 + kind: Release + resourceRef: + name: {{ $release }} + {{- end }} +{{- end }} +{{- end }} diff --git a/functions/render/150-nodepool.yaml.gotmpl b/functions/render/150-nodepool.yaml.gotmpl deleted file mode 100644 index b88a9ef..0000000 --- a/functions/render/150-nodepool.yaml.gotmpl +++ /dev/null @@ -1,80 +0,0 @@ -# code: language=yaml -# -# Karpenter NodePool for dedicated psql workloads. -# StackGres operator + REST API + jobs and the Atlas operator are scheduled -# here via nodeSelector + tolerations. Opt-in via spec.nodePool.enabled. -# - -{{- if $state.nodePool.enabled }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-nodepool-psql - annotations: - {{ setResourceNameAnnotation "nodepool-psql" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: karpenter.sh/v1 - kind: NodePool - metadata: - name: {{ $state.nodePool.name }} - spec: - template: - metadata: - labels: - workload-type: psql - spec: - nodeClassRef: - group: eks.amazonaws.com - kind: NodeClass - name: {{ $state.nodePool.nodeClassName }} - taints: - - key: {{ $state.nodePool.taintKey }} - value: {{ $state.nodePool.taintValue | quote }} - effect: NoSchedule - requirements: {{ $state.nodePool.requirements | toJson }} - limits: {{ $state.nodePool.limits | toJson }} - disruption: {{ $state.nodePool.disruption | toJson }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} - -# ============================================================================== -# Usages: protect NodePool from being deleted before the Helm releases. -# If the NodePool is deleted first, operator pods lose their nodes and the -# release cleanup hangs. -# ============================================================================== -{{- $pinnedReleases := dict - "stackgres-operator" "stackgresOperator" - "atlas-operator" "atlasOperator" -}} -{{- range $release, $obsKey := $pinnedReleases }} - {{- $releaseReady := (get (get $state.observed $obsKey | default dict) "ready") }} - {{- if and $state.observed.nodepoolPsql.ready $releaseReady }} ---- -apiVersion: protection.crossplane.io/v1beta1 -kind: Usage -metadata: - name: {{ $state.name }}-delete-{{ $release }}-before-nodepool - annotations: - {{ setResourceNameAnnotation (printf "usage-np-%s" $release) }} - labels: {{ $state.labels | toJson }} -spec: - replayDeletion: true - of: - apiVersion: kubernetes.m.crossplane.io/v1alpha1 - kind: Object - resourceRef: - name: {{ $state.name }}-nodepool-psql - by: - apiVersion: helm.m.crossplane.io/v1beta1 - kind: Release - resourceRef: - name: {{ $release }} - {{- end }} -{{- end }} -{{- end }} From d6b5648c8655452341037529174b04ce07c5454b Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 24 Apr 2026 20:01:31 -0500 Subject: [PATCH 09/44] feat: install cnpg-i-scale-to-zero plugin (phase 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Inlines the upstream cnpg-i-scale-to-zero v0.1.7 release manifest as 9 Crossplane Kubernetes Objects: - ServiceAccount cnpg-scale-to-zero-plugin - ClusterRole cnpg-scale-to-zero-sidecar-role + ClusterRoleBinding - Secret scale-to-zero-config (sidecar image reference, paired to plugin version via stringData — k8s base64-encodes on apply) - Self-signed cert-manager Issuer + 2 Certificates (server + client) for the gRPC TLS material the CNPG operator uses to reach the plugin - Service scale-to-zero (cnpg.io/pluginPort=9090, cnpg.io/pluginName annotations) - Deployment scale-to-zero (the plugin gRPC server) Plugin and sidecar images both pin to spec.scaleToZeroPlugin.version (default v0.1.7). Secret is renamed scale-to-zero-config (was scale-to-zero-config-c2c2544fbk in upstream — drops the kustomize hash suffix since we emit the resources as separate Objects). All resources are gated on spec.scaleToZeroPlugin.enabled (default true — the plugin is zero-cost when no PSQLCluster opts in). Source URL is annotated for renovate tracking: source: https://github.com/xataio/cnpg-i-scale-to-zero/releases/download/$VER/manifest.yaml renovate: datasource=github-releases depName=xataio/cnpg-i-scale-to-zero Prereq: cert-manager must be installed (provided by the dns-stack in hops-ops). Without it, the Issuer + Certificate resources won't reconcile and the plugin Deployment won't have its TLS volumes available. When PSQLClusters opt into scale-to-zero, they add: metadata.annotations: xata.io/scale-to-zero-enabled: "true" xata.io/scale-to-zero-inactivity-minutes: "10" spec.plugins: - name: cnpg-i-scale-to-zero.xata.io Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- .../render/210-cnpg-scale-to-zero.yaml.gotmpl | 380 ++++++++++++++++++ 1 file changed, 380 insertions(+) create mode 100644 functions/render/210-cnpg-scale-to-zero.yaml.gotmpl diff --git a/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl b/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl new file mode 100644 index 0000000..bcb2583 --- /dev/null +++ b/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl @@ -0,0 +1,380 @@ +# code: language=yaml +# +# cnpg-i-scale-to-zero plugin install +# +# Materializes the upstream release manifest as a set of Crossplane Objects. +# Source: https://github.com/xataio/cnpg-i-scale-to-zero/releases/download/$VERSION/manifest.yaml +# (renovate: datasource=github-releases depName=xataio/cnpg-i-scale-to-zero) +# +# Prereq: cert-manager must already be installed on the target cluster — the +# plugin uses cert-manager Certificate + Issuer to provision the gRPC server +# and client TLS material. In hops-ops, cert-manager is provided by +# the dns-stack (deploy that first). +# +# When PSQLClusters opt into scale-to-zero (via annotations + plugins:[] entry), +# the operator routes lifecycle hooks here over gRPC; the plugin injects a +# sidecar (image $state.scaleToZeroPlugin.version) into each cluster pod that +# monitors connection activity and toggles spec.instances=0 on idle. +# + +{{- $s2z := $state.scaleToZeroPlugin }} +{{- if $s2z.enabled }} + +{{- $ns := $s2z.namespace }} +{{- $ver := $s2z.version }} +{{- $pluginImage := printf "ghcr.io/xataio/cnpg-i-scale-to-zero:%s" $ver }} +{{- $sidecarImage := printf "ghcr.io/xataio/cnpg-i-scale-to-zero-sidecar:%s" $ver }} + +# ServiceAccount +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-serviceaccount + annotations: + {{ setResourceNameAnnotation "s2z-serviceaccount" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: v1 + kind: ServiceAccount + metadata: + name: cnpg-scale-to-zero-plugin + namespace: {{ $ns }} + labels: {{ $state.labels | toJson }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# ClusterRole — sidecar's required permissions on CNPG Clusters / ScheduledBackups +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-clusterrole + annotations: + {{ setResourceNameAnnotation "s2z-clusterrole" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: cnpg-scale-to-zero-sidecar-role + labels: {{ $state.labels | toJson }} + rules: + - apiGroups: ["postgresql.cnpg.io"] + resources: ["clusters", "scheduledbackups"] + verbs: ["get", "list", "watch", "update", "patch"] + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# ClusterRoleBinding +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-clusterrolebinding + annotations: + {{ setResourceNameAnnotation "s2z-clusterrolebinding" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: cnpg-scale-to-zero-plugin-binding + labels: {{ $state.labels | toJson }} + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cnpg-scale-to-zero-sidecar-role + subjects: + - kind: ServiceAccount + name: cnpg-scale-to-zero-plugin + namespace: {{ $ns }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# Config Secret — sidecar image reference (paired with plugin version) +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-config-secret + annotations: + {{ setResourceNameAnnotation "s2z-config-secret" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: v1 + kind: Secret + metadata: + name: scale-to-zero-config + namespace: {{ $ns }} + labels: {{ $state.labels | toJson }} + type: Opaque + stringData: + SIDECAR_IMAGE: {{ $sidecarImage | quote }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# Self-signed Issuer (cert-manager) for gRPC TLS +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-issuer + annotations: + {{ setResourceNameAnnotation "s2z-issuer" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: cnpg-scale-to-zero-selfsigned-issuer + namespace: {{ $ns }} + labels: {{ $state.labels | toJson }} + spec: + selfSigned: {} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# Server Certificate — gRPC server TLS (used by plugin Deployment) +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-cert-server + annotations: + {{ setResourceNameAnnotation "s2z-cert-server" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: scaletozero-server + namespace: {{ $ns }} + labels: {{ $state.labels | toJson }} + spec: + commonName: scale-to-zero + dnsNames: + - scale-to-zero + - scale-to-zero.{{ $ns }} + - scale-to-zero.{{ $ns }}.svc + - scale-to-zero.{{ $ns }}.svc.cluster.local + duration: 2160h + renewBefore: 360h + isCA: false + issuerRef: + group: cert-manager.io + kind: Issuer + name: cnpg-scale-to-zero-selfsigned-issuer + secretName: scaletozero-server-tls + usages: + - server auth + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# Client Certificate — gRPC client TLS (used by CNPG operator to call plugin) +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-cert-client + annotations: + {{ setResourceNameAnnotation "s2z-cert-client" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: scaletozero-client + namespace: {{ $ns }} + labels: {{ $state.labels | toJson }} + spec: + commonName: scaletozero-client + duration: 2160h + renewBefore: 360h + isCA: false + issuerRef: + group: cert-manager.io + kind: Issuer + name: cnpg-scale-to-zero-selfsigned-issuer + secretName: scaletozero-client-tls + usages: + - client auth + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# Service — exposes plugin gRPC endpoint to CNPG operator +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-service + annotations: + {{ setResourceNameAnnotation "s2z-service" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: v1 + kind: Service + metadata: + name: scale-to-zero + namespace: {{ $ns }} + annotations: + cnpg.io/pluginClientSecret: scaletozero-client-tls + cnpg.io/pluginPort: "9090" + cnpg.io/pluginServerSecret: scaletozero-server-tls + labels: + {{- range $k, $v := $state.labels }} + {{ $k }}: {{ $v | quote }} + {{- end }} + app: scale-to-zero + cnpg.io/pluginName: cnpg-i-scale-to-zero.xata.io + spec: + ports: + - name: grpc + port: 9090 + protocol: TCP + targetPort: 9090 + selector: + app: scale-to-zero + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# Deployment — the plugin's gRPC server +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-s2z-deployment + annotations: + {{ setResourceNameAnnotation "cnpg-scale-to-zero" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: scale-to-zero + namespace: {{ $ns }} + labels: + {{- range $k, $v := $state.labels }} + {{ $k }}: {{ $v | quote }} + {{- end }} + app: scale-to-zero + spec: + replicas: 1 + selector: + matchLabels: + app: scale-to-zero + template: + metadata: + labels: + app: scale-to-zero + spec: + serviceAccountName: cnpg-scale-to-zero-plugin + {{- if $state.nodePool.enabled }} + nodeSelector: {{ $state.nodePool.nodeSelector | toJson }} + tolerations: {{ $state.nodePool.tolerations | toJson }} + {{- end }} + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + containers: + - name: cnpg-i-scale-to-zero + image: {{ $pluginImage | quote }} + args: + - plugin + - --server-cert=/server/tls.crt + - --server-key=/server/tls.key + - --client-cert=/client/tls.crt + - --server-address=:9090 + env: + - name: LOG_LEVEL + value: info + - name: SIDECAR_CPU_REQUEST + value: 50m + - name: SIDECAR_CPU_LIMIT + value: 200m + - name: SIDECAR_MEMORY_REQUEST + value: 64Mi + - name: SIDECAR_MEMORY_LIMIT + value: 64Mi + envFrom: + - secretRef: + name: scale-to-zero-config + ports: + - name: grpc + containerPort: 9090 + protocol: TCP + livenessProbe: + tcpSocket: + port: 9090 + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: 9090 + initialDelaySeconds: 5 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 128Mi + volumeMounts: + - name: server + mountPath: /server + - name: client + mountPath: /client + volumes: + - name: server + secret: + secretName: scaletozero-server-tls + - name: client + secret: + secretName: scaletozero-client-tls + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +{{- end }} From 5d6c3ff6827394c937cce8af92fb720e940ce99d Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 24 Apr 2026 21:59:21 -0500 Subject: [PATCH 10/44] docs/test: refresh README, examples, and render tests for CNPG pivot (phase 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README rewritten: CNPG architecture, three-stage journey (EBS-only → +LVM CoW → +Mayastor HA), full Spec Reference table, prereq notes (cert-manager via cert-manager-stack; node prep deferred to phase 3b) - examples refreshed: - minimal: just clusterName, EBS-only baseline - standard: full production posture (NodePool sub-pools, Mayastor + LVM + EBS, S2Z plugin, Atlas) - local: dev cluster with LVM CoW only (no Mayastor since replication needs >1 node, no NodePool, default Helm provider config) - tests/test-render/main.k: rewritten against the CNPG schema - dropped stackgresOperator-specific tests (field removed) - 11 tests covering: minimal renders platform operators; custom labels propagate; cnpg.overrideAllValues replaces defaults; atlas values merge; namespace propagation; per-component namespace override; helmProviderConfigRef defaults; scaleToZeroPlugin can be disabled; storage.{mayastor,lvm}.enabled compose Helm + StorageClass + VolumeSnapshotClass; nodePool.enabled composes both sub-pools All 11 tests pass; render + validate green on minimal + standard. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 288 ++++++++++++++++-------------- examples/psqlstacks/local.yaml | 15 ++ examples/psqlstacks/minimal.yaml | 11 ++ examples/psqlstacks/standard.yaml | 35 +++- tests/test-render/main.k | 247 +++++++++++++++++++------ 5 files changed, 398 insertions(+), 198 deletions(-) diff --git a/README.md b/README.md index c2e0a7e..44b574a 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,47 @@ # psql-stack -PostgreSQL management stack deploying StackGres and Atlas Operator as Helm releases with safe deletion ordering. +PostgreSQL platform stack on top of [CloudNativePG](https://cloudnative-pg.io/). Installs the operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), Atlas Operator (schema migrations), and a tiered storage layer (OpenEBS Mayastor / OpenEBS LVM LocalPV / EBS gp3) on a target Kubernetes cluster. + +This is the **platform layer** — it does not create any serving Postgres clusters. Per-app DBs live in [`PSQLCluster`](../../psql-cluster/) (separate XR), ephemeral forks in [`PSQLBranch`](../../psql-branch/) (separate XR). ## Why psql-stack? **Without psql-stack:** -- Manual Helm installs of StackGres and Atlas on every cluster -- No guaranteed deletion order — removing StackGres before Atlas leaves orphaned migration state -- Inconsistent operator versions and configuration across environments -- No declarative, reviewable representation of your database tooling +- Manual Helm installs of CNPG, Atlas, and any chosen storage engines on every cluster +- No deletion ordering — removing CNPG before Atlas can leave migrations dangling +- Inconsistent operator + plugin versions across environments +- No declarative substrate for the `cnpg-i-scale-to-zero` plugin (cert-manager-backed gRPC TLS, ServiceAccount + RBAC, sidecar config) **With psql-stack:** -- Single claim deploys both operators with production defaults -- Deletion ordering enforced via Usage resources — Atlas is always removed before StackGres -- Consistent configuration across clusters with customizable Helm values -- Crossplane manages lifecycle, drift detection, and rollback - -## What Gets Deployed - -- **StackGres Operator** — Full PostgreSQL lifecycle management with native Citus support for distributed PostgreSQL via `SGShardedCluster` CRDs -- **Atlas Operator** — Declarative database schema migrations via `AtlasMigration` and `AtlasSchema` CRDs -- **StorageClass** *(on by default, `storageClass.create: true`)* — `psql` class backed by the EKS Auto Mode EBS CSI driver (`ebs.csi.eks.amazonaws.com`). Name mirrors the per-stack convention used by the observe stack (`loki`/`prometheus`/`tempo`). The legacy `gp2` class on EKS Auto Mode uses a deprecated in-tree provisioner that no longer works. -- **Karpenter NodePool** *(opt-in, `nodePool.enabled: true`)* — Dedicated nodes for database workloads. Default: arm64 spot on `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` (memory-optimized Graviton for cheap, low-contention scheduling). StackGres operator + REST API + jobs and Atlas are pinned here via nodeSelector + tolerations. -- **Usage resources** — Atlas is deleted before StackGres to prevent orphaned migration state; Helm releases are deleted before the NodePool so pods drain cleanly. +- Single claim deploys CNPG + S2Z plugin + Atlas + storage backends with production defaults +- Deletion order enforced via `protection.crossplane.io/Usage` resources +- Three storage profiles available — replicated NVMe-oF (Mayastor), single-node CoW (LVM), durable EBS — each independently toggleable +- Optional dedicated Karpenter NodePools (branches sub-pool spot, primary sub-pool on-demand) targeting NVMe instance-store nodes +- Pinnable upstream chart / plugin versions; Renovate keeps them current + +## Components + +| Component | Default | Purpose | +|---|---|---| +| **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | +| **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. Pinnable via `spec.scaleToZeroPlugin.version`. **Requires cert-manager** (provided by [`cert-manager-stack`](../cert-manager/) or `aws-external-dns-stack`). | +| **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | +| **Karpenter NodePools** | off (`spec.nodePool.enabled: false`) | When enabled: `branches` sub-pool (spot arm64 NVMe) and `primary` sub-pool (on-demand arm64 NVMe). Targets `i4g.2xlarge` / `i4g.4xlarge` / `im4gn.2xlarge` by default. | +| **OpenEBS Mayastor** | off (`spec.storage.mayastor.enabled: false`) | Replicated NVMe-oF. Enterprise default for serving primaries — provides CoW snapshots + HA across N nodes. | +| **OpenEBS LVM LocalPV** | off (`spec.storage.lvm.enabled: false`) | Single-node CoW via LVM thin volumes. Cheaper than Mayastor; right for branches and dev. | +| **EBS gp3 StorageClass** | on (`spec.storage.ebs.enabled: true`) | Durable, no CoW. Always-on fallback profile. | + +## Prerequisites + +- **cert-manager** must be installed on the target cluster (the S2Z plugin uses cert-manager `Issuer` + `Certificate`s for its gRPC TLS material). Install via [`cert-manager-stack`](../cert-manager/) — single claim, no AWS deps. +- **Karpenter + EKS Auto Mode** if using `spec.nodePool` (the default `nodeClassName: default` references the EKS Auto Mode managed `NodeClass`). +- **Mayastor + LVM node-side prep** (hugepages, `nvme-tcp` kernel module, LVM volume group on instance-store NVMe) — currently a manual / out-of-stack concern. Phase 3b of the CNPG pivot will add a node-prep DaemonSet template; until it lands, leave `spec.storage.mayastor.enabled` and `spec.storage.lvm.enabled` at `false` unless you've prepped nodes yourself. ## The Journey ### Stage 1: Getting Started -Deploy the stack on a single cluster with defaults. StackGres gets the REST API enabled, Atlas gets dev DB prewarming, and everything lands in the `stackgres` namespace. +Deploy the platform on a single cluster with EBS-only storage. No CoW, no NodePool — just CNPG + S2Z plugin + Atlas + a `psql-ebs` StorageClass. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -40,9 +53,11 @@ spec: clusterName: my-cluster ``` -### Stage 2: Team Usage +`PSQLCluster` resources targeting this stack pick `psql-ebs` as their storage class and use `pg_basebackup`-style branching (works on EBS; takes minutes for non-trivial DBs). -Add labels for ownership tracking, pin operators to a dedicated NodePool, and customize Helm values. +### Stage 2: Adding Local CoW + +Enable LVM LocalPV for fast single-node CoW branches without committing to replicated storage. Useful for dev clusters and preview-environment workflows where node loss is acceptable. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -51,24 +66,22 @@ metadata: name: psql namespace: default spec: - clusterName: production-cluster - namespace: stackgres - labels: - team: platform + clusterName: dev-cluster nodePool: enabled: true - stackgresOperator: - values: - deploy: - restapi: true - atlasOperator: - values: - prewarmDevDB: true + primary: + enabled: false # branches-only sub-pool for dev + storage: + lvm: + enabled: true + volumeGroup: psql-vg ``` -### Stage 3: Multi-Cluster / Advanced +`PSQLBranch` resources can now reference `psql-lvm` for instant CoW forks via VolumeSnapshot. + +### Stage 3: Production HA + Replicated CoW -Override namespaces per component, use a `ClusterProviderConfig`, or fully replace chart defaults. +Enable Mayastor for replicated NVMe-oF storage. Primary serving DBs get CoW + HA replication across the on-demand NodePool; branches stay on LVM (cheaper). ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -78,128 +91,127 @@ metadata: namespace: default spec: clusterName: production-cluster - helmProviderConfigRef: - name: production-cluster - kind: ClusterProviderConfig - stackgresOperator: - namespace: stackgres-system + namespace: cnpg-system + labels: + team: platform + nodePool: + enabled: true + branches: + enabled: true + limits: { cpu: "32", memory: "128Gi" } + primary: + enabled: true + limits: { cpu: "64", memory: "256Gi" } + storage: + mayastor: + enabled: true + replicationFactor: 3 + lvm: + enabled: true + ebs: + enabled: true + scaleToZeroPlugin: + enabled: true atlasOperator: - namespace: atlas-system - overrideAllValues: - prewarmDevDB: false - extraEnvs: - - name: ATLAS_NO_UPDATE_NOTIFIER - value: "true" -``` - -### Local Development - -For local clusters (e.g. kind, k3d), point at the default Helm provider: - -```yaml -apiVersion: hops.ops.com.ai/v1alpha1 -kind: PSQLStack -metadata: - name: psql - namespace: default -spec: - clusterName: local - helmProviderConfigRef: - name: default + values: + prewarmDevDB: true ``` ## Spec Reference -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `clusterName` | string | Yes | — | Target cluster name. Used as default for `helmProviderConfigRef.name` | -| `namespace` | string | No | `stackgres` | Shared namespace for both operators | -| `labels` | object | No | — | Custom labels merged with defaults | -| `managementPolicies` | string[] | No | `["*"]` | Crossplane management policies | -| `helmProviderConfigRef.name` | string | No | `clusterName` | Helm ProviderConfig name | -| `helmProviderConfigRef.kind` | enum | No | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` | -| `kubernetesProviderConfigRef.name` | string | No | `clusterName` | Kubernetes ProviderConfig name (for the NodePool Object) | -| `kubernetesProviderConfigRef.kind` | enum | No | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` | -| `nodePool.enabled` | boolean | No | `false` | Create a dedicated Karpenter NodePool and schedule operators on it | -| `nodePool.nodeClassName` | string | No | `default` | EKS NodeClass name | -| `nodePool.limits.cpu` | string | No | `16` | Pool CPU limit | -| `nodePool.limits.memory` | string | No | `64Gi` | Pool memory limit | -| `nodePool.requirements` | array | No | arm64 spot `r7g.large`/`r7g.xlarge`/`m7g.large`/`m7g.xlarge` | Karpenter scheduling requirements | -| `nodePool.disruption.consolidationPolicy` | enum | No | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | -| `nodePool.disruption.consolidateAfter` | string | No | `60s` | Consolidation delay | -| `storageClass.create` | boolean | No | `true` | Create a StorageClass on the target cluster | -| `storageClass.name` | string | No | `psql` | StorageClass name (mirrors observe stack's per-stack naming) | -| `storageClass.provisioner` | string | No | `ebs.csi.eks.amazonaws.com` | CSI provisioner | -| `storageClass.parameters` | object | No | `{type: gp3, fsType: ext4}` | Provisioner parameters | -| `storageClass.volumeBindingMode` | enum | No | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | -| `storageClass.allowVolumeExpansion` | boolean | No | `true` | Allow PVC resize | -| `storageClass.reclaimPolicy` | enum | No | `Delete` | `Delete` or `Retain` | -| `stackgresOperator.name` | string | No | `stackgres-operator` | Helm release name | -| `stackgresOperator.namespace` | string | No | shared `namespace` | Namespace override | -| `stackgresOperator.values` | object | No | — | Helm values merged with chart defaults | -| `stackgresOperator.overrideAllValues` | object | No | — | Helm values replacing all defaults | -| `atlasOperator.name` | string | No | `atlas-operator` | Helm release name | -| `atlasOperator.namespace` | string | No | shared `namespace` | Namespace override | -| `atlasOperator.values` | object | No | — | Helm values merged with chart defaults | -| `atlasOperator.overrideAllValues` | object | No | — | Helm values replacing all defaults | - -### Helm Values Merging - -Each operator supports two modes: - -- **`values`** — Merged with chart defaults. Use this to tweak individual settings. -- **`overrideAllValues`** — Replaces all defaults entirely. Use this when you need full control. - -If both are set, `overrideAllValues` wins. - -**Chart defaults for StackGres:** -```yaml -deploy: - operator: true - restapi: true -``` - -**Chart defaults for Atlas:** -```yaml -prewarmDevDB: true -``` - -## Status - -| Field | Type | Description | -|-------|------|-------------| -| `status.ready` | boolean | `true` when both operators are healthy | +| Field | Type | Default | Description | +|---|---|---|---| +| `clusterName` | string | _required_ | Target cluster name; default for `helmProviderConfigRef.name` and resource naming | +| `namespace` | string | `cnpg-system` | Shared namespace for CNPG, S2Z plugin, and Atlas | +| `labels` | object | — | Custom labels merged with stack defaults | +| `managementPolicies` | string[] | `["*"]` | Crossplane management policies | +| `helmProviderConfigRef.name` | string | `clusterName` | Helm ProviderConfig name | +| `helmProviderConfigRef.kind` | enum | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` | +| `kubernetesProviderConfigRef.name` | string | `clusterName` | Kubernetes ProviderConfig name | +| `kubernetesProviderConfigRef.kind` | enum | `ProviderConfig` | Same as above | +| **CNPG operator** | | | | +| `cnpg.name` | string | `cloudnative-pg` | Helm release name | +| `cnpg.chartVersion` | string | `0.27.1` | CNPG Helm chart version (tracks CNPG 1.29.x) | +| `cnpg.values` | object | — | Helm values merged with chart defaults | +| `cnpg.overrideAllValues` | object | — | Helm values that replace all defaults | +| **Scale-to-zero plugin** | | | | +| `scaleToZeroPlugin.enabled` | bool | `true` | Install the plugin (zero-cost when no `Cluster` opts in) | +| `scaleToZeroPlugin.version` | string | `v0.1.7` | Plugin release tag | +| `scaleToZeroPlugin.namespace` | string | shared `namespace` | Override | +| **Atlas operator** | | | | +| `atlasOperator.name` | string | `atlas-operator` | Helm release name | +| `atlasOperator.namespace` | string | shared `namespace` | Override | +| `atlasOperator.values` | object | — | Helm values merged with chart defaults | +| `atlasOperator.overrideAllValues` | object | — | Helm values that replace all defaults | +| **NodePool** | | | | +| `nodePool.enabled` | bool | `false` | Master toggle for both sub-pools | +| `nodePool.nodeClassName` | string | `default` | EKS NodeClass referenced by both sub-pools | +| `nodePool.disruption.consolidationPolicy` | enum | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | +| `nodePool.disruption.consolidateAfter` | string | `60s` | Consolidation delay | +| `nodePool.branches.enabled` | bool | `true` | Spot arm64 NVMe sub-pool | +| `nodePool.branches.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | +| `nodePool.branches.requirements` | array | arm64 spot `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | +| `nodePool.primary.enabled` | bool | `true` | On-demand arm64 NVMe sub-pool | +| `nodePool.primary.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | +| `nodePool.primary.requirements` | array | arm64 on-demand `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | +| **Storage: Mayastor** | | | | +| `storage.mayastor.enabled` | bool | `false` | Install OpenEBS Mayastor + create `psql-mayastor` SC | +| `storage.mayastor.chartVersion` | string | `2.10.0` | Helm chart version | +| `storage.mayastor.storageClassName` | string | `psql-mayastor` | StorageClass name | +| `storage.mayastor.replicationFactor` | int | `3` | Replicas per volume | +| `storage.mayastor.thin` | bool | `true` | Thin provisioning (CoW) | +| `storage.mayastor.values` | object | — | Helm values merged with chart defaults | +| `storage.mayastor.overrideAllValues` | object | — | Helm values that replace all defaults | +| **Storage: LVM** | | | | +| `storage.lvm.enabled` | bool | `false` | Install OpenEBS LVM LocalPV + create `psql-lvm` SC | +| `storage.lvm.chartVersion` | string | `1.7.0` | Helm chart version | +| `storage.lvm.storageClassName` | string | `psql-lvm` | StorageClass name | +| `storage.lvm.volumeGroup` | string | `psql-vg` | LVM Volume Group on each node | +| `storage.lvm.values` | object | — | Helm values merged with chart defaults | +| `storage.lvm.overrideAllValues` | object | — | Helm values that replace all defaults | +| **Storage: EBS** | | | | +| `storage.ebs.enabled` | bool | `true` | Create the `psql-ebs` SC | +| `storage.ebs.storageClassName` | string | `psql-ebs` | StorageClass name | +| `storage.ebs.provisioner` | string | `ebs.csi.eks.amazonaws.com` | CSI provisioner | +| `storage.ebs.parameters` | object | `{type: gp3, fsType: ext4}` | Provisioner parameters | +| `storage.ebs.reclaimPolicy` | enum | `Delete` | `Delete` or `Retain` | +| `storage.ebs.volumeBindingMode` | enum | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | +| `storage.ebs.allowVolumeExpansion` | bool | `true` | Allow PVC resize | ## Composed Resources -| Resource | Kind | Purpose | -|----------|------|---------| -| `storageclass` | `kubernetes.m.crossplane.io/Object` | StorageClass (default name `psql`; when `storageClass.create: true`) | -| `nodepool-psql` | `kubernetes.m.crossplane.io/Object` | Karpenter NodePool (only when `nodePool.enabled: true`) | -| `stackgres-operator` | `helm.m.crossplane.io/Release` | StackGres Helm release | -| `atlas-operator` | `helm.m.crossplane.io/Release` | Atlas Operator Helm release | -| `usage-sg-atlas` | `protection.crossplane.io/Usage` | Atlas deleted before StackGres | -| `usage-np-stackgres-operator` | `protection.crossplane.io/Usage` | StackGres drained before NodePool is deleted (when NodePool enabled) | -| `usage-np-atlas-operator` | `protection.crossplane.io/Usage` | Atlas drained before NodePool is deleted (when NodePool enabled) | +| Resource | Kind | When | +|---|---|---| +| `cloudnative-pg` | `helm.m.crossplane.io/Release` | always | +| `atlas-operator` | `helm.m.crossplane.io/Release` | always | +| 9× `-s2z-*` | `kubernetes.m.crossplane.io/Object` | `scaleToZeroPlugin.enabled: true` (default) | +| `-storageclass-ebs` | `kubernetes.m.crossplane.io/Object` | `storage.ebs.enabled: true` (default) | +| `openebs-lvm` | `helm.m.crossplane.io/Release` | `storage.lvm.enabled: true` | +| `-storageclass-lvm` | `kubernetes.m.crossplane.io/Object` | `storage.lvm.enabled: true` | +| `-volumesnapshotclass-lvm` | `kubernetes.m.crossplane.io/Object` | `storage.lvm.enabled: true` | +| `openebs-mayastor` | `helm.m.crossplane.io/Release` | `storage.mayastor.enabled: true` | +| `-storageclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `storage.mayastor.enabled: true` | +| `-volumesnapshotclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `storage.mayastor.enabled: true` | +| `-nodepool-branches` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true && nodePool.branches.enabled: true` | +| `-nodepool-primary` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true && nodePool.primary.enabled: true` | +| Various `Usage` | `protection.crossplane.io/Usage` | when both ends Ready (deletion ordering) | ## Dependencies | Kind | Package | Version | -|------|---------|---------| -| Function | `crossplane-contrib/function-auto-ready` | `>=v0.6.0` | +|---|---|---| +| Function | `crossplane-contrib/function-auto-ready` | `>=v0.6.2` | | Provider | `crossplane-contrib/provider-helm` | `>=v1` | -| Provider | `crossplane-contrib/provider-kubernetes` | `>=v1` (only used when `nodePool.enabled`) | +| Provider | `crossplane-contrib/provider-kubernetes` | `>=v1` | ## Development ```bash -make render # Render all examples -make validate # Validate against Crossplane schemas -make test # Run unit tests (KCL) -make e2e # Run E2E tests (requires cluster) -make build # Build the package -make render:minimal # Render a single example -make validate:standard +make render # Render all examples +make validate # Validate against Crossplane schemas +make test # KCL render tests (assert composed resource shapes) +make build # Build the package +make render:standard # Render a single example ``` ## License diff --git a/examples/psqlstacks/local.yaml b/examples/psqlstacks/local.yaml index 4c3c403..a588752 100644 --- a/examples/psqlstacks/local.yaml +++ b/examples/psqlstacks/local.yaml @@ -1,3 +1,10 @@ +# Local: single-node CoW with LVM, no Mayastor (replication needs >1 node), +# no dedicated NodePool. Useful for kind / k3d dev clusters where you want +# fast branching via LVM thin clones but don't need replicated storage. +# +# Requires LVM tools + a `psql-vg` volume group on the local node before +# psql-lvm PVCs can bind. +# apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack metadata: @@ -7,3 +14,11 @@ spec: clusterName: local helmProviderConfigRef: name: default + storage: + lvm: + enabled: true + volumeGroup: psql-vg + ebs: + enabled: false + mayastor: + enabled: false diff --git a/examples/psqlstacks/minimal.yaml b/examples/psqlstacks/minimal.yaml index 1959e6c..aebd1c5 100644 --- a/examples/psqlstacks/minimal.yaml +++ b/examples/psqlstacks/minimal.yaml @@ -1,3 +1,14 @@ +# Minimal: just the platform operators on a target cluster. +# +# Composes: +# - CloudNativePG operator (Helm) +# - cnpg-i-scale-to-zero plugin (set of Objects, default-on) +# - Atlas operator (Helm) +# - psql-ebs StorageClass (default-on, EBS gp3) +# +# No NodePool, no Mayastor, no LVM. Suitable for clusters that just need +# CNPG with EBS-backed storage and pg_basebackup-only branching. +# apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack metadata: diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index c443fdb..e9ad5f0 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -1,3 +1,20 @@ +# Standard: production posture with Mayastor + LVM CoW storage and dedicated +# NVMe NodePools. +# +# Composes: +# - Karpenter NodePool sub-pools: branches (spot) and primary (on-demand) +# on i4g/im4gn arm64 NVMe instance-store nodes +# - OpenEBS LVM LocalPV (single-node CoW for branches) +# - OpenEBS Mayastor (replicated NVMe-oF for serving primaries) +# - Three StorageClasses: psql-mayastor, psql-lvm, psql-ebs +# - VolumeSnapshotClasses for the two CoW backends +# - CloudNativePG operator (scheduled on the primary NodePool) +# - cnpg-i-scale-to-zero plugin +# - Atlas operator +# +# Requires phase 3b node-prep on each NVMe node (hugepages, nvme-tcp module, +# LVM volume group on instance-store) before Mayastor / LVM PVCs can bind. +# apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack metadata: @@ -10,12 +27,24 @@ spec: team: platform nodePool: enabled: true + branches: + enabled: true + limits: { cpu: "32", memory: "128Gi" } + primary: + enabled: true + limits: { cpu: "32", memory: "128Gi" } storage: + mayastor: + enabled: true + replicationFactor: 3 + thin: true + lvm: + enabled: true + volumeGroup: psql-vg ebs: enabled: true - storageClassName: psql-ebs - cnpg: - values: {} + scaleToZeroPlugin: + enabled: true atlasOperator: values: prewarmDevDB: true diff --git a/tests/test-render/main.k b/tests/test-render/main.k index 0aa1b7c..e3d44d9 100644 --- a/tests/test-render/main.k +++ b/tests/test-render/main.k @@ -3,19 +3,22 @@ import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 import models.k8s.apimachinery.pkg.apis.meta.v1 as metav1 # ============================================================================== -# Unit tests for Psql XRD +# Unit tests for PSQLStack XRD # -# Philosophy: Test YOUR API, not the provider's API. +# Philosophy: test YOUR API, not the provider's API. # - render/validate already checks provider schema correctness # - Tests should verify XRD spec fields produce expected behavior # ============================================================================== _items = [ # ========================================================================== - # Test 1: Minimal input renders both Helm Releases + # Test 1: minimal claim renders the platform operators (CNPG, Atlas) and + # the scale-to-zero plugin install (default-on, gated on + # spec.scaleToZeroPlugin.enabled which defaults true). Storage defaults to + # ebs-only (mayastor + lvm default off until phase 3b lands node prep). # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "minimal-renders-all-components" + metadata.name = "minimal-renders-platform-operators" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" @@ -29,19 +32,32 @@ _items = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "stackgres-operator" + metadata.name = "cloudnative-pg" } { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" metadata.name = "atlas-operator" } + # scale-to-zero plugin Deployment Object (default enabled) + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-s2z-deployment" + } + # ebs StorageClass Object (default enabled) + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-storageclass-ebs" + } ] } } # ========================================================================== - # Test 2: Custom labels are merged with defaults + # Test 2: custom labels merge with the stack's defaults on every composed + # Release. # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "custom-labels-merged-with-defaults" @@ -62,7 +78,7 @@ _items = [ apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" metadata = { - name = "stackgres-operator" + name = "cloudnative-pg" labels = { "hops.ops.com.ai/managed" = "true" "hops.ops.com.ai/psql" = "labeled-psql" @@ -89,10 +105,10 @@ _items = [ } # ========================================================================== - # Test 3: overrideAllValues on StackGres replaces all defaults + # Test 3: spec.cnpg.overrideAllValues replaces all default values # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "stackgres-override-all-values-replaces-defaults" + metadata.name = "cnpg-override-all-values-replaces-defaults" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" @@ -102,9 +118,10 @@ _items = [ metadata.name = "override-test" spec = { clusterName = "my-cluster" - stackgresOperator = { + cnpg = { overrideAllValues = { - deploy = {operator = True, restapi = False} + replicaCount = 2 + config = {data = {LOG_LEVEL = "debug"}} } } } @@ -113,9 +130,10 @@ _items = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "stackgres-operator" + metadata.name = "cloudnative-pg" spec.forProvider.values = { - deploy = {operator = True, restapi = False} + replicaCount = 2 + config = {data = {LOG_LEVEL = "debug"}} } } ] @@ -123,42 +141,7 @@ _items = [ } # ========================================================================== - # Test 4: StackGres user values merge with defaults - # ========================================================================== - metav1alpha1.CompositionTest { - metadata.name = "stackgres-values-merge-with-defaults" - spec = { - compositionPath = "apis/psqlstacks/composition.yaml" - xrdPath = "apis/psqlstacks/definition.yaml" - timeoutSeconds = 60 - validate = False - xr = stacksv1alpha1.PSQLStack { - metadata.name = "sg-merge" - spec = { - clusterName = "my-cluster" - stackgresOperator = { - values = { - grafana = {autoEmbed = True} - } - } - } - } - assertResources = [ - { - apiVersion = "helm.m.crossplane.io/v1beta1" - kind = "Release" - metadata.name = "stackgres-operator" - spec.forProvider.values = { - deploy = {operator = True, restapi = True} - grafana = {autoEmbed = True} - } - } - ] - } - } - - # ========================================================================== - # Test 5: Atlas Operator user values merge with defaults + # Test 4: spec.atlasOperator.values merge with chart defaults # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "atlas-values-merge-with-defaults" @@ -192,7 +175,7 @@ _items = [ } # ========================================================================== - # Test 6: Custom namespace propagates to both components + # Test 5: custom namespace propagates to all platform operators # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "custom-namespace-propagates" @@ -212,7 +195,7 @@ _items = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "stackgres-operator" + metadata.name = "cloudnative-pg" spec.forProvider.namespace = "database-system" } { @@ -226,7 +209,7 @@ _items = [ } # ========================================================================== - # Test 7: Per-component namespace overrides shared namespace + # Test 6: per-component namespace overrides the shared namespace # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "per-component-namespace-override" @@ -239,7 +222,7 @@ _items = [ metadata.name = "ns-override" spec = { clusterName = "my-cluster" - namespace = "stackgres" + namespace = "cnpg-system" atlasOperator = { namespace = "atlas-system" } @@ -249,8 +232,8 @@ _items = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "stackgres-operator" - spec.forProvider.namespace = "stackgres" + metadata.name = "cloudnative-pg" + spec.forProvider.namespace = "cnpg-system" } { apiVersion = "helm.m.crossplane.io/v1beta1" @@ -263,7 +246,7 @@ _items = [ } # ========================================================================== - # Test 8: Helm provider config ref defaults to clusterName + # Test 7: helmProviderConfigRef defaults to clusterName # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "helm-provider-config-defaults-to-cluster-name" @@ -280,7 +263,7 @@ _items = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "stackgres-operator" + metadata.name = "cloudnative-pg" spec.providerConfigRef = { name = "prod-cluster" kind = "ProviderConfig" @@ -298,6 +281,156 @@ _items = [ ] } } + + # ========================================================================== + # Test 8: spec.scaleToZeroPlugin.enabled=false suppresses the plugin install + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "scale-to-zero-plugin-can-be-disabled" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "no-s2z" + spec = { + clusterName = "my-cluster" + scaleToZeroPlugin = {enabled = False} + } + } + # When disabled, no s2z-* Objects are composed. We assert the + # expected platform Releases are still present and ebs StorageClass + # is created — the assertion is a positive presence check; the + # absence of s2z-* resources is implicit (composition gates them + # on enabled=true). + assertResources = [ + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "cloudnative-pg" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "no-s2z-storageclass-ebs" + } + ] + } + } + + # ========================================================================== + # Test 9: storage.mayastor.enabled=true composes the mayastor Helm release + # and StorageClass Object + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "storage-mayastor-enabled-composes-helm-and-sc" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "mayastor-on" + spec = { + clusterName = "my-cluster" + storage = { + mayastor = {enabled = True} + } + } + } + assertResources = [ + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "openebs-mayastor" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "mayastor-on-storageclass-mayastor" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "mayastor-on-volumesnapshotclass-mayastor" + } + ] + } + } + + # ========================================================================== + # Test 10: storage.lvm.enabled=true composes the LVM LocalPV Helm release + # and StorageClass Object + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "storage-lvm-enabled-composes-helm-and-sc" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "lvm-on" + spec = { + clusterName = "my-cluster" + storage = { + lvm = {enabled = True} + } + } + } + assertResources = [ + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "openebs-lvm" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "lvm-on-storageclass-lvm" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "lvm-on-volumesnapshotclass-lvm" + } + ] + } + } + + # ========================================================================== + # Test 11: nodePool.enabled=true composes both branches and primary + # sub-pools by default (each can be individually disabled) + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "nodepool-enabled-composes-both-sub-pools" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "np-on" + spec = { + clusterName = "prod" + nodePool = {enabled = True} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "np-on-nodepool-branches" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "np-on-nodepool-primary" + } + ] + } + } ] items = _items From 2f6f2c8c49deb8983b1ce31c3b43fca33251677e Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 24 Apr 2026 22:56:38 -0500 Subject: [PATCH 11/44] feat: node-prep DaemonSet for Mayastor/LVM prereqs (phase 3b) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a privileged DaemonSet that runs on each NVMe NodePool node and configures the host-level state Mayastor + OpenEBS LVM LocalPV expect to find: - Hugepages (vm.nr_hugepages, default 1024 → 2GiB of 2MiB pages, required by Mayastor SPDK) - nvme-tcp kernel module loaded (Mayastor NVMe-oF transport) - LVM volume group on the first instance-store NVMe device (OpenEBS LVM LocalPV expects the VG to pre-exist) Auto-gated: composed only when nodePool.enabled AND (storage.mayastor.enabled OR storage.lvm.enabled). Inside the script, each step is conditional on the relevant storage backend. Schema additions: - spec.nodePrep.enabled (default true; auto-gated by storage backends) - spec.nodePrep.hugepages.count (default 1024) - spec.nodePrep.image (default alpine:3.20; apk-installs lvm2 + util-linux at startup) Render template 155-node-prep-daemonset.yaml.gotmpl: - DaemonSet with nodeSelector workload-type=psql + tolerations for both psql=true:NoSchedule and sub-pool=*:NoSchedule taints - hostPID + hostNetwork + privileged init container - Init script falls back gracefully on Bottlerocket / Auto Mode where modprobe inside containers is restricted (warns and continues) - Tiny pause container keeps the DS Ready state-init / state-status / 010-state-status updated for the new resource. Validate clean (22 resources on standard), 11/11 render tests still pass. Caveat: live verification on pat-local deferred until Mayastor or LVM is enabled in a PSQLStack claim there. Schema/composition is sound; e2e testing follows when storage backends are turned on. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- apis/psqlstacks/definition.yaml | 18 ++ functions/render/000-state-init.yaml.gotmpl | 19 ++ functions/render/010-state-status.yaml.gotmpl | 3 +- .../155-node-prep-daemonset.yaml.gotmpl | 170 ++++++++++++++++++ 4 files changed, 209 insertions(+), 1 deletion(-) create mode 100644 functions/render/155-node-prep-daemonset.yaml.gotmpl diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index ae9d840..46a454f 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -259,6 +259,24 @@ spec: items: type: string required: [key, operator] + nodePrep: + description: "Node-prep DaemonSet for Mayastor / LVM prerequisites. Runs as a privileged init container on each NVMe NodePool node and configures hugepages (vm.nr_hugepages, required by Mayastor SPDK), loads the nvme-tcp kernel module (Mayastor NVMe-oF transport), and creates the LVM volume group on the first instance-store NVMe device (OpenEBS LVM LocalPV). Composed only when nodePool.enabled AND (storage.mayastor.enabled OR storage.lvm.enabled). EKS Auto Mode / Bottlerocket may pre-load nvme-tcp; verify with `lsmod | grep nvme_tcp` before assuming the modprobe step is needed." + type: object + properties: + enabled: + description: Whether to compose the node-prep DaemonSet. Defaults to true (auto-gated by storage backends — when neither mayastor nor lvm is enabled the DaemonSet is skipped regardless). + type: boolean + default: true + hugepages: + type: object + properties: + count: + description: Number of 2MiB hugepages to allocate on each node. Defaults to 1024 (2GiB total). Required for Mayastor; ignored when storage.mayastor.enabled is false. + type: integer + default: 1024 + image: + description: Container image for the prep init container. Must have apk (or be pre-baked with lvm2 + util-linux). Defaults to alpine:3.20 (apk-installs lvm2 at startup). + type: string cnpg: description: Configuration for the CloudNativePG operator Helm release. type: object diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index ac925e1..fe3a0b0 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -159,6 +159,24 @@ }} {{- end }} +# ============================================================================== +# Node prep — hugepages, nvme-tcp module, LVM volume group on NVMe nodes +# Auto-gated: only emitted when nodePool.enabled AND (mayastor || lvm enabled) +# ============================================================================== +{{- $nodePrepSpec := $spec.nodePrep | default dict }} +{{- $nodePrepEnabled := true }} +{{- if hasKey $nodePrepSpec "enabled" }} + {{- $nodePrepEnabled = $nodePrepSpec.enabled }} +{{- end }} +{{- $nodePrepNeeded := and $nodePoolEnabled (or $mayastorEnabled $lvmEnabled) }} +{{- $nodePrep := dict + "enabled" (and $nodePrepEnabled $nodePrepNeeded) + "hugepages" (dict + "count" (((($nodePrepSpec.hugepages | default dict)).count) | default 1024) + ) + "image" ($nodePrepSpec.image | default "alpine:3.20") +}} + # ============================================================================== # Per-component configuration # ============================================================================== @@ -205,6 +223,7 @@ "requirements" $primaryRequirements ) ) + "nodePrep" $nodePrep "cnpg" (dict "name" ($cnpg.name | default "cloudnative-pg") "namespace" ($cnpg.namespace | default $namespace) diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index ade157b..217498e 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "openebs-lvm" "openebs-mayastor" "storageclass-mayastor" "storageclass-lvm" "storageclass-ebs" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "node-prep" "openebs-lvm" "openebs-mayastor" "storageclass-mayastor" "storageclass-lvm" "storageclass-ebs" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -32,6 +32,7 @@ "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) "nodepoolBranches" (dict "ready" (get $checkReady "nodepool-branches")) "nodepoolPrimary" (dict "ready" (get $checkReady "nodepool-primary")) + "nodePrep" (dict "ready" (get $checkReady "node-prep")) "openebsLvm" (dict "ready" (get $checkReady "openebs-lvm")) "openebsMayastor" (dict "ready" (get $checkReady "openebs-mayastor")) "storageClassMayastor" (dict "ready" (get $checkReady "storageclass-mayastor")) diff --git a/functions/render/155-node-prep-daemonset.yaml.gotmpl b/functions/render/155-node-prep-daemonset.yaml.gotmpl new file mode 100644 index 0000000..e25f66e --- /dev/null +++ b/functions/render/155-node-prep-daemonset.yaml.gotmpl @@ -0,0 +1,170 @@ +# code: language=yaml +# +# Node-prep DaemonSet for Mayastor / LVM prerequisites +# +# Runs as a privileged init container on each NVMe NodePool node and +# configures the host-level state that storage backends require: +# - Hugepages (vm.nr_hugepages, required by Mayastor SPDK) +# - nvme-tcp kernel module (Mayastor NVMe-oF transport) +# - LVM volume group on the first instance-store NVMe device +# (OpenEBS LVM LocalPV expects the VG to pre-exist) +# +# After init, a tiny pause container keeps the pod up so the DaemonSet +# is reported Ready by Kubernetes. +# +# Auto-gated: composed only when nodePool.enabled AND +# (storage.mayastor.enabled OR storage.lvm.enabled). Hugepages + nvme-tcp +# only run when mayastor is enabled; LVM VG only when lvm is enabled. +# +# Caveats: +# - On EKS Auto Mode (Bottlerocket OS) `nvme-tcp` may already be +# pre-loaded; verify with `lsmod | grep nvme_tcp` before assuming +# this step is needed. modprobe inside a container may also be +# restricted on Bottlerocket — script falls back gracefully. +# - Instance-store NVMe device detection uses lsblk model match for +# "Amazon EC2 NVMe Instance Storage". Manual override possible +# by pre-creating the volume group on the host. +# + +{{- if $state.nodePrep.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-node-prep + annotations: + {{ setResourceNameAnnotation "node-prep" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: apps/v1 + kind: DaemonSet + metadata: + name: psql-node-prep + namespace: {{ $state.namespace }} + labels: + {{- range $k, $v := $state.labels }} + {{ $k }}: {{ $v | quote }} + {{- end }} + app: psql-node-prep + spec: + selector: + matchLabels: + app: psql-node-prep + template: + metadata: + labels: + app: psql-node-prep + spec: + hostPID: true + hostNetwork: true + nodeSelector: + workload-type: psql + tolerations: + - key: psql + value: "true" + effect: NoSchedule + - key: sub-pool + operator: Exists + effect: NoSchedule + initContainers: + - name: prep + image: {{ $state.nodePrep.image | quote }} + securityContext: + privileged: true + command: + - /bin/sh + - -c + - | + set -e + # Install lvm + util-linux if not already present (alpine default) + if command -v apk >/dev/null 2>&1; then + apk add --no-cache lvm2 util-linux + fi + + {{- if $state.storage.mayastor.enabled }} + # ---- Hugepages (Mayastor SPDK requires) ---- + echo "Setting vm.nr_hugepages={{ $state.nodePrep.hugepages.count }}" + if [ -w /proc/sys/vm/nr_hugepages ]; then + echo {{ $state.nodePrep.hugepages.count }} > /proc/sys/vm/nr_hugepages + else + echo "WARN: /proc/sys/vm/nr_hugepages not writable; allocate manually" + fi + + # ---- nvme-tcp module ---- + if lsmod 2>/dev/null | grep -q "^nvme_tcp"; then + echo "nvme-tcp already loaded" + else + echo "Attempting modprobe nvme-tcp" + modprobe nvme-tcp 2>/dev/null \ + || nsenter -t 1 -m modprobe nvme-tcp 2>/dev/null \ + || echo "WARN: could not load nvme-tcp; verify on host (Bottlerocket may pre-load it)" + fi + {{- end }} + + {{- if $state.storage.lvm.enabled }} + # ---- LVM volume group on instance-store NVMe ---- + VG_NAME='{{ $state.storage.lvm.volumeGroup }}' + if vgs "$VG_NAME" >/dev/null 2>&1; then + echo "Volume group $VG_NAME already exists" + else + DEVICE=$(lsblk -dno NAME,TYPE,MODEL 2>/dev/null \ + | awk '$2=="disk" && /Amazon EC2 NVMe Instance Storage/ {print "/dev/"$1; exit}') + if [ -z "$DEVICE" ]; then + echo "WARN: no instance-store NVMe device detected; LVM VG not created" + else + echo "Creating PV on $DEVICE and VG $VG_NAME" + pvcreate -f "$DEVICE" + vgcreate "$VG_NAME" "$DEVICE" + fi + fi + {{- end }} + + echo "node-prep complete" + volumeMounts: + - name: dev + mountPath: /dev + - name: sys + mountPath: /sys + - name: host-modules + mountPath: /lib/modules + readOnly: true + - name: proc + mountPath: /proc-host + containers: + - name: pause + image: {{ $state.nodePrep.image | quote }} + command: + - /bin/sh + - -c + - | + # Stay alive so DaemonSet reports Ready. + # Re-running prep on container restart is safe (idempotent). + trap 'exit 0' TERM + while true; do sleep 3600 & wait $!; done + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 50m + memory: 32Mi + volumes: + - name: dev + hostPath: + path: /dev + - name: sys + hostPath: + path: /sys + - name: host-modules + hostPath: + path: /lib/modules + - name: proc + hostPath: + path: /proc + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} From 0b7e2216a7a67b2a9d37568cd00bc430db857756 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 24 Apr 2026 23:41:01 -0500 Subject: [PATCH 12/44] refactor: drop LVM + EBS storage profiles, Mayastor-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-claim cost on Mayastor is controlled via replicationFactor (3 for primaries, 1 for ephemeral branches). LVM was a redundant single-node CoW backend; EBS was just wrapping the cluster's existing default SC and contradicted the stack's CoW-by-default identity. Schema changes: - spec.storage flattened — no more {mayastor,lvm,ebs} profile selector. Top-level fields now: chartVersion, namespace, storageClassName (default "psql"), replicationFactor, thin, reclaimPolicy, volumeBindingMode, allowVolumeExpansion, values, overrideAllValues. - spec.nodePool.enabled defaults to TRUE now (was false). The stack's whole point is dedicated NVMe nodes; the default should make it work. - Mayastor + StorageClass + node-prep DaemonSet all gated on nodePool.enabled. Disable nodePool to opt out (gets you only CNPG + Atlas + S2Z plugin running on the cluster's default SC). Render templates removed: - 160-openebs-lvm.yaml.gotmpl - 175-storageclass-lvm.yaml.gotmpl - 180-storageclass-ebs.yaml.gotmpl Render templates updated: - 165-openebs-mayastor.yaml.gotmpl: gated on nodePool.enabled (was storage.mayastor.enabled), uses flat $state.storage shape - 170-storageclass-mayastor.yaml.gotmpl: same gating + shape - 155-node-prep-daemonset.yaml.gotmpl: dropped LVM VG step, gated on just nodePool.enabled - 010-state-status.yaml.gotmpl: dropped LVM/EBS observed keys PSQLClusters that don't want CoW can specify any other StorageClass that exists on the target cluster (e.g., the EKS Auto Mode default gp3 SC). The stack does NOT compose a non-CoW SC — that's outside its identity. Examples + README rewritten. KCL tests updated: 10/10 passing. Validate clean: 18 resources on minimal, 18 on standard. Live verification on pat-local pending — would need to delete the existing claim (with ebs/lvm enabled) and reapply the simplified one. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 139 +++++++---------- apis/psqlstacks/definition.yaml | 145 +++++------------- examples/psqlstacks/local.yaml | 20 +-- examples/psqlstacks/minimal.yaml | 15 +- examples/psqlstacks/standard.yaml | 31 +--- functions/render/000-state-init.yaml.gotmpl | 97 ++++-------- functions/render/010-state-status.yaml.gotmpl | 9 +- .../155-node-prep-daemonset.yaml.gotmpl | 38 +---- functions/render/160-openebs-lvm.yaml.gotmpl | 56 ------- .../render/165-openebs-mayastor.yaml.gotmpl | 14 +- .../170-storageclass-mayastor.yaml.gotmpl | 26 ++-- .../render/175-storageclass-lvm.yaml.gotmpl | 65 -------- .../render/180-storageclass-ebs.yaml.gotmpl | 39 ----- tests/test-render/main.k | 130 +++++++--------- 14 files changed, 223 insertions(+), 601 deletions(-) delete mode 100644 functions/render/160-openebs-lvm.yaml.gotmpl delete mode 100644 functions/render/175-storageclass-lvm.yaml.gotmpl delete mode 100644 functions/render/180-storageclass-ebs.yaml.gotmpl diff --git a/README.md b/README.md index 44b574a..933ae04 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,22 @@ # psql-stack -PostgreSQL platform stack on top of [CloudNativePG](https://cloudnative-pg.io/). Installs the operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), Atlas Operator (schema migrations), and a tiered storage layer (OpenEBS Mayastor / OpenEBS LVM LocalPV / EBS gp3) on a target Kubernetes cluster. +PostgreSQL platform stack on top of [CloudNativePG](https://cloudnative-pg.io/) with [OpenEBS Mayastor](https://github.com/openebs/mayastor) for replicated NVMe-oF storage. Installs the operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), Atlas Operator (schema migrations), Mayastor + matching StorageClass + VolumeSnapshotClass, two Karpenter NodePools, and a node-prep DaemonSet. This is the **platform layer** — it does not create any serving Postgres clusters. Per-app DBs live in [`PSQLCluster`](../../psql-cluster/) (separate XR), ephemeral forks in [`PSQLBranch`](../../psql-branch/) (separate XR). ## Why psql-stack? **Without psql-stack:** -- Manual Helm installs of CNPG, Atlas, and any chosen storage engines on every cluster +- Manual Helm installs of CNPG, Atlas, Mayastor on every cluster - No deletion ordering — removing CNPG before Atlas can leave migrations dangling -- Inconsistent operator + plugin versions across environments +- No node-side prep for Mayastor (hugepages + `nvme-tcp` module need to exist before the IO engine pods will run) - No declarative substrate for the `cnpg-i-scale-to-zero` plugin (cert-manager-backed gRPC TLS, ServiceAccount + RBAC, sidecar config) **With psql-stack:** -- Single claim deploys CNPG + S2Z plugin + Atlas + storage backends with production defaults +- Single claim deploys CNPG + S2Z plugin + Atlas + Mayastor + StorageClass with production defaults - Deletion order enforced via `protection.crossplane.io/Usage` resources -- Three storage profiles available — replicated NVMe-oF (Mayastor), single-node CoW (LVM), durable EBS — each independently toggleable -- Optional dedicated Karpenter NodePools (branches sub-pool spot, primary sub-pool on-demand) targeting NVMe instance-store nodes +- Replicated NVMe-oF CoW storage — `replicationFactor: 3` for primaries, `replicationFactor: 1` for ephemeral branches (per-PVC override). Single backend covers both uses. +- Dedicated Karpenter NodePools (`branches` spot, `primary` on-demand) targeting NVMe instance-store nodes (`i4g.2xlarge` / `i4g.4xlarge` / `im4gn.2xlarge` arm64 Graviton) - Pinnable upstream chart / plugin versions; Renovate keeps them current ## Components @@ -24,24 +24,22 @@ This is the **platform layer** — it does not create any serving Postgres clust | Component | Default | Purpose | |---|---|---| | **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | -| **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. Pinnable via `spec.scaleToZeroPlugin.version`. **Requires cert-manager** (provided by [`cert-manager-stack`](../cert-manager/) or `aws-external-dns-stack`). | +| **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. Pinnable via `spec.scaleToZeroPlugin.version`. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | | **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | -| **Karpenter NodePools** | off (`spec.nodePool.enabled: false`) | When enabled: `branches` sub-pool (spot arm64 NVMe) and `primary` sub-pool (on-demand arm64 NVMe). Targets `i4g.2xlarge` / `i4g.4xlarge` / `im4gn.2xlarge` by default. | -| **OpenEBS Mayastor** | off (`spec.storage.mayastor.enabled: false`) | Replicated NVMe-oF. Enterprise default for serving primaries — provides CoW snapshots + HA across N nodes. | -| **OpenEBS LVM LocalPV** | off (`spec.storage.lvm.enabled: false`) | Single-node CoW via LVM thin volumes. Cheaper than Mayastor; right for branches and dev. | -| **EBS gp3 StorageClass** | on (`spec.storage.ebs.enabled: true`) | Durable, no CoW. Always-on fallback profile. | +| **Karpenter NodePools** | on (`spec.nodePool.enabled: true`) | `branches` (spot arm64 NVMe) and `primary` (on-demand arm64 NVMe). | +| **OpenEBS Mayastor** | on when `nodePool.enabled` | Replicated NVMe-oF storage with CoW snapshots. Single `psql` StorageClass + matching VolumeSnapshotClass. | +| **node-prep DaemonSet** | on when `nodePool.enabled` | Configures hugepages + loads `nvme-tcp` kernel module on each NVMe node (Mayastor prereqs). | ## Prerequisites -- **cert-manager** must be installed on the target cluster (the S2Z plugin uses cert-manager `Issuer` + `Certificate`s for its gRPC TLS material). Install via [`cert-manager-stack`](../cert-manager/) — single claim, no AWS deps. -- **Karpenter + EKS Auto Mode** if using `spec.nodePool` (the default `nodeClassName: default` references the EKS Auto Mode managed `NodeClass`). -- **Mayastor + LVM node-side prep** (hugepages, `nvme-tcp` kernel module, LVM volume group on instance-store NVMe) — currently a manual / out-of-stack concern. Phase 3b of the CNPG pivot will add a node-prep DaemonSet template; until it lands, leave `spec.storage.mayastor.enabled` and `spec.storage.lvm.enabled` at `false` unless you've prepped nodes yourself. +- **cert-manager** must be installed on the target cluster (the S2Z plugin uses cert-manager `Issuer` + `Certificate`s for its gRPC TLS material). Install via [`aws-cert-stack`](../../aws/cert/). +- **Karpenter + EKS Auto Mode** for the NodePools (the default `nodeClassName: default` references Auto Mode's managed `NodeClass`). ## The Journey -### Stage 1: Getting Started +### Stage 1: Default install -Deploy the platform on a single cluster with EBS-only storage. No CoW, no NodePool — just CNPG + S2Z plugin + Atlas + a `psql-ebs` StorageClass. +Deploy with all defaults: CNPG + Atlas + S2Z plugin + Mayastor + dedicated NodePools. Karpenter provisions NVMe instance-store nodes when something requests them. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -53,11 +51,9 @@ spec: clusterName: my-cluster ``` -`PSQLCluster` resources targeting this stack pick `psql-ebs` as their storage class and use `pg_basebackup`-style branching (works on EBS; takes minutes for non-trivial DBs). +### Stage 2: Production sizing -### Stage 2: Adding Local CoW - -Enable LVM LocalPV for fast single-node CoW branches without committing to replicated storage. Useful for dev clusters and preview-environment workflows where node loss is acceptable. +Tune NodePool limits, override Helm values, label for cost allocation. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -66,22 +62,27 @@ metadata: name: psql namespace: default spec: - clusterName: dev-cluster + clusterName: production-cluster + namespace: cnpg-system + labels: + team: platform nodePool: enabled: true + branches: + limits: { cpu: "32", memory: "128Gi" } primary: - enabled: false # branches-only sub-pool for dev + limits: { cpu: "64", memory: "256Gi" } storage: - lvm: - enabled: true - volumeGroup: psql-vg + replicationFactor: 3 + thin: true + atlasOperator: + values: + prewarmDevDB: true ``` -`PSQLBranch` resources can now reference `psql-lvm` for instant CoW forks via VolumeSnapshot. - -### Stage 3: Production HA + Replicated CoW +### Stage 3: Local / no-NVMe clusters -Enable Mayastor for replicated NVMe-oF storage. Primary serving DBs get CoW + HA replication across the on-demand NodePool; branches stay on LVM (cheaper). +For kind / k3d / clusters that can't run Mayastor, disable the NodePool. The stack then ships only CNPG + Atlas + S2Z plugin, and your PSQLClusters target whatever StorageClass the cluster provides. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -90,31 +91,11 @@ metadata: name: psql namespace: default spec: - clusterName: production-cluster - namespace: cnpg-system - labels: - team: platform + clusterName: local + helmProviderConfigRef: + name: default nodePool: - enabled: true - branches: - enabled: true - limits: { cpu: "32", memory: "128Gi" } - primary: - enabled: true - limits: { cpu: "64", memory: "256Gi" } - storage: - mayastor: - enabled: true - replicationFactor: 3 - lvm: - enabled: true - ebs: - enabled: true - scaleToZeroPlugin: - enabled: true - atlasOperator: - values: - prewarmDevDB: true + enabled: false ``` ## Spec Reference @@ -144,7 +125,7 @@ spec: | `atlasOperator.values` | object | — | Helm values merged with chart defaults | | `atlasOperator.overrideAllValues` | object | — | Helm values that replace all defaults | | **NodePool** | | | | -| `nodePool.enabled` | bool | `false` | Master toggle for both sub-pools | +| `nodePool.enabled` | bool | `true` | Master toggle. When false, Mayastor + StorageClass + node-prep are skipped too. | | `nodePool.nodeClassName` | string | `default` | EKS NodeClass referenced by both sub-pools | | `nodePool.disruption.consolidationPolicy` | enum | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | | `nodePool.disruption.consolidateAfter` | string | `60s` | Consolidation delay | @@ -154,29 +135,18 @@ spec: | `nodePool.primary.enabled` | bool | `true` | On-demand arm64 NVMe sub-pool | | `nodePool.primary.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | | `nodePool.primary.requirements` | array | arm64 on-demand `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | -| **Storage: Mayastor** | | | | -| `storage.mayastor.enabled` | bool | `false` | Install OpenEBS Mayastor + create `psql-mayastor` SC | -| `storage.mayastor.chartVersion` | string | `2.10.0` | Helm chart version | -| `storage.mayastor.storageClassName` | string | `psql-mayastor` | StorageClass name | -| `storage.mayastor.replicationFactor` | int | `3` | Replicas per volume | -| `storage.mayastor.thin` | bool | `true` | Thin provisioning (CoW) | -| `storage.mayastor.values` | object | — | Helm values merged with chart defaults | -| `storage.mayastor.overrideAllValues` | object | — | Helm values that replace all defaults | -| **Storage: LVM** | | | | -| `storage.lvm.enabled` | bool | `false` | Install OpenEBS LVM LocalPV + create `psql-lvm` SC | -| `storage.lvm.chartVersion` | string | `1.7.0` | Helm chart version | -| `storage.lvm.storageClassName` | string | `psql-lvm` | StorageClass name | -| `storage.lvm.volumeGroup` | string | `psql-vg` | LVM Volume Group on each node | -| `storage.lvm.values` | object | — | Helm values merged with chart defaults | -| `storage.lvm.overrideAllValues` | object | — | Helm values that replace all defaults | -| **Storage: EBS** | | | | -| `storage.ebs.enabled` | bool | `true` | Create the `psql-ebs` SC | -| `storage.ebs.storageClassName` | string | `psql-ebs` | StorageClass name | -| `storage.ebs.provisioner` | string | `ebs.csi.eks.amazonaws.com` | CSI provisioner | -| `storage.ebs.parameters` | object | `{type: gp3, fsType: ext4}` | Provisioner parameters | -| `storage.ebs.reclaimPolicy` | enum | `Delete` | `Delete` or `Retain` | -| `storage.ebs.volumeBindingMode` | enum | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | -| `storage.ebs.allowVolumeExpansion` | bool | `true` | Allow PVC resize | +| **Storage (Mayastor)** | | | | +| `storage.chartVersion` | string | `2.10.0` | Mayastor Helm chart version | +| `storage.namespace` | string | `mayastor` | Helm release namespace | +| `storage.storageClassName` | string | `psql` | StorageClass name | +| `storage.replicationFactor` | int | `3` | Default replicas per volume (per-PVC override via parameters) | +| `storage.thin` | bool | `true` | Thin-provisioning (CoW) | +| `storage.values` | object | — | Helm values merged with chart defaults | +| `storage.overrideAllValues` | object | — | Helm values that replace all defaults | +| **Node prep** | | | | +| `nodePrep.enabled` | bool | `true` (when `nodePool.enabled`) | Compose the privileged DaemonSet | +| `nodePrep.hugepages.count` | int | `1024` | Number of 2MiB hugepages per node | +| `nodePrep.image` | string | `alpine:3.20` | Init container image (apk-install at runtime) | ## Composed Resources @@ -185,15 +155,12 @@ spec: | `cloudnative-pg` | `helm.m.crossplane.io/Release` | always | | `atlas-operator` | `helm.m.crossplane.io/Release` | always | | 9× `-s2z-*` | `kubernetes.m.crossplane.io/Object` | `scaleToZeroPlugin.enabled: true` (default) | -| `-storageclass-ebs` | `kubernetes.m.crossplane.io/Object` | `storage.ebs.enabled: true` (default) | -| `openebs-lvm` | `helm.m.crossplane.io/Release` | `storage.lvm.enabled: true` | -| `-storageclass-lvm` | `kubernetes.m.crossplane.io/Object` | `storage.lvm.enabled: true` | -| `-volumesnapshotclass-lvm` | `kubernetes.m.crossplane.io/Object` | `storage.lvm.enabled: true` | -| `openebs-mayastor` | `helm.m.crossplane.io/Release` | `storage.mayastor.enabled: true` | -| `-storageclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `storage.mayastor.enabled: true` | -| `-volumesnapshotclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `storage.mayastor.enabled: true` | -| `-nodepool-branches` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true && nodePool.branches.enabled: true` | -| `-nodepool-primary` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true && nodePool.primary.enabled: true` | +| `openebs-mayastor` | `helm.m.crossplane.io/Release` | `nodePool.enabled: true` (default) | +| `-storageclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | +| `-volumesnapshotclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | +| `-nodepool-branches` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled && nodePool.branches.enabled` | +| `-nodepool-primary` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled && nodePool.primary.enabled` | +| `-node-prep` | `kubernetes.m.crossplane.io/Object` (DaemonSet) | `nodePool.enabled && nodePrep.enabled` | | Various `Usage` | `protection.crossplane.io/Usage` | when both ends Ready (deletion ordering) | ## Dependencies diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 46a454f..69bae98 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -67,124 +67,51 @@ spec: description: Shared namespace for CNPG operator, scale-to-zero plugin, and Atlas operator. Defaults to cnpg-system. Per-component namespace overrides this. type: string storage: - description: "Three-profile storage layer. mayastor = replicated NVMe-oF (CoW + HA), lvm = single-node LVM thin clones (CoW, cheap), ebs = EBS gp3 (durable, no CoW). Each profile is independently toggleable. Mayastor + LVM require node-side prerequisites (hugepages, nvme-tcp module, NVMe instance-store devices, LVM volume groups) — these are provided by the NodePool config in phase 3 of the pivot. Until phase 3 lands, leave mayastor.enabled and lvm.enabled at false on clusters that don't already have those prereqs." + description: "Storage backend (OpenEBS Mayastor — replicated NVMe-oF with CoW snapshots). The stack is opinionated: it ships exactly one storage backend, sized for CoW + HA. Per-cluster cost is controlled via replicationFactor (3 for primaries, 1 for ephemeral branches). Requires NVMe instance-store nodes (provided by the NodePool block) plus hugepages and the nvme-tcp kernel module (provided by the node-prep DaemonSet). For non-CoW workloads, PSQLCluster claims can target any other StorageClass that exists on the target cluster (e.g., the cluster's default gp3 SC) — no need for the stack to compose that." type: object properties: - mayastor: - description: OpenEBS Mayastor (Replicated NVMe-oF) — enterprise default for primary serving clusters. Provides CoW snapshots + HA replication across NVMe nodes. - type: object - properties: - enabled: - type: boolean - default: false - chartVersion: - description: Mayastor Helm chart version. Defaults to 2.10.0. - type: string - namespace: - description: Namespace override for the Mayastor install. Defaults to mayastor. - type: string - storageClassName: - description: StorageClass name. Defaults to psql-mayastor. - type: string - replicationFactor: - description: Number of replicas per volume. Defaults to 3 (quorum-safe HA). - type: integer - default: 3 - thin: - description: Thin-provision volumes (CoW). Defaults to true. - type: boolean - default: true - reclaimPolicy: - type: string - enum: [Delete, Retain] - default: Delete - volumeBindingMode: - type: string - enum: [Immediate, WaitForFirstConsumer] - default: WaitForFirstConsumer - allowVolumeExpansion: - type: boolean - default: true - values: - type: object - x-kubernetes-preserve-unknown-fields: true - overrideAllValues: - type: object - x-kubernetes-preserve-unknown-fields: true - lvm: - description: OpenEBS LVM LocalPV — single-node CoW via LVM thin volumes. Cheaper than Mayastor; appropriate for branches and dev clusters. + chartVersion: + description: Mayastor Helm chart version. Defaults to 2.10.0. + type: string + namespace: + description: Namespace for the Mayastor install. Defaults to mayastor. + type: string + storageClassName: + description: StorageClass name. Defaults to "psql". + type: string + replicationFactor: + description: Default replication factor. Defaults to 3 (quorum-safe HA). Per-PVC override via the StorageClass parameters. + type: integer + default: 3 + thin: + description: Thin-provision volumes (CoW). Defaults to true. + type: boolean + default: true + reclaimPolicy: + type: string + enum: [Delete, Retain] + default: Delete + volumeBindingMode: + type: string + enum: [Immediate, WaitForFirstConsumer] + default: WaitForFirstConsumer + allowVolumeExpansion: + type: boolean + default: true + values: type: object - properties: - enabled: - type: boolean - default: false - chartVersion: - description: lvm-localpv Helm chart version. Defaults to 1.7.0. - type: string - namespace: - description: Namespace override. Defaults to openebs. - type: string - storageClassName: - description: StorageClass name. Defaults to psql-lvm. - type: string - volumeGroup: - description: LVM Volume Group name configured on each NVMe node. Defaults to psql-vg. - type: string - reclaimPolicy: - type: string - enum: [Delete, Retain] - default: Delete - volumeBindingMode: - type: string - enum: [Immediate, WaitForFirstConsumer] - default: WaitForFirstConsumer - allowVolumeExpansion: - type: boolean - default: true - values: - type: object - x-kubernetes-preserve-unknown-fields: true - overrideAllValues: - type: object - x-kubernetes-preserve-unknown-fields: true - ebs: - description: EBS gp3 via the EKS Auto Mode CSI driver. Durable but no CoW; pg_basebackup-only branching. Always-on fallback profile. + x-kubernetes-preserve-unknown-fields: true + overrideAllValues: type: object - properties: - enabled: - type: boolean - default: true - storageClassName: - description: StorageClass name. Defaults to psql-ebs. - type: string - provisioner: - description: CSI provisioner. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). - type: string - parameters: - description: Provisioner parameters. Defaults to gp3 + ext4. - type: object - additionalProperties: - type: string - x-kubernetes-preserve-unknown-fields: true - reclaimPolicy: - type: string - enum: [Delete, Retain] - default: Delete - volumeBindingMode: - type: string - enum: [Immediate, WaitForFirstConsumer] - default: WaitForFirstConsumer - allowVolumeExpansion: - type: boolean - default: true + x-kubernetes-preserve-unknown-fields: true nodePool: description: "Dedicated Karpenter NodePools for PostgreSQL workloads. Two sub-pools — `branches` (spot arm64 NVMe, for ephemeral PSQLBranch workloads) and `primary` (on-demand arm64 NVMe, for replicated-cow PSQLCluster primaries where spot preemption would lose a Mayastor replica). Defaults to disabled — node-side prep for Mayastor/LVM (hugepages, nvme-tcp, LVM volume groups) is a separate concern handled in phase 3b. CNPG / scale-to-zero / Atlas schedule on the primary pool when enabled." type: object properties: enabled: - description: Master toggle. When false, neither sub-pool is created. When true, each sub-pool's `enabled` controls individual creation. + description: Master toggle. When false, neither sub-pool is created and Mayastor / node-prep are skipped (the stack would have nothing to schedule on otherwise). Defaults to true — provisioning NVMe nodes is the stack's whole point. type: boolean - default: false + default: true nodeClassName: description: EKS NodeClass shared by both sub-pools. Defaults to "default". type: string @@ -260,7 +187,7 @@ spec: type: string required: [key, operator] nodePrep: - description: "Node-prep DaemonSet for Mayastor / LVM prerequisites. Runs as a privileged init container on each NVMe NodePool node and configures hugepages (vm.nr_hugepages, required by Mayastor SPDK), loads the nvme-tcp kernel module (Mayastor NVMe-oF transport), and creates the LVM volume group on the first instance-store NVMe device (OpenEBS LVM LocalPV). Composed only when nodePool.enabled AND (storage.mayastor.enabled OR storage.lvm.enabled). EKS Auto Mode / Bottlerocket may pre-load nvme-tcp; verify with `lsmod | grep nvme_tcp` before assuming the modprobe step is needed." + description: "Node-prep DaemonSet for Mayastor prerequisites. Runs as a privileged init container on each NVMe NodePool node and configures hugepages (vm.nr_hugepages, required by Mayastor SPDK) plus loads the nvme-tcp kernel module (Mayastor's NVMe-oF transport). Composed when nodePool.enabled. EKS Auto Mode / Bottlerocket may pre-load nvme-tcp; verify with `lsmod | grep nvme_tcp` before assuming the modprobe step is needed." type: object properties: enabled: diff --git a/examples/psqlstacks/local.yaml b/examples/psqlstacks/local.yaml index a588752..3f42a1e 100644 --- a/examples/psqlstacks/local.yaml +++ b/examples/psqlstacks/local.yaml @@ -1,9 +1,9 @@ -# Local: single-node CoW with LVM, no Mayastor (replication needs >1 node), -# no dedicated NodePool. Useful for kind / k3d dev clusters where you want -# fast branching via LVM thin clones but don't need replicated storage. +# Local: NodePool / Mayastor disabled — gets just CNPG + Atlas + S2Z plugin +# installed on whatever StorageClass already exists on the local cluster. # -# Requires LVM tools + a `psql-vg` volume group on the local node before -# psql-lvm PVCs can bind. +# Useful for kind / k3d dev clusters that can't run Mayastor. +# PSQLClusters on this stack will need to specify a non-Mayastor SC +# (`spec.storage.class: standard` for k3d, etc.). # apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack @@ -14,11 +14,5 @@ spec: clusterName: local helmProviderConfigRef: name: default - storage: - lvm: - enabled: true - volumeGroup: psql-vg - ebs: - enabled: false - mayastor: - enabled: false + nodePool: + enabled: false diff --git a/examples/psqlstacks/minimal.yaml b/examples/psqlstacks/minimal.yaml index aebd1c5..324b300 100644 --- a/examples/psqlstacks/minimal.yaml +++ b/examples/psqlstacks/minimal.yaml @@ -1,13 +1,18 @@ -# Minimal: just the platform operators on a target cluster. +# Minimal: defaults across the board. # # Composes: # - CloudNativePG operator (Helm) -# - cnpg-i-scale-to-zero plugin (set of Objects, default-on) +# - cnpg-i-scale-to-zero plugin (set of Objects) # - Atlas operator (Helm) -# - psql-ebs StorageClass (default-on, EBS gp3) +# - Karpenter NodePool sub-pools: branches (spot) + primary (on-demand) +# - OpenEBS Mayastor (Helm) on the primary sub-pool nodes +# - psql StorageClass + VolumeSnapshotClass (Mayastor-backed, replicated, CoW) +# - node-prep DaemonSet (hugepages + nvme-tcp on each NVMe node) # -# No NodePool, no Mayastor, no LVM. Suitable for clusters that just need -# CNPG with EBS-backed storage and pg_basebackup-only branching. +# This is the stack's whole point: CoW Postgres on dedicated NVMe nodes. +# To opt out of NVMe / Mayastor entirely (and just get CNPG + Atlas + S2Z +# on the cluster's existing default StorageClass), set nodePool.enabled: false +# and have your PSQLClusters target a different SC. # apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index e9ad5f0..73d8a09 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -1,19 +1,5 @@ -# Standard: production posture with Mayastor + LVM CoW storage and dedicated -# NVMe NodePools. -# -# Composes: -# - Karpenter NodePool sub-pools: branches (spot) and primary (on-demand) -# on i4g/im4gn arm64 NVMe instance-store nodes -# - OpenEBS LVM LocalPV (single-node CoW for branches) -# - OpenEBS Mayastor (replicated NVMe-oF for serving primaries) -# - Three StorageClasses: psql-mayastor, psql-lvm, psql-ebs -# - VolumeSnapshotClasses for the two CoW backends -# - CloudNativePG operator (scheduled on the primary NodePool) -# - cnpg-i-scale-to-zero plugin -# - Atlas operator -# -# Requires phase 3b node-prep on each NVMe node (hugepages, nvme-tcp module, -# LVM volume group on instance-store) before Mayastor / LVM PVCs can bind. +# Standard: production posture with explicit overrides for sizing, +# scheduling, and chart values. # apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack @@ -32,17 +18,10 @@ spec: limits: { cpu: "32", memory: "128Gi" } primary: enabled: true - limits: { cpu: "32", memory: "128Gi" } + limits: { cpu: "64", memory: "256Gi" } storage: - mayastor: - enabled: true - replicationFactor: 3 - thin: true - lvm: - enabled: true - volumeGroup: psql-vg - ebs: - enabled: true + replicationFactor: 3 + thin: true scaleToZeroPlugin: enabled: true atlasOperator: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index fe3a0b0..d4bd4d0 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -37,67 +37,33 @@ }} # ============================================================================== -# Storage — three-profile model (mayastor / lvm / ebs) -# Each profile is independently toggleable. Mayastor and LVM require node-side -# prerequisites (hugepages, nvme-tcp module, NVMe instance-store devices, -# LVM volume groups) provided by phase 3's NodePool config — keep them -# disabled until those prereqs land. EBS is always-safe as a fallback. +# Storage — OpenEBS Mayastor (replicated NVMe-oF + CoW) +# Single backend; per-PVC cost is controlled via replicationFactor. Requires +# NVMe instance-store nodes (NodePool block) plus hugepages + nvme-tcp module +# (node-prep DaemonSet). PSQLClusters that don't want CoW can target any other +# StorageClass that exists on the cluster (e.g., the default gp3 SC) — the +# stack doesn't compose a non-CoW SC itself. # ============================================================================== -{{- $storage := $spec.storage | default dict }} - -# Mayastor — replicated NVMe-oF (HA + CoW) -{{- $mayastorSpec := $storage.mayastor | default dict }} -{{- $mayastorEnabled := false }} -{{- if hasKey $mayastorSpec "enabled" }} - {{- $mayastorEnabled = $mayastorSpec.enabled }} +{{- $storageSpec := $spec.storage | default dict }} +{{- $thin := true }} +{{- if hasKey $storageSpec "thin" }} + {{- $thin = $storageSpec.thin }} {{- end }} -{{- $mayastor := dict - "enabled" $mayastorEnabled - "chartVersion" ($mayastorSpec.chartVersion | default "2.10.0") - "namespace" ($mayastorSpec.namespace | default "mayastor") - "storageClassName" ($mayastorSpec.storageClassName | default "psql-mayastor") - "replicationFactor" ($mayastorSpec.replicationFactor | default 3) - "thin" (or (eq (toString ($mayastorSpec.thin | default true)) "true") false) - "reclaimPolicy" ($mayastorSpec.reclaimPolicy | default "Delete") - "volumeBindingMode" ($mayastorSpec.volumeBindingMode | default "WaitForFirstConsumer") - "allowVolumeExpansion" (or (eq (toString ($mayastorSpec.allowVolumeExpansion | default true)) "true") false) - "values" ($mayastorSpec.values | default dict) - "overrideAllValues" ($mayastorSpec.overrideAllValues | default dict) -}} - -# LVM LocalPV — single-node CoW via LVM thin clones -{{- $lvmSpec := $storage.lvm | default dict }} -{{- $lvmEnabled := false }} -{{- if hasKey $lvmSpec "enabled" }} - {{- $lvmEnabled = $lvmSpec.enabled }} +{{- $allowVolumeExpansion := true }} +{{- if hasKey $storageSpec "allowVolumeExpansion" }} + {{- $allowVolumeExpansion = $storageSpec.allowVolumeExpansion }} {{- end }} -{{- $lvm := dict - "enabled" $lvmEnabled - "chartVersion" ($lvmSpec.chartVersion | default "1.7.0") - "namespace" ($lvmSpec.namespace | default "openebs") - "storageClassName" ($lvmSpec.storageClassName | default "psql-lvm") - "volumeGroup" ($lvmSpec.volumeGroup | default "psql-vg") - "reclaimPolicy" ($lvmSpec.reclaimPolicy | default "Delete") - "volumeBindingMode" ($lvmSpec.volumeBindingMode | default "WaitForFirstConsumer") - "allowVolumeExpansion" (or (eq (toString ($lvmSpec.allowVolumeExpansion | default true)) "true") false) - "values" ($lvmSpec.values | default dict) - "overrideAllValues" ($lvmSpec.overrideAllValues | default dict) -}} - -# EBS gp3 (EKS Auto Mode CSI) — durable fallback, default on -{{- $ebsSpec := $storage.ebs | default dict }} -{{- $ebsEnabled := true }} -{{- if hasKey $ebsSpec "enabled" }} - {{- $ebsEnabled = $ebsSpec.enabled }} -{{- end }} -{{- $ebs := dict - "enabled" $ebsEnabled - "storageClassName" ($ebsSpec.storageClassName | default "psql-ebs") - "provisioner" ($ebsSpec.provisioner | default "ebs.csi.eks.amazonaws.com") - "parameters" ($ebsSpec.parameters | default (dict "type" "gp3" "fsType" "ext4")) - "reclaimPolicy" ($ebsSpec.reclaimPolicy | default "Delete") - "volumeBindingMode" ($ebsSpec.volumeBindingMode | default "WaitForFirstConsumer") - "allowVolumeExpansion" (or (eq (toString ($ebsSpec.allowVolumeExpansion | default true)) "true") false) +{{- $storage := dict + "chartVersion" ($storageSpec.chartVersion | default "2.10.0") + "namespace" ($storageSpec.namespace | default "mayastor") + "storageClassName" ($storageSpec.storageClassName | default "psql") + "replicationFactor" ($storageSpec.replicationFactor | default 3) + "thin" $thin + "reclaimPolicy" ($storageSpec.reclaimPolicy | default "Delete") + "volumeBindingMode" ($storageSpec.volumeBindingMode | default "WaitForFirstConsumer") + "allowVolumeExpansion" $allowVolumeExpansion + "values" ($storageSpec.values | default dict) + "overrideAllValues" ($storageSpec.overrideAllValues | default dict) }} # ============================================================================== @@ -111,7 +77,7 @@ # can target one pool specifically. # ============================================================================== {{- $nodePoolSpec := $spec.nodePool | default dict }} -{{- $nodePoolEnabled := false }} +{{- $nodePoolEnabled := true }} {{- if hasKey $nodePoolSpec "enabled" }} {{- $nodePoolEnabled = $nodePoolSpec.enabled }} {{- end }} @@ -160,17 +126,16 @@ {{- end }} # ============================================================================== -# Node prep — hugepages, nvme-tcp module, LVM volume group on NVMe nodes -# Auto-gated: only emitted when nodePool.enabled AND (mayastor || lvm enabled) +# Node prep — hugepages + nvme-tcp module on NVMe nodes (Mayastor prereqs) +# Auto-gated: emitted when nodePool.enabled (no nodes → DS no-op anyway) # ============================================================================== {{- $nodePrepSpec := $spec.nodePrep | default dict }} {{- $nodePrepEnabled := true }} {{- if hasKey $nodePrepSpec "enabled" }} {{- $nodePrepEnabled = $nodePrepSpec.enabled }} {{- end }} -{{- $nodePrepNeeded := and $nodePoolEnabled (or $mayastorEnabled $lvmEnabled) }} {{- $nodePrep := dict - "enabled" (and $nodePrepEnabled $nodePrepNeeded) + "enabled" (and $nodePrepEnabled $nodePoolEnabled) "hugepages" (dict "count" (((($nodePrepSpec.hugepages | default dict)).count) | default 1024) ) @@ -199,11 +164,7 @@ "labels" $labels "helmProviderConfigRef" $helmProviderConfigRef "kubernetesProviderConfigRef" $k8sProviderConfigRef - "storage" (dict - "mayastor" $mayastor - "lvm" $lvm - "ebs" $ebs - ) + "storage" $storage "nodePool" (dict "enabled" $nodePoolEnabled "nodeClassName" $nodePoolNodeClassName diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index 217498e..d393811 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "node-prep" "openebs-lvm" "openebs-mayastor" "storageclass-mayastor" "storageclass-lvm" "storageclass-ebs" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "node-prep" "openebs-mayastor" "storageclass-mayastor" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -33,11 +33,8 @@ "nodepoolBranches" (dict "ready" (get $checkReady "nodepool-branches")) "nodepoolPrimary" (dict "ready" (get $checkReady "nodepool-primary")) "nodePrep" (dict "ready" (get $checkReady "node-prep")) - "openebsLvm" (dict "ready" (get $checkReady "openebs-lvm")) - "openebsMayastor" (dict "ready" (get $checkReady "openebs-mayastor")) - "storageClassMayastor" (dict "ready" (get $checkReady "storageclass-mayastor")) - "storageClassLvm" (dict "ready" (get $checkReady "storageclass-lvm")) - "storageClassEbs" (dict "ready" (get $checkReady "storageclass-ebs")) + "mayastor" (dict "ready" (get $checkReady "openebs-mayastor")) + "storageClass" (dict "ready" (get $checkReady "storageclass-mayastor")) ) }} # ============================================================================== diff --git a/functions/render/155-node-prep-daemonset.yaml.gotmpl b/functions/render/155-node-prep-daemonset.yaml.gotmpl index e25f66e..529be5f 100644 --- a/functions/render/155-node-prep-daemonset.yaml.gotmpl +++ b/functions/render/155-node-prep-daemonset.yaml.gotmpl @@ -1,29 +1,23 @@ # code: language=yaml # -# Node-prep DaemonSet for Mayastor / LVM prerequisites +# Node-prep DaemonSet for Mayastor prerequisites # # Runs as a privileged init container on each NVMe NodePool node and -# configures the host-level state that storage backends require: +# configures the host-level state Mayastor requires: # - Hugepages (vm.nr_hugepages, required by Mayastor SPDK) # - nvme-tcp kernel module (Mayastor NVMe-oF transport) -# - LVM volume group on the first instance-store NVMe device -# (OpenEBS LVM LocalPV expects the VG to pre-exist) # # After init, a tiny pause container keeps the pod up so the DaemonSet # is reported Ready by Kubernetes. # -# Auto-gated: composed only when nodePool.enabled AND -# (storage.mayastor.enabled OR storage.lvm.enabled). Hugepages + nvme-tcp -# only run when mayastor is enabled; LVM VG only when lvm is enabled. +# Auto-gated: composed when nodePool.enabled (otherwise no nodes match +# the workload-type=psql selector and the DS is a no-op anyway). # # Caveats: # - On EKS Auto Mode (Bottlerocket OS) `nvme-tcp` may already be # pre-loaded; verify with `lsmod | grep nvme_tcp` before assuming # this step is needed. modprobe inside a container may also be # restricted on Bottlerocket — script falls back gracefully. -# - Instance-store NVMe device detection uses lsblk model match for -# "Amazon EC2 NVMe Instance Storage". Manual override possible -# by pre-creating the volume group on the host. # {{- if $state.nodePrep.enabled }} @@ -79,12 +73,7 @@ spec: - -c - | set -e - # Install lvm + util-linux if not already present (alpine default) - if command -v apk >/dev/null 2>&1; then - apk add --no-cache lvm2 util-linux - fi - {{- if $state.storage.mayastor.enabled }} # ---- Hugepages (Mayastor SPDK requires) ---- echo "Setting vm.nr_hugepages={{ $state.nodePrep.hugepages.count }}" if [ -w /proc/sys/vm/nr_hugepages ]; then @@ -102,25 +91,6 @@ spec: || nsenter -t 1 -m modprobe nvme-tcp 2>/dev/null \ || echo "WARN: could not load nvme-tcp; verify on host (Bottlerocket may pre-load it)" fi - {{- end }} - - {{- if $state.storage.lvm.enabled }} - # ---- LVM volume group on instance-store NVMe ---- - VG_NAME='{{ $state.storage.lvm.volumeGroup }}' - if vgs "$VG_NAME" >/dev/null 2>&1; then - echo "Volume group $VG_NAME already exists" - else - DEVICE=$(lsblk -dno NAME,TYPE,MODEL 2>/dev/null \ - | awk '$2=="disk" && /Amazon EC2 NVMe Instance Storage/ {print "/dev/"$1; exit}') - if [ -z "$DEVICE" ]; then - echo "WARN: no instance-store NVMe device detected; LVM VG not created" - else - echo "Creating PV on $DEVICE and VG $VG_NAME" - pvcreate -f "$DEVICE" - vgcreate "$VG_NAME" "$DEVICE" - fi - fi - {{- end }} echo "node-prep complete" volumeMounts: diff --git a/functions/render/160-openebs-lvm.yaml.gotmpl b/functions/render/160-openebs-lvm.yaml.gotmpl deleted file mode 100644 index 55e861d..0000000 --- a/functions/render/160-openebs-lvm.yaml.gotmpl +++ /dev/null @@ -1,56 +0,0 @@ -# code: language=yaml -# -# Helm Release: OpenEBS LVM LocalPV -# -# Provides single-node CoW storage via LVM thin volumes. Cheaper than Mayastor -# but no replication — suitable for branches and dev clusters where node loss -# is acceptable. Requires an LVM Volume Group ($state.storage.lvm.volumeGroup) -# pre-configured on each NVMe node — phase 3 of the pivot wires this up via -# Karpenter NodePool userData. -# -# Chart: https://openebs.github.io/lvm-localpv (chart name: lvm-localpv) -# Upstream: https://github.com/openebs/lvm-localpv -# - -{{- $lvm := $state.storage.lvm }} -{{- if $lvm.enabled }} ---- -apiVersion: helm.m.crossplane.io/v1beta1 -kind: Release -metadata: - name: openebs-lvm - annotations: - {{ setResourceNameAnnotation "openebs-lvm" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - chart: - name: lvm-localpv - repository: https://openebs.github.io/lvm-localpv - version: {{ $lvm.chartVersion | quote }} - namespace: {{ $lvm.namespace }} - {{- if $lvm.overrideAllValues }} - values: - {{- toYaml $lvm.overrideAllValues | nindent 6 }} - {{- else }} - {{- $chartDefaults := dict }} - {{- if $state.nodePool.enabled }} - {{- $_ := set $chartDefaults "lvmNode" (dict - "nodeSelector" $state.nodePool.nodeSelector - "tolerations" $state.nodePool.tolerations - ) }} - {{- $_ := set $chartDefaults "lvmController" (dict - "nodeSelector" $state.nodePool.nodeSelector - "tolerations" $state.nodePool.tolerations - ) }} - {{- end }} - {{- $mergedValues := mergeOverwrite $chartDefaults ($lvm.values | default dict) }} - values: - {{- toYaml $mergedValues | nindent 6 }} - {{- end }} - rollbackLimit: 3 - providerConfigRef: - name: {{ $state.helmProviderConfigRef.name }} - kind: {{ $state.helmProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/165-openebs-mayastor.yaml.gotmpl b/functions/render/165-openebs-mayastor.yaml.gotmpl index 6fdcba5..112a5a6 100644 --- a/functions/render/165-openebs-mayastor.yaml.gotmpl +++ b/functions/render/165-openebs-mayastor.yaml.gotmpl @@ -15,8 +15,8 @@ # Upstream: https://github.com/openebs/mayastor # -{{- $mayastor := $state.storage.mayastor }} -{{- if $mayastor.enabled }} +{{- $storage := $state.storage }} +{{- if $state.nodePool.enabled }} --- apiVersion: helm.m.crossplane.io/v1beta1 kind: Release @@ -31,11 +31,11 @@ spec: chart: name: mayastor repository: https://openebs.github.io/mayastor-extensions - version: {{ $mayastor.chartVersion | quote }} - namespace: {{ $mayastor.namespace }} - {{- if $mayastor.overrideAllValues }} + version: {{ $storage.chartVersion | quote }} + namespace: {{ $storage.namespace }} + {{- if $storage.overrideAllValues }} values: - {{- toYaml $mayastor.overrideAllValues | nindent 6 }} + {{- toYaml $storage.overrideAllValues | nindent 6 }} {{- else }} {{- $chartDefaults := dict }} {{- if $state.nodePool.enabled }} @@ -44,7 +44,7 @@ spec: "tolerations" $state.nodePool.tolerations ) }} {{- end }} - {{- $mergedValues := mergeOverwrite $chartDefaults ($mayastor.values | default dict) }} + {{- $mergedValues := mergeOverwrite $chartDefaults ($storage.values | default dict) }} values: {{- toYaml $mergedValues | nindent 6 }} {{- end }} diff --git a/functions/render/170-storageclass-mayastor.yaml.gotmpl b/functions/render/170-storageclass-mayastor.yaml.gotmpl index b52097d..025f251 100644 --- a/functions/render/170-storageclass-mayastor.yaml.gotmpl +++ b/functions/render/170-storageclass-mayastor.yaml.gotmpl @@ -1,14 +1,14 @@ # code: language=yaml # -# StorageClass + VolumeSnapshotClass: psql-mayastor (replicated CoW) +# StorageClass + VolumeSnapshotClass — Mayastor (replicated CoW) # -# Created only when storage.mayastor.enabled. Enterprise default for primary -# PSQLClusters — replicated NVMe-oF + thin-provisioned + CSI snapshots for -# instant CoW branching via PSQLBranch. +# The stack's only StorageClass. Replicated NVMe-oF, thin-provisioned by +# default. PSQLClusters that don't want CoW can target a different SC the +# cluster already provides; the stack doesn't compose a non-CoW SC. # -{{- $mayastor := $state.storage.mayastor }} -{{- if $mayastor.enabled }} +{{- $storage := $state.storage }} +{{- if $state.nodePool.enabled }} --- apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object @@ -24,18 +24,18 @@ spec: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: - name: {{ $mayastor.storageClassName }} + name: {{ $storage.storageClassName }} labels: {{ $state.labels | toJson }} provisioner: io.openebs.csi-mayastor parameters: - repl: "{{ $mayastor.replicationFactor }}" + repl: "{{ $storage.replicationFactor }}" protocol: nvmf - thin: "{{ $mayastor.thin }}" + thin: "{{ $storage.thin }}" ioTimeout: "60" fsType: ext4 - reclaimPolicy: {{ $mayastor.reclaimPolicy }} - volumeBindingMode: {{ $mayastor.volumeBindingMode }} - allowVolumeExpansion: {{ $mayastor.allowVolumeExpansion }} + reclaimPolicy: {{ $storage.reclaimPolicy }} + volumeBindingMode: {{ $storage.volumeBindingMode }} + allowVolumeExpansion: {{ $storage.allowVolumeExpansion }} providerConfigRef: name: {{ $state.kubernetesProviderConfigRef.name }} kind: {{ $state.kubernetesProviderConfigRef.kind }} @@ -54,7 +54,7 @@ spec: apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: - name: {{ $mayastor.storageClassName }} + name: {{ $storage.storageClassName }} labels: {{ $state.labels | toJson }} driver: io.openebs.csi-mayastor deletionPolicy: Delete diff --git a/functions/render/175-storageclass-lvm.yaml.gotmpl b/functions/render/175-storageclass-lvm.yaml.gotmpl deleted file mode 100644 index f1de387..0000000 --- a/functions/render/175-storageclass-lvm.yaml.gotmpl +++ /dev/null @@ -1,65 +0,0 @@ -# code: language=yaml -# -# StorageClass + VolumeSnapshotClass: psql-lvm (single-node CoW) -# -# Created only when storage.lvm.enabled. LVM thin volumes provide CoW on a -# single node; suitable for branches and dev clusters where node-loss is -# tolerable. References the volume group set up by phase 3's NodePool config. -# - -{{- $lvm := $state.storage.lvm }} -{{- if $lvm.enabled }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-storageclass-lvm - annotations: - {{ setResourceNameAnnotation "storageclass-lvm" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: storage.k8s.io/v1 - kind: StorageClass - metadata: - name: {{ $lvm.storageClassName }} - labels: {{ $state.labels | toJson }} - provisioner: local.csi.openebs.io - parameters: - storage: lvm - volgroup: {{ $lvm.volumeGroup | quote }} - thinProvision: "yes" - fsType: ext4 - reclaimPolicy: {{ $lvm.reclaimPolicy }} - volumeBindingMode: {{ $lvm.volumeBindingMode }} - allowVolumeExpansion: {{ $lvm.allowVolumeExpansion }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-volumesnapshotclass-lvm - annotations: - {{ setResourceNameAnnotation "volumesnapshotclass-lvm" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: snapshot.storage.k8s.io/v1 - kind: VolumeSnapshotClass - metadata: - name: {{ $lvm.storageClassName }} - labels: {{ $state.labels | toJson }} - driver: local.csi.openebs.io - deletionPolicy: Delete - parameters: - snapSize: "1073741824" - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/180-storageclass-ebs.yaml.gotmpl b/functions/render/180-storageclass-ebs.yaml.gotmpl deleted file mode 100644 index c167f78..0000000 --- a/functions/render/180-storageclass-ebs.yaml.gotmpl +++ /dev/null @@ -1,39 +0,0 @@ -# code: language=yaml -# -# StorageClass: psql-ebs (durable EBS gp3, no CoW) -# -# Always-on fallback profile. Uses the EKS Auto Mode CSI driver -# (ebs.csi.eks.amazonaws.com). Branching against this profile uses -# pg_basebackup (no VolumeSnapshotClass — EBS snapshots are not CoW -# at restore, so they don't give the instant-fork semantics PSQLBranch -# expects from a snapshot path). -# - -{{- $ebs := $state.storage.ebs }} -{{- if $ebs.enabled }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-storageclass-ebs - annotations: - {{ setResourceNameAnnotation "storageclass-ebs" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: storage.k8s.io/v1 - kind: StorageClass - metadata: - name: {{ $ebs.storageClassName }} - labels: {{ $state.labels | toJson }} - provisioner: {{ $ebs.provisioner }} - parameters: {{ $ebs.parameters | toJson }} - reclaimPolicy: {{ $ebs.reclaimPolicy }} - volumeBindingMode: {{ $ebs.volumeBindingMode }} - allowVolumeExpansion: {{ $ebs.allowVolumeExpansion }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} diff --git a/tests/test-render/main.k b/tests/test-render/main.k index e3d44d9..4904fd9 100644 --- a/tests/test-render/main.k +++ b/tests/test-render/main.k @@ -12,13 +12,12 @@ import models.k8s.apimachinery.pkg.apis.meta.v1 as metav1 _items = [ # ========================================================================== - # Test 1: minimal claim renders the platform operators (CNPG, Atlas) and - # the scale-to-zero plugin install (default-on, gated on - # spec.scaleToZeroPlugin.enabled which defaults true). Storage defaults to - # ebs-only (mayastor + lvm default off until phase 3b lands node prep). + # Test 1: minimal claim renders the platform operators (CNPG, Atlas), the + # scale-to-zero plugin (default-on), Mayastor + StorageClass + node-prep + + # both NodePool sub-pools (nodePool default-on now — stack's whole point). # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "minimal-renders-platform-operators" + metadata.name = "minimal-renders-platform-stack" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" @@ -39,17 +38,40 @@ _items = [ kind = "Release" metadata.name = "atlas-operator" } - # scale-to-zero plugin Deployment Object (default enabled) + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "openebs-mayastor" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-storageclass-mayastor" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-volumesnapshotclass-mayastor" + } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" metadata.name = "test-psql-s2z-deployment" } - # ebs StorageClass Object (default enabled) { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "test-psql-storageclass-ebs" + metadata.name = "test-psql-nodepool-branches" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-nodepool-primary" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-node-prep" } ] } @@ -313,120 +335,80 @@ _items = [ { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "no-s2z-storageclass-ebs" + metadata.name = "no-s2z-storageclass-mayastor" } ] } } # ========================================================================== - # Test 9: storage.mayastor.enabled=true composes the mayastor Helm release - # and StorageClass Object + # Test 9: nodePool.enabled=false skips Mayastor, StorageClass, and node-prep + # (the stack's storage requires the NodePool to have something to schedule + # on — disabling the NodePool reasonably skips everything that needs it). # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "storage-mayastor-enabled-composes-helm-and-sc" + metadata.name = "nodepool-disabled-skips-storage" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" timeoutSeconds = 60 validate = False xr = stacksv1alpha1.PSQLStack { - metadata.name = "mayastor-on" + metadata.name = "np-off" spec = { clusterName = "my-cluster" - storage = { - mayastor = {enabled = True} - } + nodePool = {enabled = False} } } + # Positive presence: cnpg + atlas still composed assertResources = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "openebs-mayastor" - } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "mayastor-on-storageclass-mayastor" + metadata.name = "cloudnative-pg" } { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "mayastor-on-volumesnapshotclass-mayastor" + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "atlas-operator" } ] } } # ========================================================================== - # Test 10: storage.lvm.enabled=true composes the LVM LocalPV Helm release - # and StorageClass Object + # Test 10: spec.storage settings flow through to the StorageClass + # parameters (replicationFactor, thin) # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "storage-lvm-enabled-composes-helm-and-sc" + metadata.name = "storage-settings-flow-through-to-sc" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" timeoutSeconds = 60 validate = False xr = stacksv1alpha1.PSQLStack { - metadata.name = "lvm-on" + metadata.name = "storage-tweaks" spec = { clusterName = "my-cluster" storage = { - lvm = {enabled = True} + replicationFactor = 1 + thin = False } } } assertResources = [ - { - apiVersion = "helm.m.crossplane.io/v1beta1" - kind = "Release" - metadata.name = "openebs-lvm" - } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "lvm-on-storageclass-lvm" - } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "lvm-on-volumesnapshotclass-lvm" - } - ] - } - } - - # ========================================================================== - # Test 11: nodePool.enabled=true composes both branches and primary - # sub-pools by default (each can be individually disabled) - # ========================================================================== - metav1alpha1.CompositionTest { - metadata.name = "nodepool-enabled-composes-both-sub-pools" - spec = { - compositionPath = "apis/psqlstacks/composition.yaml" - xrdPath = "apis/psqlstacks/definition.yaml" - timeoutSeconds = 60 - validate = False - xr = stacksv1alpha1.PSQLStack { - metadata.name = "np-on" - spec = { - clusterName = "prod" - nodePool = {enabled = True} - } - } - assertResources = [ - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "np-on-nodepool-branches" - } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "np-on-nodepool-primary" + metadata.name = "storage-tweaks-storageclass-mayastor" + spec.forProvider.manifest.parameters = { + repl = "1" + protocol = "nvmf" + thin = "false" + ioTimeout = "60" + fsType = "ext4" + } } ] } From e242e9aeb2431e7930b7d3d9d3115eea5a6783ae Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Sat, 25 Apr 2026 08:41:14 -0500 Subject: [PATCH 13/44] nodepool + node-prep: spot primary, drop cluster prefix, DS creates DiskPools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes prompted by review: 1. Primary sub-pool default is now spot (was on-demand). Mayastor's replicationFactor=3 absorbs preemption — losing one replica triggers a rebuild on a fresh node, not data loss. Override to on-demand via spec.nodePool.primary.requirements when needed. 2. Karpenter NodePool inner names lose the cluster prefix. Was `-psql-{branches,primary}`, now `-{branches,primary}` (e.g. `psql-branches`, `psql-primary`). Less repetition; uses the stack's XR name for disambiguation when multiple PSQLStacks share a cluster (which is rare). Wrapper Crossplane Object names unchanged. 3. node-prep DaemonSet now also registers the local NVMe instance-store device with Mayastor by creating a per-node DiskPool CR. New ServiceAccount + ClusterRole/Binding granting get/create/list/watch on diskpools.openebs.io. Init script: detects /dev/nvme[1-9]n1 by lsblk model match, kubectl-applies a DiskPool named psql-pool- (idempotent — skip if already present). Closes the gap that left Mayastor pools empty and PVCs stuck Pending. This means no separate ObservedObjectCollection or custom function for DiskPool registration — the same DS that handles host-level prereqs (hugepages, nvme-tcp) also handles pool registration. One declarative artifact per node-prep concern. Also fixes a YAML colon issue in the primary sub-pool description. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- apis/psqlstacks/definition.yaml | 4 +- functions/render/000-state-init.yaml.gotmpl | 9 +- .../155-node-prep-daemonset.yaml.gotmpl | 149 ++++++++++++++++-- 3 files changed, 142 insertions(+), 20 deletions(-) diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 69bae98..3973285 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -157,7 +157,7 @@ spec: type: string required: [key, operator] primary: - description: On-demand arm64 NVMe sub-pool for replicated-cow PSQLCluster primaries. Spot preemption would lose a Mayastor replica, so on-demand is the right default for serving workloads. + description: "Spot arm64 NVMe sub-pool for PSQLCluster primaries. Mayastor's replicationFactor=3 absorbs spot preemption — losing one replica triggers a rebuild on a fresh node, not data loss. Override capacity-type to on-demand via spec.nodePool.primary.requirements if your durability requirements rule out preemption." type: object properties: enabled: @@ -171,7 +171,7 @@ spec: memory: type: string requirements: - description: Karpenter scheduling requirements. Defaults to arm64 on-demand on i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge (NVMe instance-store, Graviton). + description: Karpenter scheduling requirements. Defaults to arm64 spot on i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge (NVMe instance-store, Graviton). Set capacity-type to on-demand if Mayastor replication isn't enough for your durability bar. type: array items: type: object diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index d4bd4d0..fe5a00f 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -100,7 +100,8 @@ (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) ) }} -# primary sub-pool (on-demand) +# primary sub-pool (spot by default — Mayastor's replication absorbs preemption; +# losing one replica of three triggers a rebuild on a fresh node, not data loss) {{- $primarySpec := $nodePoolSpec.primary | default dict }} {{- $primaryEnabled := true }} {{- if hasKey $primarySpec "enabled" }} @@ -109,7 +110,7 @@ {{- $primaryLimits := $primarySpec.limits | default (dict "cpu" "32" "memory" "128Gi") }} {{- $primaryRequirements := $primarySpec.requirements | default (list (dict "key" "kubernetes.io/arch" "operator" "In" "values" (list "arm64")) - (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "on-demand")) + (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "spot")) (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) ) }} @@ -173,13 +174,13 @@ "tolerations" $nodePoolTolerations "branches" (dict "enabled" $branchesEnabled - "name" (printf "%s-psql-branches" $clusterName) + "name" (printf "%s-branches" $name) "limits" $branchesLimits "requirements" $branchesRequirements ) "primary" (dict "enabled" $primaryEnabled - "name" (printf "%s-psql-primary" $clusterName) + "name" (printf "%s-primary" $name) "limits" $primaryLimits "requirements" $primaryRequirements ) diff --git a/functions/render/155-node-prep-daemonset.yaml.gotmpl b/functions/render/155-node-prep-daemonset.yaml.gotmpl index 529be5f..0c3c231 100644 --- a/functions/render/155-node-prep-daemonset.yaml.gotmpl +++ b/functions/render/155-node-prep-daemonset.yaml.gotmpl @@ -2,25 +2,111 @@ # # Node-prep DaemonSet for Mayastor prerequisites # -# Runs as a privileged init container on each NVMe NodePool node and -# configures the host-level state Mayastor requires: -# - Hugepages (vm.nr_hugepages, required by Mayastor SPDK) -# - nvme-tcp kernel module (Mayastor NVMe-oF transport) +# Runs once per NVMe NodePool node and configures the host-level state +# Mayastor requires (hugepages, nvme-tcp), then registers the local NVMe +# instance-store device with Mayastor by creating a per-node DiskPool CR. # -# After init, a tiny pause container keeps the pod up so the DaemonSet -# is reported Ready by Kubernetes. +# Composes: +# - ServiceAccount + ClusterRole + ClusterRoleBinding granting the DS +# permission to get/create DiskPools in the mayastor namespace +# - DaemonSet that, per node: +# 1. Sysctls vm.nr_hugepages (required by Mayastor SPDK) +# 2. modprobes nvme-tcp (Mayastor NVMe-oF transport) +# 3. Detects /dev/nvme[1-9]n1 instance-store device by lsblk model +# 4. Creates a DiskPool CR `psql-pool-` if not present # -# Auto-gated: composed when nodePool.enabled (otherwise no nodes match -# the workload-type=psql selector and the DS is a no-op anyway). +# After the init container finishes, a tiny pause container keeps the +# pod up so the DaemonSet reports Ready. # # Caveats: -# - On EKS Auto Mode (Bottlerocket OS) `nvme-tcp` may already be -# pre-loaded; verify with `lsmod | grep nvme_tcp` before assuming -# this step is needed. modprobe inside a container may also be -# restricted on Bottlerocket — script falls back gracefully. +# - On EKS Auto Mode (Bottlerocket OS), nvme-tcp may already be +# pre-loaded; modprobe inside a container may be restricted — +# script falls back gracefully. +# - Auto Mode's NodeClass doesn't support userData, so runtime +# allocation via this DS is currently the only path for hugepages. # {{- if $state.nodePrep.enabled }} + +# ---- ServiceAccount for the DaemonSet ------------------------------------ +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-node-prep-sa + annotations: + {{ setResourceNameAnnotation "node-prep-sa" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: v1 + kind: ServiceAccount + metadata: + name: psql-node-prep + namespace: {{ $state.namespace }} + labels: {{ $state.labels | toJson }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# ---- ClusterRole: get/create DiskPools in mayastor namespace ------------- +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-node-prep-clusterrole + annotations: + {{ setResourceNameAnnotation "node-prep-clusterrole" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psql-node-prep + labels: {{ $state.labels | toJson }} + rules: + - apiGroups: ["openebs.io"] + resources: ["diskpools"] + verbs: ["get", "create", "list", "watch"] + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-node-prep-clusterrolebinding + annotations: + {{ setResourceNameAnnotation "node-prep-clusterrolebinding" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psql-node-prep + labels: {{ $state.labels | toJson }} + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psql-node-prep + subjects: + - kind: ServiceAccount + name: psql-node-prep + namespace: {{ $state.namespace }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} + +# ---- DaemonSet ----------------------------------------------------------- --- apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object @@ -52,6 +138,7 @@ spec: labels: app: psql-node-prep spec: + serviceAccountName: psql-node-prep hostPID: true hostNetwork: true nodeSelector: @@ -68,11 +155,22 @@ spec: image: {{ $state.nodePrep.image | quote }} securityContext: privileged: true + env: + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: MAYASTOR_NS + value: {{ $state.storage.namespace | quote }} command: - /bin/sh - -c - | set -e + # Toolchain (kubectl + lsblk via util-linux + curl for K8s API fallback) + if command -v apk >/dev/null 2>&1; then + apk add --no-cache util-linux curl kubectl + fi # ---- Hugepages (Mayastor SPDK requires) ---- echo "Setting vm.nr_hugepages={{ $state.nodePrep.hugepages.count }}" @@ -92,6 +190,31 @@ spec: || echo "WARN: could not load nvme-tcp; verify on host (Bottlerocket may pre-load it)" fi + # ---- Register Mayastor DiskPool for this node ---- + DEVICE=$(lsblk -dno NAME,TYPE,MODEL 2>/dev/null \ + | awk '$2=="disk" && /Amazon EC2 NVMe Instance Storage/ {print "/dev/"$1; exit}') + if [ -z "$DEVICE" ]; then + echo "WARN: no instance-store NVMe device on $NODE_NAME; skipping DiskPool" + else + POOL_NAME="psql-pool-${NODE_NAME}" + if kubectl get diskpool -n "$MAYASTOR_NS" "$POOL_NAME" >/dev/null 2>&1; then + echo "DiskPool $POOL_NAME already exists" + else + echo "Creating DiskPool $POOL_NAME on $NODE_NAME ($DEVICE)" + cat < Date: Sat, 25 Apr 2026 08:45:46 -0500 Subject: [PATCH 14/44] feat: stack-wide HA mode (replicaCount + topology spread by zone) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds spec.ha block: a single toggle that enables production-style HA defaults across every HA-able platform component without users needing to know each chart's specific values keys. When spec.ha.enabled: true (default false): - CNPG operator: replicaCount=3 + topologySpreadConstraints by zone - Atlas operator: same - cnpg-i-scale-to-zero plugin Deployment (directly composed): same - OpenEBS Mayastor: agents.core / csi.controller / etcd replicaCount=3 Per-component values can still override via the existing values block — HA values land in chartDefaults, user values mergeOverwrite them. Schema: - spec.ha.enabled (bool, default false) - spec.ha.replicas (int, default 3) - spec.ha.topologySpreadByZone (bool, default true) Standard example now demonstrates HA. New KCL test asserts replicaCount flows through to CNPG + Atlas Releases when ha.enabled=true. README updated with the new fields + a Components table entry. Render + validate clean (21 resources). 11/11 KCL tests pass. Implements [[tasks/psql-stack-cnpg]] Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 5 +++ apis/psqlstacks/definition.yaml | 16 +++++++++ examples/psqlstacks/standard.yaml | 4 +++ functions/render/000-state-init.yaml.gotmpl | 22 ++++++++++++ .../render/165-openebs-mayastor.yaml.gotmpl | 11 ++++++ .../render/200-cnpg-operator.yaml.gotmpl | 11 ++++++ .../render/210-cnpg-scale-to-zero.yaml.gotmpl | 11 +++++- .../render/220-atlas-operator.yaml.gotmpl | 11 ++++++ tests/test-render/main.k | 35 +++++++++++++++++++ 9 files changed, 125 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 933ae04..fea25c0 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ This is the **platform layer** — it does not create any serving Postgres clust | **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | | **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. Pinnable via `spec.scaleToZeroPlugin.version`. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | | **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | +| **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas of every HA-able platform component + `topologySpreadConstraints` by zone. Affects CNPG operator, Atlas, S2Z plugin, Mayastor control plane. | | **Karpenter NodePools** | on (`spec.nodePool.enabled: true`) | `branches` (spot arm64 NVMe) and `primary` (on-demand arm64 NVMe). | | **OpenEBS Mayastor** | on when `nodePool.enabled` | Replicated NVMe-oF storage with CoW snapshots. Single `psql` StorageClass + matching VolumeSnapshotClass. | | **node-prep DaemonSet** | on when `nodePool.enabled` | Configures hugepages + loads `nvme-tcp` kernel module on each NVMe node (Mayastor prereqs). | @@ -124,6 +125,10 @@ spec: | `atlasOperator.namespace` | string | shared `namespace` | Override | | `atlasOperator.values` | object | — | Helm values merged with chart defaults | | `atlasOperator.overrideAllValues` | object | — | Helm values that replace all defaults | +| **HA mode** | | | | +| `ha.enabled` | bool | `false` | Stack-wide HA toggle. When true, sets replicaCount + topology spread by zone on CNPG, Atlas, S2Z plugin, Mayastor control plane components. | +| `ha.replicas` | int | `3` | Replica count for HA-able platform components | +| `ha.topologySpreadByZone` | bool | `true` | Add topologySpreadConstraint with topologyKey=topology.kubernetes.io/zone, maxSkew=1, whenUnsatisfiable=ScheduleAnyway | | **NodePool** | | | | | `nodePool.enabled` | bool | `true` | Master toggle. When false, Mayastor + StorageClass + node-prep are skipped too. | | `nodePool.nodeClassName` | string | `default` | EKS NodeClass referenced by both sub-pools | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 3973285..2ac815f 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -204,6 +204,22 @@ spec: image: description: Container image for the prep init container. Must have apk (or be pre-baked with lvm2 + util-linux). Defaults to alpine:3.20 (apk-installs lvm2 at startup). type: string + ha: + description: "Stack-wide HA mode. When enabled, sets replicaCount + topology spread (one replica per zone) on all platform components: CNPG operator, Atlas operator, scale-to-zero plugin, Mayastor control plane. Per-component values can still be overridden via each component's values block." + type: object + properties: + enabled: + description: Master toggle. Default false. Flip true for production. + type: boolean + default: false + replicas: + description: Replica count for HA-able platform components. Defaults to 3. + type: integer + default: 3 + topologySpreadByZone: + description: Add a topologySpreadConstraint with topologyKey topology.kubernetes.io/zone, maxSkew 1, whenUnsatisfiable ScheduleAnyway. Defaults to true. + type: boolean + default: true cnpg: description: Configuration for the CloudNativePG operator Helm release. type: object diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index 73d8a09..d423ad5 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -22,6 +22,10 @@ spec: storage: replicationFactor: 3 thin: true + ha: + enabled: true # 3 replicas + topology spread by zone on + replicas: 3 # all platform components (CNPG, Atlas, + topologySpreadByZone: true # S2Z plugin, Mayastor control plane) scaleToZeroPlugin: enabled: true atlasOperator: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index fe5a00f..33aa234 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -143,6 +143,27 @@ "image" ($nodePrepSpec.image | default "alpine:3.20") }} +# ============================================================================== +# HA mode — stack-wide HA defaults injected into each platform component's +# Helm values (or our directly-composed Deployment specs) when enabled. +# Per-component overrides can still win via the component's values block. +# ============================================================================== +{{- $haSpec := $spec.ha | default dict }} +{{- $haEnabled := false }} +{{- if hasKey $haSpec "enabled" }} + {{- $haEnabled = $haSpec.enabled }} +{{- end }} +{{- $haReplicas := $haSpec.replicas | default 3 }} +{{- $haSpreadByZone := true }} +{{- if hasKey $haSpec "topologySpreadByZone" }} + {{- $haSpreadByZone = $haSpec.topologySpreadByZone }} +{{- end }} +{{- $ha := dict + "enabled" $haEnabled + "replicas" $haReplicas + "topologySpreadByZone" $haSpreadByZone +}} + # ============================================================================== # Per-component configuration # ============================================================================== @@ -186,6 +207,7 @@ ) ) "nodePrep" $nodePrep + "ha" $ha "cnpg" (dict "name" ($cnpg.name | default "cloudnative-pg") "namespace" ($cnpg.namespace | default $namespace) diff --git a/functions/render/165-openebs-mayastor.yaml.gotmpl b/functions/render/165-openebs-mayastor.yaml.gotmpl index 112a5a6..2315360 100644 --- a/functions/render/165-openebs-mayastor.yaml.gotmpl +++ b/functions/render/165-openebs-mayastor.yaml.gotmpl @@ -44,6 +44,17 @@ spec: "tolerations" $state.nodePool.tolerations ) }} {{- end }} + {{- if $state.ha.enabled }} + {{- /* Mayastor's chart has many deployments. Hit the most-impactful + ones (agents.core, csi.controller, etcd, eventing, obs, operators.pool) + with replica counts. Per-component overrides are still possible + via spec.storage.values. */}} + {{- $haReplicas := $state.ha.replicas }} + {{- $_ := set $chartDefaults "agents" (dict "core" (dict "replicaCount" $haReplicas)) }} + {{- $_ := set $chartDefaults "csi" (dict "controller" (dict "replicaCount" $haReplicas)) }} + {{- $_ := set $chartDefaults "etcd" (dict "replicaCount" $haReplicas) }} + {{- /* Fields not commonly there but harmless if ignored */}} + {{- end }} {{- $mergedValues := mergeOverwrite $chartDefaults ($storage.values | default dict) }} values: {{- toYaml $mergedValues | nindent 6 }} diff --git a/functions/render/200-cnpg-operator.yaml.gotmpl b/functions/render/200-cnpg-operator.yaml.gotmpl index 98e8f7c..58a42fa 100644 --- a/functions/render/200-cnpg-operator.yaml.gotmpl +++ b/functions/render/200-cnpg-operator.yaml.gotmpl @@ -39,6 +39,17 @@ spec: {{- $_ := set $chartDefaults "nodeSelector" $state.nodePool.nodeSelector }} {{- $_ := set $chartDefaults "tolerations" $state.nodePool.tolerations }} {{- end }} + {{- if $state.ha.enabled }} + {{- $_ := set $chartDefaults "replicaCount" $state.ha.replicas }} + {{- if $state.ha.topologySpreadByZone }} + {{- $_ := set $chartDefaults "topologySpreadConstraints" (list (dict + "maxSkew" 1 + "topologyKey" "topology.kubernetes.io/zone" + "whenUnsatisfiable" "ScheduleAnyway" + "labelSelector" (dict "matchLabels" (dict "app.kubernetes.io/name" "cloudnative-pg")) + )) }} + {{- end }} + {{- end }} {{- $mergedValues := mergeOverwrite $chartDefaults ($cnpg.values | default dict) }} values: {{- toYaml $mergedValues | nindent 6 }} diff --git a/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl b/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl index bcb2583..ebabca1 100644 --- a/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl +++ b/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl @@ -294,7 +294,7 @@ spec: {{- end }} app: scale-to-zero spec: - replicas: 1 + replicas: {{ if $state.ha.enabled }}{{ $state.ha.replicas }}{{ else }}1{{ end }} selector: matchLabels: app: scale-to-zero @@ -308,6 +308,15 @@ spec: nodeSelector: {{ $state.nodePool.nodeSelector | toJson }} tolerations: {{ $state.nodePool.tolerations | toJson }} {{- end }} + {{- if and $state.ha.enabled $state.ha.topologySpreadByZone }} + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app: scale-to-zero + {{- end }} securityContext: runAsNonRoot: true runAsUser: 10001 diff --git a/functions/render/220-atlas-operator.yaml.gotmpl b/functions/render/220-atlas-operator.yaml.gotmpl index 1d1a1cf..6b617d7 100644 --- a/functions/render/220-atlas-operator.yaml.gotmpl +++ b/functions/render/220-atlas-operator.yaml.gotmpl @@ -35,6 +35,17 @@ spec: {{- $_ := set $chartDefaults "nodeSelector" $state.nodePool.nodeSelector }} {{- $_ := set $chartDefaults "tolerations" $state.nodePool.tolerations }} {{- end }} + {{- if $state.ha.enabled }} + {{- $_ := set $chartDefaults "replicaCount" $state.ha.replicas }} + {{- if $state.ha.topologySpreadByZone }} + {{- $_ := set $chartDefaults "topologySpreadConstraints" (list (dict + "maxSkew" 1 + "topologyKey" "topology.kubernetes.io/zone" + "whenUnsatisfiable" "ScheduleAnyway" + "labelSelector" (dict "matchLabels" (dict "app.kubernetes.io/name" "atlas-operator")) + )) }} + {{- end }} + {{- end }} {{- $mergedValues := mergeOverwrite $chartDefaults ($atlas.values | default dict) }} values: {{- toYaml $mergedValues | nindent 6 }} diff --git a/tests/test-render/main.k b/tests/test-render/main.k index 4904fd9..2aad6fd 100644 --- a/tests/test-render/main.k +++ b/tests/test-render/main.k @@ -376,6 +376,41 @@ _items = [ } } + # ========================================================================== + # Test 11: ha.enabled=true sets replicaCount + topology spread on CNPG and + # Atlas Helm releases, and on the directly-composed S2Z Deployment. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "ha-enabled-injects-replica-count-and-spread" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "ha-on" + spec = { + clusterName = "prod" + ha = {enabled = True, replicas = 3} + } + } + assertResources = [ + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "cloudnative-pg" + spec.forProvider.values.replicaCount = 3 + } + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "atlas-operator" + spec.forProvider.values.replicaCount = 3 + } + ] + } + } + # ========================================================================== # Test 10: spec.storage settings flow through to the StorageClass # parameters (replicationFactor, thin) From da8d852e0b46bfb5f252d3c6317445a75eff0e2c Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Sat, 25 Apr 2026 09:27:58 -0500 Subject: [PATCH 15/44] fix: skip Mayastor's bundled VolumeSnapshotClass CRDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Avoids Helm ownership conflicts when the cluster already has these CRDs from another source (e.g., a previous OpenEBS LVM install, the cluster's snapshot-controller, or another chart that bundles them). The CRDs (volumesnapshotclasses.snapshot.storage.k8s.io and friends) need to come from somewhere on the cluster — the assumption is that snapshot-controller is installed separately as a cluster-level concern, not bundled with each storage backend. Encountered live during pat-local install: leftover LVM CRD annotations blocked Mayastor's upgrade. Skip CRD install in our defaults to make this robust across re-installs. Co-Authored-By: Claude Opus 4.7 (1M context) --- functions/render/165-openebs-mayastor.yaml.gotmpl | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/functions/render/165-openebs-mayastor.yaml.gotmpl b/functions/render/165-openebs-mayastor.yaml.gotmpl index 2315360..894107c 100644 --- a/functions/render/165-openebs-mayastor.yaml.gotmpl +++ b/functions/render/165-openebs-mayastor.yaml.gotmpl @@ -37,7 +37,14 @@ spec: values: {{- toYaml $storage.overrideAllValues | nindent 6 }} {{- else }} - {{- $chartDefaults := dict }} + {{- /* Skip Mayastor's bundled VolumeSnapshotClass CRD install. The CRDs + come from the cluster's snapshot-controller (must be installed + separately — typically already present on EKS Auto Mode or via + another stack). Avoids Helm "ownership" conflicts when more than + one chart bundles the same CRDs. */}} + {{- $chartDefaults := dict + "crds" (dict "csi" (dict "volumeSnapshots" (dict "enabled" false))) + }} {{- if $state.nodePool.enabled }} {{- $_ := set $chartDefaults "io_engine" (dict "nodeSelector" $state.nodePool.nodeSelector From 4f33ca96207764c63d11f790cb56c8b6d1ca9ec1 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Sat, 25 Apr 2026 14:03:35 -0500 Subject: [PATCH 16/44] fix: confine Mayastor csi-node to workload-type=psql nodes csi-node DaemonSet was running on every cluster node, but workers without the node-prep DS don't have nvme_tcp loaded, so csi-node crashloops there ("Failed to detect nvme_tcp kernel module"). Restrict csi-node via spec.csi.node.{nodeSelector,tolerations} to the same psql NodePool nodes where node-prep loads nvme_tcp. PSQLCluster workloads always schedule on workload-type=psql nodes via the existing NodePool selectors, so this doesn't restrict anything that actually consumes Mayastor PVCs. Also fixes a chartDefaults merge bug introduced when HA mode landed: both nodePool and HA were calling \`set chartDefaults "csi"\` which clobbered each other. Now build the csi sub-dict incrementally before adding it to chartDefaults. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../render/165-openebs-mayastor.yaml.gotmpl | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/functions/render/165-openebs-mayastor.yaml.gotmpl b/functions/render/165-openebs-mayastor.yaml.gotmpl index 894107c..d5302c9 100644 --- a/functions/render/165-openebs-mayastor.yaml.gotmpl +++ b/functions/render/165-openebs-mayastor.yaml.gotmpl @@ -42,8 +42,25 @@ spec: separately — typically already present on EKS Auto Mode or via another stack). Avoids Helm "ownership" conflicts when more than one chart bundles the same CRDs. */}} + {{- /* Build csi sub-dict incrementally — both nodePool and HA contribute + to it, so direct `set` would clobber. */}} + {{- $csi := dict }} + {{- if $state.nodePool.enabled }} + {{- /* Confine csi-node to NVMe nodes too — non-psql workers don't have + nvme_tcp loaded, so csi-node would crashloop there. */}} + {{- $_ := set $csi "node" (dict + "nodeSelector" $state.nodePool.nodeSelector + "tolerations" $state.nodePool.tolerations + ) }} + {{- end }} + {{- if $state.ha.enabled }} + {{- $_ := set $csi "controller" (dict "replicaCount" $state.ha.replicas) }} + {{- end }} + + {{- /* Skip Mayastor's bundled VolumeSnapshotClass CRD install. */}} {{- $chartDefaults := dict "crds" (dict "csi" (dict "volumeSnapshots" (dict "enabled" false))) + "csi" $csi }} {{- if $state.nodePool.enabled }} {{- $_ := set $chartDefaults "io_engine" (dict @@ -52,15 +69,8 @@ spec: ) }} {{- end }} {{- if $state.ha.enabled }} - {{- /* Mayastor's chart has many deployments. Hit the most-impactful - ones (agents.core, csi.controller, etcd, eventing, obs, operators.pool) - with replica counts. Per-component overrides are still possible - via spec.storage.values. */}} - {{- $haReplicas := $state.ha.replicas }} - {{- $_ := set $chartDefaults "agents" (dict "core" (dict "replicaCount" $haReplicas)) }} - {{- $_ := set $chartDefaults "csi" (dict "controller" (dict "replicaCount" $haReplicas)) }} - {{- $_ := set $chartDefaults "etcd" (dict "replicaCount" $haReplicas) }} - {{- /* Fields not commonly there but harmless if ignored */}} + {{- $_ := set $chartDefaults "agents" (dict "core" (dict "replicaCount" $state.ha.replicas)) }} + {{- $_ := set $chartDefaults "etcd" (dict "replicaCount" $state.ha.replicas) }} {{- end }} {{- $mergedValues := mergeOverwrite $chartDefaults ($storage.values | default dict) }} values: From 727dc7412a5a9b988df353ae0d14054bbeabcc4d Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Sat, 25 Apr 2026 15:01:13 -0500 Subject: [PATCH 17/44] feat: pivot storage from Mayastor to Longhorn V2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 165-longhorn.yaml.gotmpl: longhorn chart 1.10.0, V2 data engine + SPDK - 170-storageclass-longhorn.yaml.gotmpl: driver.longhorn.io SC + VSC, dataEngine=v2, diskSelector=psql - 155-node-prep-daemonset.yaml.gotmpl: modprobe nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv; replace DiskPool registration with node.longhorn.io/default-disks-config annotation - definition.yaml + state-init: chart defaults bump (1.10.0, longhorn-system); doc strings updated - 010-state-status: observe `longhorn` + `storageclass-longhorn` - README + examples updated; KCL tests updated and all 11 pass Mayastor checkpoint preserved on feat/cnpg-pivot. Longhorn V2 is "Experimental" upstream as of 1.10 — drops bundled etcd/NATS/minio/ loki/alloy footprint and unblocks arm64 Graviton (Mayastor's chart hardcodes amd64 on io_engine). --- README.md | 52 ++++----- apis/psqlstacks/definition.yaml | 20 ++-- examples/psqlstacks/local.yaml | 6 +- examples/psqlstacks/minimal.yaml | 11 +- examples/psqlstacks/standard.yaml | 2 +- functions/render/000-state-init.yaml.gotmpl | 26 +++-- functions/render/010-state-status.yaml.gotmpl | 6 +- .../155-node-prep-daemonset.yaml.gotmpl | 91 ++++++++-------- functions/render/165-longhorn.yaml.gotmpl | 103 ++++++++++++++++++ .../render/165-openebs-mayastor.yaml.gotmpl | 83 -------------- ... => 170-storageclass-longhorn.yaml.gotmpl} | 34 +++--- tests/test-render/main.k | 27 +++-- 12 files changed, 245 insertions(+), 216 deletions(-) create mode 100644 functions/render/165-longhorn.yaml.gotmpl delete mode 100644 functions/render/165-openebs-mayastor.yaml.gotmpl rename functions/render/{170-storageclass-mayastor.yaml.gotmpl => 170-storageclass-longhorn.yaml.gotmpl} (60%) diff --git a/README.md b/README.md index fea25c0..72a926c 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,24 @@ # psql-stack -PostgreSQL platform stack on top of [CloudNativePG](https://cloudnative-pg.io/) with [OpenEBS Mayastor](https://github.com/openebs/mayastor) for replicated NVMe-oF storage. Installs the operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), Atlas Operator (schema migrations), Mayastor + matching StorageClass + VolumeSnapshotClass, two Karpenter NodePools, and a node-prep DaemonSet. +PostgreSQL platform stack on top of [CloudNativePG](https://cloudnative-pg.io/) with [Longhorn](https://github.com/longhorn/longhorn) (V2 data engine — SPDK + NVMe-oF) for replicated CoW storage. Installs the operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), Atlas Operator (schema migrations), Longhorn + matching StorageClass + VolumeSnapshotClass, two Karpenter NodePools, and a node-prep DaemonSet. + +> **Storage status**: Longhorn V2 is "Experimental" upstream as of Longhorn 1.10. V1 (iSCSI) is GA but not used here — V2 was chosen for SPDK performance parity with Mayastor without Mayastor's heavy footprint. Track promotion to GA before high-stakes production use, or override `spec.storage.values.defaultSettings.v2DataEngine: false` to fall back to V1. This is the **platform layer** — it does not create any serving Postgres clusters. Per-app DBs live in [`PSQLCluster`](../../psql-cluster/) (separate XR), ephemeral forks in [`PSQLBranch`](../../psql-branch/) (separate XR). ## Why psql-stack? **Without psql-stack:** -- Manual Helm installs of CNPG, Atlas, Mayastor on every cluster +- Manual Helm installs of CNPG, Atlas, Longhorn on every cluster - No deletion ordering — removing CNPG before Atlas can leave migrations dangling -- No node-side prep for Mayastor (hugepages + `nvme-tcp` module need to exist before the IO engine pods will run) +- No node-side prep for Longhorn V2 (hugepages + `nvme_tcp` / `vfio_pci` / `uio_pci_generic` / `ublk_drv` kernel modules need to exist before the SPDK instance-manager pods will run) - No declarative substrate for the `cnpg-i-scale-to-zero` plugin (cert-manager-backed gRPC TLS, ServiceAccount + RBAC, sidecar config) **With psql-stack:** -- Single claim deploys CNPG + S2Z plugin + Atlas + Mayastor + StorageClass with production defaults +- Single claim deploys CNPG + S2Z plugin + Atlas + Longhorn + StorageClass with production defaults - Deletion order enforced via `protection.crossplane.io/Usage` resources -- Replicated NVMe-oF CoW storage — `replicationFactor: 3` for primaries, `replicationFactor: 1` for ephemeral branches (per-PVC override). Single backend covers both uses. -- Dedicated Karpenter NodePools (`branches` spot, `primary` on-demand) targeting NVMe instance-store nodes (`i4g.2xlarge` / `i4g.4xlarge` / `im4gn.2xlarge` arm64 Graviton) +- Replicated SPDK / NVMe-oF CoW storage — `replicationFactor: 3` for primaries, `replicationFactor: 1` for ephemeral branches (per-PVC override). Single backend covers both uses. +- Dedicated Karpenter NodePools (both spot — Longhorn replication absorbs preemption) targeting NVMe instance-store nodes (`i4g.2xlarge` / `i4g.4xlarge` / `im4gn.2xlarge` arm64 Graviton) - Pinnable upstream chart / plugin versions; Renovate keeps them current ## Components @@ -26,10 +28,10 @@ This is the **platform layer** — it does not create any serving Postgres clust | **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | | **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. Pinnable via `spec.scaleToZeroPlugin.version`. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | | **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | -| **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas of every HA-able platform component + `topologySpreadConstraints` by zone. Affects CNPG operator, Atlas, S2Z plugin, Mayastor control plane. | -| **Karpenter NodePools** | on (`spec.nodePool.enabled: true`) | `branches` (spot arm64 NVMe) and `primary` (on-demand arm64 NVMe). | -| **OpenEBS Mayastor** | on when `nodePool.enabled` | Replicated NVMe-oF storage with CoW snapshots. Single `psql` StorageClass + matching VolumeSnapshotClass. | -| **node-prep DaemonSet** | on when `nodePool.enabled` | Configures hugepages + loads `nvme-tcp` kernel module on each NVMe node (Mayastor prereqs). | +| **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas of every HA-able platform component + `topologySpreadConstraints` by zone. Affects CNPG operator, Atlas, S2Z plugin, Longhorn CSI controllers + UI. | +| **Karpenter NodePools** | on (`spec.nodePool.enabled: true`) | `branches` (spot arm64 NVMe) and `primary` (spot arm64 NVMe — Longhorn replication absorbs preemption). | +| **Longhorn (V2)** | on when `nodePool.enabled` | Replicated SPDK / NVMe-oF storage with CoW snapshots. Single `psql` StorageClass + matching VolumeSnapshotClass. Longhorn keeps state in CRDs — no bundled etcd / NATS / minio. | +| **node-prep DaemonSet** | on when `nodePool.enabled` | Configures hugepages + loads `nvme_tcp` / `vfio_pci` / `uio_pci_generic` / `ublk_drv` kernel modules on each NVMe node (Longhorn V2 prereqs); annotates the K8s Node so Longhorn auto-registers the local instance-store device as a V2 block-mode disk tagged `psql`. | ## Prerequisites @@ -40,7 +42,7 @@ This is the **platform layer** — it does not create any serving Postgres clust ### Stage 1: Default install -Deploy with all defaults: CNPG + Atlas + S2Z plugin + Mayastor + dedicated NodePools. Karpenter provisions NVMe instance-store nodes when something requests them. +Deploy with all defaults: CNPG + Atlas + S2Z plugin + Longhorn + dedicated NodePools. Karpenter provisions NVMe instance-store nodes when something requests them. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -83,7 +85,7 @@ spec: ### Stage 3: Local / no-NVMe clusters -For kind / k3d / clusters that can't run Mayastor, disable the NodePool. The stack then ships only CNPG + Atlas + S2Z plugin, and your PSQLClusters target whatever StorageClass the cluster provides. +For kind / k3d / clusters that can't run Longhorn V2, disable the NodePool. The stack then ships only CNPG + Atlas + S2Z plugin, and your PSQLClusters target whatever StorageClass the cluster provides. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -130,28 +132,28 @@ spec: | `ha.replicas` | int | `3` | Replica count for HA-able platform components | | `ha.topologySpreadByZone` | bool | `true` | Add topologySpreadConstraint with topologyKey=topology.kubernetes.io/zone, maxSkew=1, whenUnsatisfiable=ScheduleAnyway | | **NodePool** | | | | -| `nodePool.enabled` | bool | `true` | Master toggle. When false, Mayastor + StorageClass + node-prep are skipped too. | +| `nodePool.enabled` | bool | `true` | Master toggle. When false, Longhorn + StorageClass + node-prep are skipped too. | | `nodePool.nodeClassName` | string | `default` | EKS NodeClass referenced by both sub-pools | | `nodePool.disruption.consolidationPolicy` | enum | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | | `nodePool.disruption.consolidateAfter` | string | `60s` | Consolidation delay | | `nodePool.branches.enabled` | bool | `true` | Spot arm64 NVMe sub-pool | | `nodePool.branches.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | | `nodePool.branches.requirements` | array | arm64 spot `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | -| `nodePool.primary.enabled` | bool | `true` | On-demand arm64 NVMe sub-pool | +| `nodePool.primary.enabled` | bool | `true` | Spot arm64 NVMe sub-pool | | `nodePool.primary.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | -| `nodePool.primary.requirements` | array | arm64 on-demand `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | -| **Storage (Mayastor)** | | | | -| `storage.chartVersion` | string | `2.10.0` | Mayastor Helm chart version | -| `storage.namespace` | string | `mayastor` | Helm release namespace | +| `nodePool.primary.requirements` | array | arm64 spot `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | +| **Storage (Longhorn V2)** | | | | +| `storage.chartVersion` | string | `1.10.0` | Longhorn Helm chart version | +| `storage.namespace` | string | `longhorn-system` | Helm release namespace | | `storage.storageClassName` | string | `psql` | StorageClass name | -| `storage.replicationFactor` | int | `3` | Default replicas per volume (per-PVC override via parameters) | -| `storage.thin` | bool | `true` | Thin-provisioning (CoW) | +| `storage.replicationFactor` | int | `3` | Default replicas per volume (`numberOfReplicas` parameter on the SC) | +| `storage.thin` | bool | `true` | Thin-provisioning (CoW). Longhorn V2 is CoW by default; the flag is exposed for symmetry but is essentially always true under V2. | | `storage.values` | object | — | Helm values merged with chart defaults | | `storage.overrideAllValues` | object | — | Helm values that replace all defaults | | **Node prep** | | | | | `nodePrep.enabled` | bool | `true` (when `nodePool.enabled`) | Compose the privileged DaemonSet | -| `nodePrep.hugepages.count` | int | `1024` | Number of 2MiB hugepages per node | -| `nodePrep.image` | string | `alpine:3.20` | Init container image (apk-install at runtime) | +| `nodePrep.hugepages.count` | int | `1024` | Number of 2MiB hugepages per node (Longhorn V2 SPDK requires) | +| `nodePrep.image` | string | `alpine:3.20` | Init container image (apk-install kubectl + util-linux at runtime) | ## Composed Resources @@ -160,9 +162,9 @@ spec: | `cloudnative-pg` | `helm.m.crossplane.io/Release` | always | | `atlas-operator` | `helm.m.crossplane.io/Release` | always | | 9× `-s2z-*` | `kubernetes.m.crossplane.io/Object` | `scaleToZeroPlugin.enabled: true` (default) | -| `openebs-mayastor` | `helm.m.crossplane.io/Release` | `nodePool.enabled: true` (default) | -| `-storageclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | -| `-volumesnapshotclass-mayastor` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | +| `longhorn` | `helm.m.crossplane.io/Release` | `nodePool.enabled: true` (default) | +| `-storageclass-longhorn` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | +| `-volumesnapshotclass-longhorn` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | | `-nodepool-branches` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled && nodePool.branches.enabled` | | `-nodepool-primary` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled && nodePool.primary.enabled` | | `-node-prep` | `kubernetes.m.crossplane.io/Object` (DaemonSet) | `nodePool.enabled && nodePrep.enabled` | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 2ac815f..6e69412 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -67,14 +67,14 @@ spec: description: Shared namespace for CNPG operator, scale-to-zero plugin, and Atlas operator. Defaults to cnpg-system. Per-component namespace overrides this. type: string storage: - description: "Storage backend (OpenEBS Mayastor — replicated NVMe-oF with CoW snapshots). The stack is opinionated: it ships exactly one storage backend, sized for CoW + HA. Per-cluster cost is controlled via replicationFactor (3 for primaries, 1 for ephemeral branches). Requires NVMe instance-store nodes (provided by the NodePool block) plus hugepages and the nvme-tcp kernel module (provided by the node-prep DaemonSet). For non-CoW workloads, PSQLCluster claims can target any other StorageClass that exists on the target cluster (e.g., the cluster's default gp3 SC) — no need for the stack to compose that." + description: "Storage backend (Longhorn V2 — SPDK + NVMe-oF with replicated CoW snapshots). The stack is opinionated: it ships exactly one storage backend, sized for CoW + HA. Per-cluster cost is controlled via replicationFactor (3 for primaries, 1 for ephemeral branches). Requires NVMe instance-store nodes (provided by the NodePool block) plus hugepages and the nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv kernel modules (provided by the node-prep DaemonSet). Longhorn V2 is marked Experimental upstream (Longhorn 1.10) — track promotion to GA before high-stakes production use. For non-CoW workloads, PSQLCluster claims can target any other StorageClass that exists on the target cluster (e.g., the cluster's default gp3 SC) — no need for the stack to compose that." type: object properties: chartVersion: - description: Mayastor Helm chart version. Defaults to 2.10.0. + description: Longhorn Helm chart version. Defaults to 1.10.0. type: string namespace: - description: Namespace for the Mayastor install. Defaults to mayastor. + description: Namespace for the Longhorn install. Defaults to longhorn-system. type: string storageClassName: description: StorageClass name. Defaults to "psql". @@ -105,7 +105,7 @@ spec: type: object x-kubernetes-preserve-unknown-fields: true nodePool: - description: "Dedicated Karpenter NodePools for PostgreSQL workloads. Two sub-pools — `branches` (spot arm64 NVMe, for ephemeral PSQLBranch workloads) and `primary` (on-demand arm64 NVMe, for replicated-cow PSQLCluster primaries where spot preemption would lose a Mayastor replica). Defaults to disabled — node-side prep for Mayastor/LVM (hugepages, nvme-tcp, LVM volume groups) is a separate concern handled in phase 3b. CNPG / scale-to-zero / Atlas schedule on the primary pool when enabled." + description: "Dedicated Karpenter NodePools for PostgreSQL workloads. Two sub-pools — `branches` (spot arm64 NVMe, for ephemeral PSQLBranch workloads) and `primary` (spot arm64 NVMe, for replicated-CoW PSQLCluster primaries — Longhorn V2 replication absorbs spot preemption). Defaults to enabled — node-side prep for Longhorn V2 (hugepages + nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv) is composed automatically by the node-prep DaemonSet. CNPG / scale-to-zero / Atlas schedule on the primary pool when enabled." type: object properties: enabled: @@ -157,7 +157,7 @@ spec: type: string required: [key, operator] primary: - description: "Spot arm64 NVMe sub-pool for PSQLCluster primaries. Mayastor's replicationFactor=3 absorbs spot preemption — losing one replica triggers a rebuild on a fresh node, not data loss. Override capacity-type to on-demand via spec.nodePool.primary.requirements if your durability requirements rule out preemption." + description: "Spot arm64 NVMe sub-pool for PSQLCluster primaries. Longhorn V2's replicationFactor=3 absorbs spot preemption — losing one replica triggers a rebuild on a fresh node, not data loss. Override capacity-type to on-demand via spec.nodePool.primary.requirements if your durability requirements rule out preemption." type: object properties: enabled: @@ -187,25 +187,25 @@ spec: type: string required: [key, operator] nodePrep: - description: "Node-prep DaemonSet for Mayastor prerequisites. Runs as a privileged init container on each NVMe NodePool node and configures hugepages (vm.nr_hugepages, required by Mayastor SPDK) plus loads the nvme-tcp kernel module (Mayastor's NVMe-oF transport). Composed when nodePool.enabled. EKS Auto Mode / Bottlerocket may pre-load nvme-tcp; verify with `lsmod | grep nvme_tcp` before assuming the modprobe step is needed." + description: "Node-prep DaemonSet for Longhorn V2 prerequisites. Runs as a privileged init container on each NVMe NodePool node and configures hugepages (vm.nr_hugepages, required by SPDK) plus loads the nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv kernel modules (Longhorn V2 NVMe-oF + UBLK transports). Also annotates the K8s Node with `node.longhorn.io/default-disks-config` so Longhorn auto-registers the local NVMe instance-store device as a V2 block-mode disk. Composed when nodePool.enabled." type: object properties: enabled: - description: Whether to compose the node-prep DaemonSet. Defaults to true (auto-gated by storage backends — when neither mayastor nor lvm is enabled the DaemonSet is skipped regardless). + description: Whether to compose the node-prep DaemonSet. Defaults to true (auto-gated by nodePool.enabled — no NVMe nodes means no prep needed). type: boolean default: true hugepages: type: object properties: count: - description: Number of 2MiB hugepages to allocate on each node. Defaults to 1024 (2GiB total). Required for Mayastor; ignored when storage.mayastor.enabled is false. + description: Number of 2MiB hugepages to allocate on each node. Defaults to 1024 (2GiB total). Required by Longhorn V2 SPDK. type: integer default: 1024 image: - description: Container image for the prep init container. Must have apk (or be pre-baked with lvm2 + util-linux). Defaults to alpine:3.20 (apk-installs lvm2 at startup). + description: Container image for the prep init container. Must have apk (for kubectl + util-linux install). Defaults to alpine:3.20. type: string ha: - description: "Stack-wide HA mode. When enabled, sets replicaCount + topology spread (one replica per zone) on all platform components: CNPG operator, Atlas operator, scale-to-zero plugin, Mayastor control plane. Per-component values can still be overridden via each component's values block." + description: "Stack-wide HA mode. When enabled, sets replicaCount + topology spread (one replica per zone) on all platform components: CNPG operator, Atlas operator, scale-to-zero plugin, Longhorn control plane. Per-component values can still be overridden via each component's values block." type: object properties: enabled: diff --git a/examples/psqlstacks/local.yaml b/examples/psqlstacks/local.yaml index 3f42a1e..385f0a9 100644 --- a/examples/psqlstacks/local.yaml +++ b/examples/psqlstacks/local.yaml @@ -1,8 +1,8 @@ -# Local: NodePool / Mayastor disabled — gets just CNPG + Atlas + S2Z plugin +# Local: NodePool / Longhorn disabled — gets just CNPG + Atlas + S2Z plugin # installed on whatever StorageClass already exists on the local cluster. # -# Useful for kind / k3d dev clusters that can't run Mayastor. -# PSQLClusters on this stack will need to specify a non-Mayastor SC +# Useful for kind / k3d dev clusters that can't run Longhorn V2. +# PSQLClusters on this stack will need to specify a non-Longhorn SC # (`spec.storage.class: standard` for k3d, etc.). # apiVersion: hops.ops.com.ai/v1alpha1 diff --git a/examples/psqlstacks/minimal.yaml b/examples/psqlstacks/minimal.yaml index 324b300..1baead3 100644 --- a/examples/psqlstacks/minimal.yaml +++ b/examples/psqlstacks/minimal.yaml @@ -4,13 +4,14 @@ # - CloudNativePG operator (Helm) # - cnpg-i-scale-to-zero plugin (set of Objects) # - Atlas operator (Helm) -# - Karpenter NodePool sub-pools: branches (spot) + primary (on-demand) -# - OpenEBS Mayastor (Helm) on the primary sub-pool nodes -# - psql StorageClass + VolumeSnapshotClass (Mayastor-backed, replicated, CoW) -# - node-prep DaemonSet (hugepages + nvme-tcp on each NVMe node) +# - Karpenter NodePool sub-pools: branches (spot) + primary (spot) +# - Longhorn V2 (Helm) on the primary sub-pool nodes +# - psql StorageClass + VolumeSnapshotClass (Longhorn-backed, replicated, CoW) +# - node-prep DaemonSet (hugepages + nvme_tcp / vfio_pci / uio_pci_generic +# / ublk_drv on each NVMe node, plus Longhorn disk auto-registration) # # This is the stack's whole point: CoW Postgres on dedicated NVMe nodes. -# To opt out of NVMe / Mayastor entirely (and just get CNPG + Atlas + S2Z +# To opt out of NVMe / Longhorn entirely (and just get CNPG + Atlas + S2Z # on the cluster's existing default StorageClass), set nodePool.enabled: false # and have your PSQLClusters target a different SC. # diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index d423ad5..4f9e79a 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -25,7 +25,7 @@ spec: ha: enabled: true # 3 replicas + topology spread by zone on replicas: 3 # all platform components (CNPG, Atlas, - topologySpreadByZone: true # S2Z plugin, Mayastor control plane) + topologySpreadByZone: true # S2Z plugin, Longhorn CSI controllers) scaleToZeroPlugin: enabled: true atlasOperator: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 33aa234..59e452d 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -37,12 +37,14 @@ }} # ============================================================================== -# Storage — OpenEBS Mayastor (replicated NVMe-oF + CoW) +# Storage — Longhorn V2 (replicated SPDK + NVMe-oF + CoW) # Single backend; per-PVC cost is controlled via replicationFactor. Requires -# NVMe instance-store nodes (NodePool block) plus hugepages + nvme-tcp module -# (node-prep DaemonSet). PSQLClusters that don't want CoW can target any other -# StorageClass that exists on the cluster (e.g., the default gp3 SC) — the -# stack doesn't compose a non-CoW SC itself. +# NVMe instance-store nodes (NodePool block) plus hugepages + nvme_tcp / +# vfio_pci / uio_pci_generic / ublk_drv modules (node-prep DaemonSet). +# Longhorn V2 is Experimental upstream as of Longhorn 1.10. +# PSQLClusters that don't want CoW can target any other StorageClass that +# exists on the cluster (e.g., the default gp3 SC) — the stack doesn't +# compose a non-CoW SC itself. # ============================================================================== {{- $storageSpec := $spec.storage | default dict }} {{- $thin := true }} @@ -54,8 +56,8 @@ {{- $allowVolumeExpansion = $storageSpec.allowVolumeExpansion }} {{- end }} {{- $storage := dict - "chartVersion" ($storageSpec.chartVersion | default "2.10.0") - "namespace" ($storageSpec.namespace | default "mayastor") + "chartVersion" ($storageSpec.chartVersion | default "1.10.0") + "namespace" ($storageSpec.namespace | default "longhorn-system") "storageClassName" ($storageSpec.storageClassName | default "psql") "replicationFactor" ($storageSpec.replicationFactor | default 3) "thin" $thin @@ -100,8 +102,9 @@ (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) ) }} -# primary sub-pool (spot by default — Mayastor's replication absorbs preemption; -# losing one replica of three triggers a rebuild on a fresh node, not data loss) +# primary sub-pool (spot by default — Longhorn V2's replication absorbs +# preemption; losing one replica of three triggers a rebuild on a fresh +# node, not data loss) {{- $primarySpec := $nodePoolSpec.primary | default dict }} {{- $primaryEnabled := true }} {{- if hasKey $primarySpec "enabled" }} @@ -127,7 +130,10 @@ {{- end }} # ============================================================================== -# Node prep — hugepages + nvme-tcp module on NVMe nodes (Mayastor prereqs) +# Node prep — hugepages + nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv +# kernel modules on NVMe nodes (Longhorn V2 prereqs); also annotates the +# K8s Node with node.longhorn.io/default-disks-config so Longhorn registers +# the local NVMe instance-store device as a V2 block-mode disk. # Auto-gated: emitted when nodePool.enabled (no nodes → DS no-op anyway) # ============================================================================== {{- $nodePrepSpec := $spec.nodePrep | default dict }} diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index d393811..ceb8273 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "node-prep" "openebs-mayastor" "storageclass-mayastor" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "node-prep" "longhorn" "storageclass-longhorn" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -33,8 +33,8 @@ "nodepoolBranches" (dict "ready" (get $checkReady "nodepool-branches")) "nodepoolPrimary" (dict "ready" (get $checkReady "nodepool-primary")) "nodePrep" (dict "ready" (get $checkReady "node-prep")) - "mayastor" (dict "ready" (get $checkReady "openebs-mayastor")) - "storageClass" (dict "ready" (get $checkReady "storageclass-mayastor")) + "longhorn" (dict "ready" (get $checkReady "longhorn")) + "storageClass" (dict "ready" (get $checkReady "storageclass-longhorn")) ) }} # ============================================================================== diff --git a/functions/render/155-node-prep-daemonset.yaml.gotmpl b/functions/render/155-node-prep-daemonset.yaml.gotmpl index 0c3c231..92e2b68 100644 --- a/functions/render/155-node-prep-daemonset.yaml.gotmpl +++ b/functions/render/155-node-prep-daemonset.yaml.gotmpl @@ -1,29 +1,36 @@ # code: language=yaml # -# Node-prep DaemonSet for Mayastor prerequisites +# Node-prep DaemonSet for Longhorn V2 prerequisites # # Runs once per NVMe NodePool node and configures the host-level state -# Mayastor requires (hugepages, nvme-tcp), then registers the local NVMe -# instance-store device with Mayastor by creating a per-node DiskPool CR. +# Longhorn V2's SPDK data engine requires, then tells Longhorn to register +# the local NVMe instance-store device as a V2 block-mode disk by labeling +# + annotating the K8s Node. # # Composes: # - ServiceAccount + ClusterRole + ClusterRoleBinding granting the DS -# permission to get/create DiskPools in the mayastor namespace +# permission to get/patch K8s Nodes (to set Longhorn disk labels and +# annotations). # - DaemonSet that, per node: -# 1. Sysctls vm.nr_hugepages (required by Mayastor SPDK) -# 2. modprobes nvme-tcp (Mayastor NVMe-oF transport) +# 1. Sysctls vm.nr_hugepages (required by SPDK) +# 2. modprobes nvme_tcp, vfio_pci, uio_pci_generic, ublk_drv +# (Longhorn V2 transports + UBLK ingress) # 3. Detects /dev/nvme[1-9]n1 instance-store device by lsblk model -# 4. Creates a DiskPool CR `psql-pool-` if not present +# 4. Labels the K8s Node `node.longhorn.io/create-default-disk=config` +# and annotates `node.longhorn.io/default-disks-config=[...]` so +# Longhorn auto-registers the device as a V2 block-mode disk +# tagged `psql` (matched by the StorageClass `diskSelector: psql`). # # After the init container finishes, a tiny pause container keeps the # pod up so the DaemonSet reports Ready. # # Caveats: -# - On EKS Auto Mode (Bottlerocket OS), nvme-tcp may already be -# pre-loaded; modprobe inside a container may be restricted — -# script falls back gracefully. -# - Auto Mode's NodeClass doesn't support userData, so runtime -# allocation via this DS is currently the only path for hugepages. +# - On EKS Auto Mode (Bottlerocket OS), nvme_tcp is typically pre-loaded; +# vfio_pci / uio_pci_generic / ublk_drv may need the modprobe attempt; +# script falls back gracefully on each. +# - Auto Mode's NodeClass doesn't support userData, so runtime allocation +# via this DS is currently the only path for hugepages + non-default +# kernel modules. # {{- if $state.nodePrep.enabled }} @@ -51,7 +58,7 @@ spec: name: {{ $state.kubernetesProviderConfigRef.name }} kind: {{ $state.kubernetesProviderConfigRef.kind }} -# ---- ClusterRole: get/create DiskPools in mayastor namespace ------------- +# ---- ClusterRole: patch labels + annotations on K8s Nodes ---------------- --- apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object @@ -70,9 +77,9 @@ spec: name: psql-node-prep labels: {{ $state.labels | toJson }} rules: - - apiGroups: ["openebs.io"] - resources: ["diskpools"] - verbs: ["get", "create", "list", "watch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "patch", "update"] providerConfigRef: name: {{ $state.kubernetesProviderConfigRef.name }} kind: {{ $state.kubernetesProviderConfigRef.kind }} @@ -160,8 +167,6 @@ spec: valueFrom: fieldRef: fieldPath: spec.nodeName - - name: MAYASTOR_NS - value: {{ $state.storage.namespace | quote }} command: - /bin/sh - -c @@ -172,7 +177,7 @@ spec: apk add --no-cache util-linux curl kubectl fi - # ---- Hugepages (Mayastor SPDK requires) ---- + # ---- Hugepages (SPDK requires) ---- echo "Setting vm.nr_hugepages={{ $state.nodePrep.hugepages.count }}" if [ -w /proc/sys/vm/nr_hugepages ]; then echo {{ $state.nodePrep.hugepages.count }} > /proc/sys/vm/nr_hugepages @@ -180,39 +185,31 @@ spec: echo "WARN: /proc/sys/vm/nr_hugepages not writable; allocate manually" fi - # ---- nvme-tcp module ---- - if lsmod 2>/dev/null | grep -q "^nvme_tcp"; then - echo "nvme-tcp already loaded" - else - echo "Attempting modprobe nvme-tcp" - modprobe nvme-tcp 2>/dev/null \ - || nsenter -t 1 -m modprobe nvme-tcp 2>/dev/null \ - || echo "WARN: could not load nvme-tcp; verify on host (Bottlerocket may pre-load it)" - fi + # ---- Kernel modules required by Longhorn V2 ---- + for mod in nvme_tcp vfio_pci uio_pci_generic ublk_drv; do + if lsmod 2>/dev/null | grep -q "^${mod}"; then + echo "${mod} already loaded" + else + echo "Attempting modprobe ${mod}" + modprobe ${mod} 2>/dev/null \ + || nsenter -t 1 -m modprobe ${mod} 2>/dev/null \ + || echo "WARN: could not load ${mod} (Bottlerocket may not ship it; verify on host)" + fi + done - # ---- Register Mayastor DiskPool for this node ---- + # ---- Register the local NVMe instance-store device as a + # Longhorn V2 block-mode disk via Node label + annotation DEVICE=$(lsblk -dno NAME,TYPE,MODEL 2>/dev/null \ | awk '$2=="disk" && /Amazon EC2 NVMe Instance Storage/ {print "/dev/"$1; exit}') if [ -z "$DEVICE" ]; then - echo "WARN: no instance-store NVMe device on $NODE_NAME; skipping DiskPool" + echo "WARN: no instance-store NVMe device on $NODE_NAME; skipping disk registration" else - POOL_NAME="psql-pool-${NODE_NAME}" - if kubectl get diskpool -n "$MAYASTOR_NS" "$POOL_NAME" >/dev/null 2>&1; then - echo "DiskPool $POOL_NAME already exists" - else - echo "Creating DiskPool $POOL_NAME on $NODE_NAME ($DEVICE)" - cat </v2-data-engine/ +# +# Chart: https://charts.longhorn.io (chart name: longhorn) +# Upstream: https://github.com/longhorn/longhorn +# + +{{- $storage := $state.storage }} +{{- if $state.nodePool.enabled }} +--- +apiVersion: helm.m.crossplane.io/v1beta1 +kind: Release +metadata: + name: longhorn + annotations: + {{ setResourceNameAnnotation "longhorn" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + chart: + name: longhorn + repository: https://charts.longhorn.io + version: {{ $storage.chartVersion | quote }} + namespace: {{ $storage.namespace }} + {{- if $storage.overrideAllValues }} + values: + {{- toYaml $storage.overrideAllValues | nindent 6 }} + {{- else }} + {{- /* Build defaultSettings — V2 data engine + replication defaults. */}} + {{- $defaultSettings := dict + "v2DataEngine" true + "v2DataEngineHugepageLimit" "2048" + "defaultReplicaCount" $storage.replicationFactor + "defaultDataLocality" "best-effort" + "staleReplicaTimeout" 30 + "createDefaultDiskLabeledNodes" true + }} + + {{- /* Confine Longhorn components to NVMe nodes. Longhorn manager and + instance-manager are DaemonSets; csi components are Deployments. + All ride workload-type=psql nodes only. */}} + {{- $nodeSelector := dict }} + {{- $tolerations := list }} + {{- if $state.nodePool.enabled }} + {{- $nodeSelector = $state.nodePool.nodeSelector }} + {{- $tolerations = $state.nodePool.tolerations }} + {{- end }} + + {{- /* Helm chart structure: longhornManager, longhornDriver, longhornUI, + csi all accept tolerations + nodeSelector via top-level keys. */}} + {{- $chartDefaults := dict + "defaultSettings" $defaultSettings + "longhornManager" (dict + "tolerations" $tolerations + "nodeSelector" $nodeSelector + ) + "longhornDriver" (dict + "tolerations" $tolerations + "nodeSelector" $nodeSelector + ) + "longhornUI" (dict + "tolerations" $tolerations + "nodeSelector" $nodeSelector + ) + }} + + {{- if $state.ha.enabled }} + {{- /* HA: bump CSI controller + UI replicas. Longhorn manager runs as + a DaemonSet on every workload-type=psql node, so replica count + scales with the NodePool, not this knob. */}} + {{- $_ := set $chartDefaults "longhornUI" (mergeOverwrite (get $chartDefaults "longhornUI") (dict "replicas" $state.ha.replicas)) }} + {{- $_ := set $chartDefaults "csi" (dict + "attacherReplicaCount" $state.ha.replicas + "provisionerReplicaCount" $state.ha.replicas + "resizerReplicaCount" $state.ha.replicas + "snapshotterReplicaCount" $state.ha.replicas + ) }} + {{- end }} + + {{- $mergedValues := mergeOverwrite $chartDefaults ($storage.values | default dict) }} + values: + {{- toYaml $mergedValues | nindent 6 }} + {{- end }} + rollbackLimit: 3 + providerConfigRef: + name: {{ $state.helmProviderConfigRef.name }} + kind: {{ $state.helmProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/165-openebs-mayastor.yaml.gotmpl b/functions/render/165-openebs-mayastor.yaml.gotmpl deleted file mode 100644 index d5302c9..0000000 --- a/functions/render/165-openebs-mayastor.yaml.gotmpl +++ /dev/null @@ -1,83 +0,0 @@ -# code: language=yaml -# -# Helm Release: OpenEBS Mayastor (Replicated NVMe-oF) -# -# Provides replicated CoW storage via NVMe-over-TCP. Enterprise default for -# primary serving clusters — HA across N nodes (replicationFactor) plus -# instant CoW snapshots for branching. -# -# Node-side prerequisites (provided by phase 3's NodePool config): -# - 2MiB hugepages (`vm.nr_hugepages=1024` minimum) -# - `nvme-tcp` kernel module loaded -# - Local NVMe instance-store devices for DiskPool allocation -# -# Chart: https://openebs.github.io/mayastor-extensions (chart name: mayastor) -# Upstream: https://github.com/openebs/mayastor -# - -{{- $storage := $state.storage }} -{{- if $state.nodePool.enabled }} ---- -apiVersion: helm.m.crossplane.io/v1beta1 -kind: Release -metadata: - name: openebs-mayastor - annotations: - {{ setResourceNameAnnotation "openebs-mayastor" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - chart: - name: mayastor - repository: https://openebs.github.io/mayastor-extensions - version: {{ $storage.chartVersion | quote }} - namespace: {{ $storage.namespace }} - {{- if $storage.overrideAllValues }} - values: - {{- toYaml $storage.overrideAllValues | nindent 6 }} - {{- else }} - {{- /* Skip Mayastor's bundled VolumeSnapshotClass CRD install. The CRDs - come from the cluster's snapshot-controller (must be installed - separately — typically already present on EKS Auto Mode or via - another stack). Avoids Helm "ownership" conflicts when more than - one chart bundles the same CRDs. */}} - {{- /* Build csi sub-dict incrementally — both nodePool and HA contribute - to it, so direct `set` would clobber. */}} - {{- $csi := dict }} - {{- if $state.nodePool.enabled }} - {{- /* Confine csi-node to NVMe nodes too — non-psql workers don't have - nvme_tcp loaded, so csi-node would crashloop there. */}} - {{- $_ := set $csi "node" (dict - "nodeSelector" $state.nodePool.nodeSelector - "tolerations" $state.nodePool.tolerations - ) }} - {{- end }} - {{- if $state.ha.enabled }} - {{- $_ := set $csi "controller" (dict "replicaCount" $state.ha.replicas) }} - {{- end }} - - {{- /* Skip Mayastor's bundled VolumeSnapshotClass CRD install. */}} - {{- $chartDefaults := dict - "crds" (dict "csi" (dict "volumeSnapshots" (dict "enabled" false))) - "csi" $csi - }} - {{- if $state.nodePool.enabled }} - {{- $_ := set $chartDefaults "io_engine" (dict - "nodeSelector" $state.nodePool.nodeSelector - "tolerations" $state.nodePool.tolerations - ) }} - {{- end }} - {{- if $state.ha.enabled }} - {{- $_ := set $chartDefaults "agents" (dict "core" (dict "replicaCount" $state.ha.replicas)) }} - {{- $_ := set $chartDefaults "etcd" (dict "replicaCount" $state.ha.replicas) }} - {{- end }} - {{- $mergedValues := mergeOverwrite $chartDefaults ($storage.values | default dict) }} - values: - {{- toYaml $mergedValues | nindent 6 }} - {{- end }} - rollbackLimit: 3 - providerConfigRef: - name: {{ $state.helmProviderConfigRef.name }} - kind: {{ $state.helmProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/170-storageclass-mayastor.yaml.gotmpl b/functions/render/170-storageclass-longhorn.yaml.gotmpl similarity index 60% rename from functions/render/170-storageclass-mayastor.yaml.gotmpl rename to functions/render/170-storageclass-longhorn.yaml.gotmpl index 025f251..1cafd54 100644 --- a/functions/render/170-storageclass-mayastor.yaml.gotmpl +++ b/functions/render/170-storageclass-longhorn.yaml.gotmpl @@ -1,10 +1,12 @@ # code: language=yaml # -# StorageClass + VolumeSnapshotClass — Mayastor (replicated CoW) +# StorageClass + VolumeSnapshotClass — Longhorn V2 (replicated CoW) # -# The stack's only StorageClass. Replicated NVMe-oF, thin-provisioned by -# default. PSQLClusters that don't want CoW can target a different SC the -# cluster already provides; the stack doesn't compose a non-CoW SC. +# The stack's only StorageClass. Backed by Longhorn V2's SPDK data engine +# over NVMe-oF (TCP transport). Replicated by default +# (numberOfReplicas = $storage.replicationFactor); thin-provisioned by default. +# PSQLClusters that don't want CoW can target a different SC the cluster +# already provides; the stack doesn't compose a non-CoW SC. # {{- $storage := $state.storage }} @@ -13,9 +15,9 @@ apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object metadata: - name: {{ $state.name }}-storageclass-mayastor + name: {{ $state.name }}-storageclass-longhorn annotations: - {{ setResourceNameAnnotation "storageclass-mayastor" }} + {{ setResourceNameAnnotation "storageclass-longhorn" }} labels: {{ $state.labels | toJson }} spec: managementPolicies: {{ $state.managementPolicies | toJson }} @@ -26,13 +28,13 @@ spec: metadata: name: {{ $storage.storageClassName }} labels: {{ $state.labels | toJson }} - provisioner: io.openebs.csi-mayastor + provisioner: driver.longhorn.io parameters: - repl: "{{ $storage.replicationFactor }}" - protocol: nvmf - thin: "{{ $storage.thin }}" - ioTimeout: "60" - fsType: ext4 + dataEngine: "v2" + numberOfReplicas: "{{ $storage.replicationFactor }}" + staleReplicaTimeout: "30" + fsType: "ext4" + diskSelector: "psql" reclaimPolicy: {{ $storage.reclaimPolicy }} volumeBindingMode: {{ $storage.volumeBindingMode }} allowVolumeExpansion: {{ $storage.allowVolumeExpansion }} @@ -43,9 +45,9 @@ spec: apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object metadata: - name: {{ $state.name }}-volumesnapshotclass-mayastor + name: {{ $state.name }}-volumesnapshotclass-longhorn annotations: - {{ setResourceNameAnnotation "volumesnapshotclass-mayastor" }} + {{ setResourceNameAnnotation "volumesnapshotclass-longhorn" }} labels: {{ $state.labels | toJson }} spec: managementPolicies: {{ $state.managementPolicies | toJson }} @@ -56,8 +58,10 @@ spec: metadata: name: {{ $storage.storageClassName }} labels: {{ $state.labels | toJson }} - driver: io.openebs.csi-mayastor + driver: driver.longhorn.io deletionPolicy: Delete + parameters: + type: snap providerConfigRef: name: {{ $state.kubernetesProviderConfigRef.name }} kind: {{ $state.kubernetesProviderConfigRef.kind }} diff --git a/tests/test-render/main.k b/tests/test-render/main.k index 2aad6fd..d4da64a 100644 --- a/tests/test-render/main.k +++ b/tests/test-render/main.k @@ -13,7 +13,7 @@ import models.k8s.apimachinery.pkg.apis.meta.v1 as metav1 _items = [ # ========================================================================== # Test 1: minimal claim renders the platform operators (CNPG, Atlas), the - # scale-to-zero plugin (default-on), Mayastor + StorageClass + node-prep + + # scale-to-zero plugin (default-on), Longhorn + StorageClass + node-prep + # both NodePool sub-pools (nodePool default-on now — stack's whole point). # ========================================================================== metav1alpha1.CompositionTest { @@ -41,17 +41,17 @@ _items = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" - metadata.name = "openebs-mayastor" + metadata.name = "longhorn" } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "test-psql-storageclass-mayastor" + metadata.name = "test-psql-storageclass-longhorn" } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "test-psql-volumesnapshotclass-mayastor" + metadata.name = "test-psql-volumesnapshotclass-longhorn" } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" @@ -335,14 +335,14 @@ _items = [ { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "no-s2z-storageclass-mayastor" + metadata.name = "no-s2z-storageclass-longhorn" } ] } } # ========================================================================== - # Test 9: nodePool.enabled=false skips Mayastor, StorageClass, and node-prep + # Test 9: nodePool.enabled=false skips Longhorn, StorageClass, and node-prep # (the stack's storage requires the NodePool to have something to schedule # on — disabling the NodePool reasonably skips everything that needs it). # ========================================================================== @@ -412,8 +412,8 @@ _items = [ } # ========================================================================== - # Test 10: spec.storage settings flow through to the StorageClass - # parameters (replicationFactor, thin) + # Test 10: spec.storage.replicationFactor flows through to the Longhorn + # StorageClass parameters (numberOfReplicas). # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "storage-settings-flow-through-to-sc" @@ -428,7 +428,6 @@ _items = [ clusterName = "my-cluster" storage = { replicationFactor = 1 - thin = False } } } @@ -436,13 +435,13 @@ _items = [ { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "storage-tweaks-storageclass-mayastor" + metadata.name = "storage-tweaks-storageclass-longhorn" spec.forProvider.manifest.parameters = { - repl = "1" - protocol = "nvmf" - thin = "false" - ioTimeout = "60" + dataEngine = "v2" + numberOfReplicas = "1" + staleReplicaTimeout = "30" fsType = "ext4" + diskSelector = "psql" } } ] From 1ec04e6340bdcaf1d1783e741e127ce51029b7a7 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Sat, 25 Apr 2026 15:39:36 -0500 Subject: [PATCH 18/44] feat: pivot psql-stack to EKS Auto Mode defaults MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drops Mayastor/Longhorn experimentation entirely: - Remove NodePool sub-pools (branches + primary) - Remove node-prep DaemonSet - Remove Mayastor / Longhorn Helm release templates - Remove Mayastor/Longhorn StorageClass Slim spec.storage block down to spec.snapshotClass {enabled, name, driver, deletionPolicy, parameters}. Default driver ebs.csi.aws.com (EKS Auto Mode default); override for non-AWS providers. The stack no longer composes a StorageClass — PSQLClusters target whatever SC the cluster already provides. Stack now composes 4 things: CNPG operator, Atlas operator, S2Z plugin (9 Objects), and one VolumeSnapshotClass. CNPG/Atlas/S2Z templates lose nodeSelector/tolerations refs; they run wherever Auto Mode schedules them. XRD shrinks from ~290 lines to ~170. README rewritten — no more NVMe/SPDK/iscsi noise. Examples collapsed to clean shapes. Replicated CoW storage (Longhorn et al) is now a separate concern, to be provided by aws-storage-stack (self-managed ASG nodes with proper userData) when the multi-tenant CoW economics justify the operational cost. Bottlerocket/Auto Mode is incompatible with iscsi- based engines (longhorn-manager env-checks for iscsiadm even with V2; Bottlerocket's immutable rootfs blocks installation). Branch checkpoints preserved on: feat/cnpg-pivot — Mayastor experiment feat/longhorn-pivot — Longhorn V2 experiment --- README.md | 129 ++++----- apis/psqlstacks/definition.yaml | 246 ++++++----------- examples/psqlstacks/local.yaml | 15 +- examples/psqlstacks/minimal.yaml | 15 +- examples/psqlstacks/standard.yaml | 16 +- functions/render/000-state-init.yaml.gotmpl | 159 ++--------- functions/render/010-state-status.yaml.gotmpl | 10 +- .../render/140-nodepool-branches.yaml.gotmpl | 53 ---- .../render/145-nodepool-primary.yaml.gotmpl | 93 ------- .../155-node-prep-daemonset.yaml.gotmpl | 258 ------------------ functions/render/165-longhorn.yaml.gotmpl | 103 ------- .../170-storageclass-longhorn.yaml.gotmpl | 68 ----- .../170-volumesnapshotclass.yaml.gotmpl | 45 +++ .../render/200-cnpg-operator.yaml.gotmpl | 7 +- .../render/210-cnpg-scale-to-zero.yaml.gotmpl | 4 - .../render/220-atlas-operator.yaml.gotmpl | 4 - tests/test-render/main.k | 127 ++++----- 17 files changed, 273 insertions(+), 1079 deletions(-) delete mode 100644 functions/render/140-nodepool-branches.yaml.gotmpl delete mode 100644 functions/render/145-nodepool-primary.yaml.gotmpl delete mode 100644 functions/render/155-node-prep-daemonset.yaml.gotmpl delete mode 100644 functions/render/165-longhorn.yaml.gotmpl delete mode 100644 functions/render/170-storageclass-longhorn.yaml.gotmpl create mode 100644 functions/render/170-volumesnapshotclass.yaml.gotmpl diff --git a/README.md b/README.md index 72a926c..b59d7e1 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,40 @@ # psql-stack -PostgreSQL platform stack on top of [CloudNativePG](https://cloudnative-pg.io/) with [Longhorn](https://github.com/longhorn/longhorn) (V2 data engine — SPDK + NVMe-oF) for replicated CoW storage. Installs the operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), Atlas Operator (schema migrations), Longhorn + matching StorageClass + VolumeSnapshotClass, two Karpenter NodePools, and a node-prep DaemonSet. +PostgreSQL platform layer on top of [CloudNativePG](https://cloudnative-pg.io/). Composes the CNPG operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), the [Atlas Operator](https://atlasgo.io/integrations/kubernetes/operator) (declarative schema migrations), and a `VolumeSnapshotClass` that PSQLBranch uses as a stable forking target. -> **Storage status**: Longhorn V2 is "Experimental" upstream as of Longhorn 1.10. V1 (iSCSI) is GA but not used here — V2 was chosen for SPDK performance parity with Mayastor without Mayastor's heavy footprint. Track promotion to GA before high-stakes production use, or override `spec.storage.values.defaultSettings.v2DataEngine: false` to fall back to V1. +This is the **platform layer** — it does not create any serving Postgres clusters. Per-app DBs live in [`PSQLCluster`](../../psql-cluster/), ephemeral forks in [`PSQLBranch`](../../psql-branch/). -This is the **platform layer** — it does not create any serving Postgres clusters. Per-app DBs live in [`PSQLCluster`](../../psql-cluster/) (separate XR), ephemeral forks in [`PSQLBranch`](../../psql-branch/) (separate XR). +## Design -## Why psql-stack? +The stack is intentionally **OS-agnostic and storage-agnostic**: -**Without psql-stack:** -- Manual Helm installs of CNPG, Atlas, Longhorn on every cluster -- No deletion ordering — removing CNPG before Atlas can leave migrations dangling -- No node-side prep for Longhorn V2 (hugepages + `nvme_tcp` / `vfio_pci` / `uio_pci_generic` / `ublk_drv` kernel modules need to exist before the SPDK instance-manager pods will run) -- No declarative substrate for the `cnpg-i-scale-to-zero` plugin (cert-manager-backed gRPC TLS, ServiceAccount + RBAC, sidecar config) +- **No StorageClass.** PSQLClusters target whatever StorageClass the cluster already provides (`gp3` on EKS Auto Mode, `standard` on kind/k3d, etc.). +- **No NodePool / node-prep.** Components run wherever the cluster's scheduler puts them. Auto Mode handles node provisioning end-to-end. +- **Single composed snapshot target.** The stack ships a `VolumeSnapshotClass` named `psql` so PSQLBranch can request snapshots without leaking driver-specific knowledge into PSQLBranch's spec. Default driver is `ebs.csi.aws.com` (EKS Auto Mode default); override for non-AWS clusters. -**With psql-stack:** -- Single claim deploys CNPG + S2Z plugin + Atlas + Longhorn + StorageClass with production defaults -- Deletion order enforced via `protection.crossplane.io/Usage` resources -- Replicated SPDK / NVMe-oF CoW storage — `replicationFactor: 3` for primaries, `replicationFactor: 1` for ephemeral branches (per-PVC override). Single backend covers both uses. -- Dedicated Karpenter NodePools (both spot — Longhorn replication absorbs preemption) targeting NVMe instance-store nodes (`i4g.2xlarge` / `i4g.4xlarge` / `im4gn.2xlarge` arm64 Graviton) -- Pinnable upstream chart / plugin versions; Renovate keeps them current +If you need replicated CoW storage (true block-level branches with delta-only economics), that's a separate concern — see `aws-storage-stack` for self-managed nodes that can host Longhorn or similar. The default psql-stack stays on the AWS-blessed Auto Mode path. ## Components | Component | Default | Purpose | |---|---|---| | **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | -| **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. Pinnable via `spec.scaleToZeroPlugin.version`. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | +| **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | | **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | -| **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas of every HA-able platform component + `topologySpreadConstraints` by zone. Affects CNPG operator, Atlas, S2Z plugin, Longhorn CSI controllers + UI. | -| **Karpenter NodePools** | on (`spec.nodePool.enabled: true`) | `branches` (spot arm64 NVMe) and `primary` (spot arm64 NVMe — Longhorn replication absorbs preemption). | -| **Longhorn (V2)** | on when `nodePool.enabled` | Replicated SPDK / NVMe-oF storage with CoW snapshots. Single `psql` StorageClass + matching VolumeSnapshotClass. Longhorn keeps state in CRDs — no bundled etcd / NATS / minio. | -| **node-prep DaemonSet** | on when `nodePool.enabled` | Configures hugepages + loads `nvme_tcp` / `vfio_pci` / `uio_pci_generic` / `ublk_drv` kernel modules on each NVMe node (Longhorn V2 prereqs); annotates the K8s Node so Longhorn auto-registers the local instance-store device as a V2 block-mode disk tagged `psql`. | +| **VolumeSnapshotClass** | on (`spec.snapshotClass.enabled: true`) | Named `psql` by default. Driver: `ebs.csi.aws.com`. PSQLBranch references this name. | +| **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas + `topologySpreadConstraints` by zone on CNPG, Atlas, S2Z plugin. | ## Prerequisites -- **cert-manager** must be installed on the target cluster (the S2Z plugin uses cert-manager `Issuer` + `Certificate`s for its gRPC TLS material). Install via [`aws-cert-stack`](../../aws/cert/). -- **Karpenter + EKS Auto Mode** for the NodePools (the default `nodeClassName: default` references Auto Mode's managed `NodeClass`). +- **A working CSI driver + StorageClass** on the cluster. EKS Auto Mode provides `gp3` + `ebs.csi.aws.com` automatically. For kind/k3d, the bundled `standard` SC works. +- **VolumeSnapshot CRDs** (snapshot.storage.k8s.io). EKS Auto Mode includes the snapshot-controller; for self-managed clusters install it from [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter). +- **cert-manager** (only when `scaleToZeroPlugin.enabled` — the plugin uses cert-manager Issuer+Certificate for its gRPC TLS). Provided by [`aws-cert-stack`](../../aws/cert/). -## The Journey +## Stages ### Stage 1: Default install -Deploy with all defaults: CNPG + Atlas + S2Z plugin + Longhorn + dedicated NodePools. Karpenter provisions NVMe instance-store nodes when something requests them. +Deploy with all defaults. CNPG + Atlas + S2Z + a `psql` VolumeSnapshotClass for EBS. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -54,9 +46,9 @@ spec: clusterName: my-cluster ``` -### Stage 2: Production sizing +### Stage 2: Production posture -Tune NodePool limits, override Helm values, label for cost allocation. +HA on; per-component value tweaks; team labels for cost allocation. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -69,23 +61,36 @@ spec: namespace: cnpg-system labels: team: platform - nodePool: + ha: enabled: true - branches: - limits: { cpu: "32", memory: "128Gi" } - primary: - limits: { cpu: "64", memory: "256Gi" } - storage: - replicationFactor: 3 - thin: true + replicas: 3 + topologySpreadByZone: true atlasOperator: values: prewarmDevDB: true ``` -### Stage 3: Local / no-NVMe clusters +### Stage 3: Non-AWS / non-EBS cluster -For kind / k3d / clusters that can't run Longhorn V2, disable the NodePool. The stack then ships only CNPG + Atlas + S2Z plugin, and your PSQLClusters target whatever StorageClass the cluster provides. +Override the snapshot driver (e.g. for a self-managed cluster running Longhorn, or a local cluster using hostpath CSI). + +```yaml +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLStack +metadata: + name: psql + namespace: default +spec: + clusterName: edge + helmProviderConfigRef: + name: default + snapshotClass: + driver: driver.longhorn.io +``` + +### Stage 4: Local / no-snapshot cluster + +For dev clusters without a snapshot-controller, disable the VSC composition. PSQLBranch won't function (it needs the VSC), but PSQLCluster still works. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -97,7 +102,9 @@ spec: clusterName: local helmProviderConfigRef: name: default - nodePool: + snapshotClass: + enabled: false + scaleToZeroPlugin: enabled: false ``` @@ -105,7 +112,7 @@ spec: | Field | Type | Default | Description | |---|---|---|---| -| `clusterName` | string | _required_ | Target cluster name; default for `helmProviderConfigRef.name` and resource naming | +| `clusterName` | string | _required_ | Target cluster name; default for `helmProviderConfigRef.name`, `kubernetesProviderConfigRef.name`, and label values | | `namespace` | string | `cnpg-system` | Shared namespace for CNPG, S2Z plugin, and Atlas | | `labels` | object | — | Custom labels merged with stack defaults | | `managementPolicies` | string[] | `["*"]` | Crossplane management policies | @@ -113,6 +120,10 @@ spec: | `helmProviderConfigRef.kind` | enum | `ProviderConfig` | `ProviderConfig` or `ClusterProviderConfig` | | `kubernetesProviderConfigRef.name` | string | `clusterName` | Kubernetes ProviderConfig name | | `kubernetesProviderConfigRef.kind` | enum | `ProviderConfig` | Same as above | +| **HA mode** | | | | +| `ha.enabled` | bool | `false` | Stack-wide HA toggle | +| `ha.replicas` | int | `3` | Replica count for HA-able platform components | +| `ha.topologySpreadByZone` | bool | `true` | Add `topologySpreadConstraint` with `topologyKey=topology.kubernetes.io/zone`, `maxSkew=1`, `whenUnsatisfiable=ScheduleAnyway` | | **CNPG operator** | | | | | `cnpg.name` | string | `cloudnative-pg` | Helm release name | | `cnpg.chartVersion` | string | `0.27.1` | CNPG Helm chart version (tracks CNPG 1.29.x) | @@ -127,33 +138,12 @@ spec: | `atlasOperator.namespace` | string | shared `namespace` | Override | | `atlasOperator.values` | object | — | Helm values merged with chart defaults | | `atlasOperator.overrideAllValues` | object | — | Helm values that replace all defaults | -| **HA mode** | | | | -| `ha.enabled` | bool | `false` | Stack-wide HA toggle. When true, sets replicaCount + topology spread by zone on CNPG, Atlas, S2Z plugin, Mayastor control plane components. | -| `ha.replicas` | int | `3` | Replica count for HA-able platform components | -| `ha.topologySpreadByZone` | bool | `true` | Add topologySpreadConstraint with topologyKey=topology.kubernetes.io/zone, maxSkew=1, whenUnsatisfiable=ScheduleAnyway | -| **NodePool** | | | | -| `nodePool.enabled` | bool | `true` | Master toggle. When false, Longhorn + StorageClass + node-prep are skipped too. | -| `nodePool.nodeClassName` | string | `default` | EKS NodeClass referenced by both sub-pools | -| `nodePool.disruption.consolidationPolicy` | enum | `WhenEmptyOrUnderutilized` | Karpenter consolidation policy | -| `nodePool.disruption.consolidateAfter` | string | `60s` | Consolidation delay | -| `nodePool.branches.enabled` | bool | `true` | Spot arm64 NVMe sub-pool | -| `nodePool.branches.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | -| `nodePool.branches.requirements` | array | arm64 spot `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | -| `nodePool.primary.enabled` | bool | `true` | Spot arm64 NVMe sub-pool | -| `nodePool.primary.limits` | object | `{cpu: "32", memory: "128Gi"}` | Sub-pool limits | -| `nodePool.primary.requirements` | array | arm64 spot `i4g.2xlarge`/`i4g.4xlarge`/`im4gn.2xlarge` | Karpenter requirements | -| **Storage (Longhorn V2)** | | | | -| `storage.chartVersion` | string | `1.10.0` | Longhorn Helm chart version | -| `storage.namespace` | string | `longhorn-system` | Helm release namespace | -| `storage.storageClassName` | string | `psql` | StorageClass name | -| `storage.replicationFactor` | int | `3` | Default replicas per volume (`numberOfReplicas` parameter on the SC) | -| `storage.thin` | bool | `true` | Thin-provisioning (CoW). Longhorn V2 is CoW by default; the flag is exposed for symmetry but is essentially always true under V2. | -| `storage.values` | object | — | Helm values merged with chart defaults | -| `storage.overrideAllValues` | object | — | Helm values that replace all defaults | -| **Node prep** | | | | -| `nodePrep.enabled` | bool | `true` (when `nodePool.enabled`) | Compose the privileged DaemonSet | -| `nodePrep.hugepages.count` | int | `1024` | Number of 2MiB hugepages per node (Longhorn V2 SPDK requires) | -| `nodePrep.image` | string | `alpine:3.20` | Init container image (apk-install kubectl + util-linux at runtime) | +| **Snapshot class** | | | | +| `snapshotClass.enabled` | bool | `true` | Compose the VolumeSnapshotClass | +| `snapshotClass.name` | string | `psql` | VolumeSnapshotClass name (PSQLBranch references this) | +| `snapshotClass.driver` | string | `ebs.csi.aws.com` | CSI driver | +| `snapshotClass.deletionPolicy` | enum | `Delete` | `Delete` or `Retain` | +| `snapshotClass.parameters` | object | — | Driver-specific parameters | ## Composed Resources @@ -161,13 +151,8 @@ spec: |---|---|---| | `cloudnative-pg` | `helm.m.crossplane.io/Release` | always | | `atlas-operator` | `helm.m.crossplane.io/Release` | always | -| 9× `-s2z-*` | `kubernetes.m.crossplane.io/Object` | `scaleToZeroPlugin.enabled: true` (default) | -| `longhorn` | `helm.m.crossplane.io/Release` | `nodePool.enabled: true` (default) | -| `-storageclass-longhorn` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | -| `-volumesnapshotclass-longhorn` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled: true` | -| `-nodepool-branches` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled && nodePool.branches.enabled` | -| `-nodepool-primary` | `kubernetes.m.crossplane.io/Object` | `nodePool.enabled && nodePool.primary.enabled` | -| `-node-prep` | `kubernetes.m.crossplane.io/Object` (DaemonSet) | `nodePool.enabled && nodePrep.enabled` | +| 9× `-s2z-*` | `kubernetes.m.crossplane.io/Object` | `scaleToZeroPlugin.enabled: true` | +| `-volumesnapshotclass` | `kubernetes.m.crossplane.io/Object` | `snapshotClass.enabled: true` | | Various `Usage` | `protection.crossplane.io/Usage` | when both ends Ready (deletion ordering) | ## Dependencies diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 6e69412..0491148 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -14,7 +14,16 @@ spec: served: true schema: openAPIV3Schema: - description: PSQLStack deploys CloudNativePG, the cnpg-i-scale-to-zero plugin, and the Atlas Operator. It's the platform layer — per-app serving clusters live in PSQLCluster; ephemeral forks live in PSQLBranch. + description: | + PSQLStack is the platform layer for Postgres-on-Kubernetes. It composes + CloudNativePG (the operator), the cnpg-i-scale-to-zero plugin, the Atlas + Operator (declarative schema migrations), and a VolumeSnapshotClass that + PSQLBranch uses as a stable target for forking primaries. + + The stack is intentionally OS-agnostic and storage-agnostic — it relies + on whatever CSI driver and StorageClass the target cluster already + provides (e.g., gp3 on EKS Auto Mode, standard on kind/k3d). Per-app + serving clusters live in PSQLCluster; ephemeral forks in PSQLBranch. type: object properties: spec: @@ -22,206 +31,65 @@ spec: type: object properties: clusterName: - description: Name of the target cluster. Used as default for helmProviderConfigRef.name and resource naming. + description: Name of the target cluster. Used as the default for helmProviderConfigRef.name, kubernetesProviderConfigRef.name, and label values. type: string + namespace: + description: Shared namespace for the CNPG operator, scale-to-zero plugin, and Atlas operator. Per-component namespace overrides take precedence. Defaults to cnpg-system. + type: string + labels: + description: Custom labels merged with the stack's default labels and applied to every composed resource. + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true managementPolicies: - description: Management policies for all composed resources. Defaults to ["*"]. + description: Crossplane managementPolicies applied to every composed resource. Defaults to ["*"]. type: array items: type: string default: - "*" - labels: - description: Custom labels applied to all managed resources. - type: object - additionalProperties: - type: string - x-kubernetes-preserve-unknown-fields: true helmProviderConfigRef: - description: Reference to the Helm ProviderConfig. Defaults to clusterName. + description: Reference to the Helm ProviderConfig used for CNPG and Atlas releases. Defaults to clusterName. type: object properties: name: - description: Name of the Helm ProviderConfig. type: string kind: - description: Kind of the ProviderConfig. Defaults to ProviderConfig. type: string enum: - ProviderConfig - ClusterProviderConfig kubernetesProviderConfigRef: - description: Reference to the Kubernetes ProviderConfig (used to create the Karpenter NodePool Object and to apply the scale-to-zero plugin manifest). Defaults to clusterName. + description: Reference to the Kubernetes ProviderConfig used for the scale-to-zero plugin manifests and the VolumeSnapshotClass. Defaults to clusterName. type: object properties: name: - description: Name of the Kubernetes ProviderConfig. type: string kind: - description: Kind of the ProviderConfig. Defaults to ProviderConfig. type: string enum: - ProviderConfig - ClusterProviderConfig - namespace: - description: Shared namespace for CNPG operator, scale-to-zero plugin, and Atlas operator. Defaults to cnpg-system. Per-component namespace overrides this. - type: string - storage: - description: "Storage backend (Longhorn V2 — SPDK + NVMe-oF with replicated CoW snapshots). The stack is opinionated: it ships exactly one storage backend, sized for CoW + HA. Per-cluster cost is controlled via replicationFactor (3 for primaries, 1 for ephemeral branches). Requires NVMe instance-store nodes (provided by the NodePool block) plus hugepages and the nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv kernel modules (provided by the node-prep DaemonSet). Longhorn V2 is marked Experimental upstream (Longhorn 1.10) — track promotion to GA before high-stakes production use. For non-CoW workloads, PSQLCluster claims can target any other StorageClass that exists on the target cluster (e.g., the cluster's default gp3 SC) — no need for the stack to compose that." - type: object - properties: - chartVersion: - description: Longhorn Helm chart version. Defaults to 1.10.0. - type: string - namespace: - description: Namespace for the Longhorn install. Defaults to longhorn-system. - type: string - storageClassName: - description: StorageClass name. Defaults to "psql". - type: string - replicationFactor: - description: Default replication factor. Defaults to 3 (quorum-safe HA). Per-PVC override via the StorageClass parameters. - type: integer - default: 3 - thin: - description: Thin-provision volumes (CoW). Defaults to true. - type: boolean - default: true - reclaimPolicy: - type: string - enum: [Delete, Retain] - default: Delete - volumeBindingMode: - type: string - enum: [Immediate, WaitForFirstConsumer] - default: WaitForFirstConsumer - allowVolumeExpansion: - type: boolean - default: true - values: - type: object - x-kubernetes-preserve-unknown-fields: true - overrideAllValues: - type: object - x-kubernetes-preserve-unknown-fields: true - nodePool: - description: "Dedicated Karpenter NodePools for PostgreSQL workloads. Two sub-pools — `branches` (spot arm64 NVMe, for ephemeral PSQLBranch workloads) and `primary` (spot arm64 NVMe, for replicated-CoW PSQLCluster primaries — Longhorn V2 replication absorbs spot preemption). Defaults to enabled — node-side prep for Longhorn V2 (hugepages + nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv) is composed automatically by the node-prep DaemonSet. CNPG / scale-to-zero / Atlas schedule on the primary pool when enabled." - type: object - properties: - enabled: - description: Master toggle. When false, neither sub-pool is created and Mayastor / node-prep are skipped (the stack would have nothing to schedule on otherwise). Defaults to true — provisioning NVMe nodes is the stack's whole point. - type: boolean - default: true - nodeClassName: - description: EKS NodeClass shared by both sub-pools. Defaults to "default". - type: string - disruption: - description: Karpenter disruption policy applied to both sub-pools. - type: object - properties: - consolidationPolicy: - type: string - enum: [WhenEmpty, WhenEmptyOrUnderutilized] - default: WhenEmptyOrUnderutilized - consolidateAfter: - type: string - default: "60s" - branches: - description: Spot arm64 NVMe sub-pool for branches and dev workloads. Cost-optimized; spot preemption is acceptable because branches are ephemeral. - type: object - properties: - enabled: - type: boolean - default: true - limits: - type: object - properties: - cpu: - type: string - memory: - type: string - requirements: - description: Karpenter scheduling requirements. Defaults to arm64 spot on i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge (NVMe instance-store, Graviton). - type: array - items: - type: object - properties: - key: - type: string - operator: - type: string - enum: [In, NotIn, Exists, DoesNotExist, Gt, Lt] - values: - type: array - items: - type: string - required: [key, operator] - primary: - description: "Spot arm64 NVMe sub-pool for PSQLCluster primaries. Longhorn V2's replicationFactor=3 absorbs spot preemption — losing one replica triggers a rebuild on a fresh node, not data loss. Override capacity-type to on-demand via spec.nodePool.primary.requirements if your durability requirements rule out preemption." - type: object - properties: - enabled: - type: boolean - default: true - limits: - type: object - properties: - cpu: - type: string - memory: - type: string - requirements: - description: Karpenter scheduling requirements. Defaults to arm64 spot on i4g.2xlarge / i4g.4xlarge / im4gn.2xlarge (NVMe instance-store, Graviton). Set capacity-type to on-demand if Mayastor replication isn't enough for your durability bar. - type: array - items: - type: object - properties: - key: - type: string - operator: - type: string - enum: [In, NotIn, Exists, DoesNotExist, Gt, Lt] - values: - type: array - items: - type: string - required: [key, operator] - nodePrep: - description: "Node-prep DaemonSet for Longhorn V2 prerequisites. Runs as a privileged init container on each NVMe NodePool node and configures hugepages (vm.nr_hugepages, required by SPDK) plus loads the nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv kernel modules (Longhorn V2 NVMe-oF + UBLK transports). Also annotates the K8s Node with `node.longhorn.io/default-disks-config` so Longhorn auto-registers the local NVMe instance-store device as a V2 block-mode disk. Composed when nodePool.enabled." - type: object - properties: - enabled: - description: Whether to compose the node-prep DaemonSet. Defaults to true (auto-gated by nodePool.enabled — no NVMe nodes means no prep needed). - type: boolean - default: true - hugepages: - type: object - properties: - count: - description: Number of 2MiB hugepages to allocate on each node. Defaults to 1024 (2GiB total). Required by Longhorn V2 SPDK. - type: integer - default: 1024 - image: - description: Container image for the prep init container. Must have apk (for kubectl + util-linux install). Defaults to alpine:3.20. - type: string ha: - description: "Stack-wide HA mode. When enabled, sets replicaCount + topology spread (one replica per zone) on all platform components: CNPG operator, Atlas operator, scale-to-zero plugin, Longhorn control plane. Per-component values can still be overridden via each component's values block." + description: | + Stack-wide HA mode. When enabled, sets replicaCount + topology spread + by zone on every HA-able platform component (CNPG operator, Atlas + operator, scale-to-zero plugin Deployment). Per-component values + blocks can still override. type: object properties: enabled: - description: Master toggle. Default false. Flip true for production. type: boolean default: false replicas: - description: Replica count for HA-able platform components. Defaults to 3. type: integer default: 3 topologySpreadByZone: - description: Add a topologySpreadConstraint with topologyKey topology.kubernetes.io/zone, maxSkew 1, whenUnsatisfiable ScheduleAnyway. Defaults to true. type: boolean default: true cnpg: - description: Configuration for the CloudNativePG operator Helm release. + description: CloudNativePG operator Helm release configuration. type: object properties: name: @@ -234,7 +102,7 @@ spec: description: CNPG Helm chart version. Defaults to 0.27.1 (tracks CNPG 1.29.x). type: string values: - description: Helm values merged with chart defaults. + description: Helm values merged with the stack's chart defaults. type: object x-kubernetes-preserve-unknown-fields: true overrideAllValues: @@ -242,21 +110,24 @@ spec: type: object x-kubernetes-preserve-unknown-fields: true scaleToZeroPlugin: - description: "Configuration for the cnpg-i-scale-to-zero plugin (github.com/xataio/cnpg-i-scale-to-zero). Installed via upstream release manifest. Zero cost when no PSQLCluster opts into scale-to-zero." + description: | + cnpg-i-scale-to-zero plugin (github.com/xataio/cnpg-i-scale-to-zero). + Installed via the upstream release manifest. Zero cost when no + PSQLCluster opts into scale-to-zero. Requires cert-manager on the + target cluster (provided by aws-cert-stack). type: object properties: enabled: - description: Default true — the plugin is zero-cost when unused and per-cluster opt-in is cheap. type: boolean default: true version: - description: Plugin release tag. The manifest URL pins to this. Defaults to v0.1.7. + description: Plugin release tag. Defaults to v0.1.7. type: string namespace: - description: Namespace override for the plugin install. Defaults to cnpg-system (plugin manifest creates the namespace if missing). + description: Namespace override. Defaults to the shared namespace. type: string atlasOperator: - description: Configuration for the Atlas Operator component (declarative database schema migrations). + description: Atlas Operator Helm release configuration. Manages declarative database schema migrations. type: object properties: name: @@ -266,13 +137,50 @@ spec: description: Namespace override for this component. type: string values: - description: Helm values merged with chart defaults. + description: Helm values merged with the stack's chart defaults. type: object x-kubernetes-preserve-unknown-fields: true overrideAllValues: description: Helm values that replace all defaults for this component. type: object x-kubernetes-preserve-unknown-fields: true + snapshotClass: + description: | + VolumeSnapshotClass used by PSQLBranch as its forking target. The + stack composes exactly one VolumeSnapshotClass named after the XR + (default: "psql"). On EKS Auto Mode the default driver + (ebs.csi.aws.com) is correct out of the box; override `driver` for + other CSI providers (e.g. driver.longhorn.io, hostpath.csi.k8s.io). + PSQLClusters target whatever StorageClass the cluster already + provides (Auto Mode ships gp3 by default) — the stack does not + compose a StorageClass. + type: object + properties: + enabled: + description: Compose the VolumeSnapshotClass. Defaults to true. + type: boolean + default: true + name: + description: VolumeSnapshotClass name. Defaults to "psql" — PSQLBranch references this name. + type: string + default: psql + driver: + description: CSI driver. Defaults to ebs.csi.aws.com (EKS Auto Mode default). + type: string + default: ebs.csi.aws.com + deletionPolicy: + description: Snapshot deletion policy. Defaults to Delete. + type: string + enum: + - Delete + - Retain + default: Delete + parameters: + description: Driver-specific parameters passed to the VolumeSnapshotClass. + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true required: - clusterName status: @@ -280,7 +188,7 @@ spec: type: object properties: ready: - description: Overall readiness of the stack (CNPG + scale-to-zero plugin + Atlas). + description: Overall readiness — true once CNPG, Atlas, the scale-to-zero plugin, and the VolumeSnapshotClass are all Ready. type: boolean required: - spec diff --git a/examples/psqlstacks/local.yaml b/examples/psqlstacks/local.yaml index 385f0a9..115aed5 100644 --- a/examples/psqlstacks/local.yaml +++ b/examples/psqlstacks/local.yaml @@ -1,9 +1,8 @@ -# Local: NodePool / Longhorn disabled — gets just CNPG + Atlas + S2Z plugin -# installed on whatever StorageClass already exists on the local cluster. +# Local: dev cluster without a CSI snapshot controller. # -# Useful for kind / k3d dev clusters that can't run Longhorn V2. -# PSQLClusters on this stack will need to specify a non-Longhorn SC -# (`spec.storage.class: standard` for k3d, etc.). +# Disables the VolumeSnapshotClass composition (PSQLBranch won't work but +# PSQLCluster still does), and points at a local Helm/Kubernetes +# ProviderConfig instead of the per-cluster ProviderConfig pattern. # apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack @@ -14,5 +13,9 @@ spec: clusterName: local helmProviderConfigRef: name: default - nodePool: + kubernetesProviderConfigRef: + name: default + snapshotClass: + enabled: false + scaleToZeroPlugin: enabled: false diff --git a/examples/psqlstacks/minimal.yaml b/examples/psqlstacks/minimal.yaml index 1baead3..1bc55ad 100644 --- a/examples/psqlstacks/minimal.yaml +++ b/examples/psqlstacks/minimal.yaml @@ -2,18 +2,13 @@ # # Composes: # - CloudNativePG operator (Helm) -# - cnpg-i-scale-to-zero plugin (set of Objects) +# - cnpg-i-scale-to-zero plugin (set of Objects, requires cert-manager) # - Atlas operator (Helm) -# - Karpenter NodePool sub-pools: branches (spot) + primary (spot) -# - Longhorn V2 (Helm) on the primary sub-pool nodes -# - psql StorageClass + VolumeSnapshotClass (Longhorn-backed, replicated, CoW) -# - node-prep DaemonSet (hugepages + nvme_tcp / vfio_pci / uio_pci_generic -# / ublk_drv on each NVMe node, plus Longhorn disk auto-registration) +# - VolumeSnapshotClass `psql` (driver: ebs.csi.aws.com) # -# This is the stack's whole point: CoW Postgres on dedicated NVMe nodes. -# To opt out of NVMe / Longhorn entirely (and just get CNPG + Atlas + S2Z -# on the cluster's existing default StorageClass), set nodePool.enabled: false -# and have your PSQLClusters target a different SC. +# PSQLClusters target the cluster's existing default StorageClass +# (gp3 on EKS Auto Mode, standard on kind/k3d, etc.). PSQLBranch +# references the `psql` VolumeSnapshotClass for forking primaries. # apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack diff --git a/examples/psqlstacks/standard.yaml b/examples/psqlstacks/standard.yaml index 4f9e79a..3a0d102 100644 --- a/examples/psqlstacks/standard.yaml +++ b/examples/psqlstacks/standard.yaml @@ -1,5 +1,4 @@ -# Standard: production posture with explicit overrides for sizing, -# scheduling, and chart values. +# Standard: production posture with HA + atlas tuning + team labels. # apiVersion: hops.ops.com.ai/v1alpha1 kind: PSQLStack @@ -11,21 +10,10 @@ spec: namespace: cnpg-system labels: team: platform - nodePool: - enabled: true - branches: - enabled: true - limits: { cpu: "32", memory: "128Gi" } - primary: - enabled: true - limits: { cpu: "64", memory: "256Gi" } - storage: - replicationFactor: 3 - thin: true ha: enabled: true # 3 replicas + topology spread by zone on replicas: 3 # all platform components (CNPG, Atlas, - topologySpreadByZone: true # S2Z plugin, Longhorn CSI controllers) + topologySpreadByZone: true # S2Z plugin Deployment). scaleToZeroPlugin: enabled: true atlasOperator: diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 59e452d..8863e6f 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -37,122 +37,8 @@ }} # ============================================================================== -# Storage — Longhorn V2 (replicated SPDK + NVMe-oF + CoW) -# Single backend; per-PVC cost is controlled via replicationFactor. Requires -# NVMe instance-store nodes (NodePool block) plus hugepages + nvme_tcp / -# vfio_pci / uio_pci_generic / ublk_drv modules (node-prep DaemonSet). -# Longhorn V2 is Experimental upstream as of Longhorn 1.10. -# PSQLClusters that don't want CoW can target any other StorageClass that -# exists on the cluster (e.g., the default gp3 SC) — the stack doesn't -# compose a non-CoW SC itself. -# ============================================================================== -{{- $storageSpec := $spec.storage | default dict }} -{{- $thin := true }} -{{- if hasKey $storageSpec "thin" }} - {{- $thin = $storageSpec.thin }} -{{- end }} -{{- $allowVolumeExpansion := true }} -{{- if hasKey $storageSpec "allowVolumeExpansion" }} - {{- $allowVolumeExpansion = $storageSpec.allowVolumeExpansion }} -{{- end }} -{{- $storage := dict - "chartVersion" ($storageSpec.chartVersion | default "1.10.0") - "namespace" ($storageSpec.namespace | default "longhorn-system") - "storageClassName" ($storageSpec.storageClassName | default "psql") - "replicationFactor" ($storageSpec.replicationFactor | default 3) - "thin" $thin - "reclaimPolicy" ($storageSpec.reclaimPolicy | default "Delete") - "volumeBindingMode" ($storageSpec.volumeBindingMode | default "WaitForFirstConsumer") - "allowVolumeExpansion" $allowVolumeExpansion - "values" ($storageSpec.values | default dict) - "overrideAllValues" ($storageSpec.overrideAllValues | default dict) -}} - -# ============================================================================== -# NodePool — branches + primary sub-pools (NVMe arm64 Graviton) -# Defaults to enabled=false; existing claims unchanged. When enabled, both -# sub-pools materialize unless individually disabled. -# -# Common scheduling: nodeSelector workload-type=psql, taint psql=true:NoSchedule. -# Each pool adds a sub-label (sub-pool=branches | sub-pool=primary) and matching -# taint (sub-pool=branches:NoSchedule | sub-pool=primary:NoSchedule) so workloads -# can target one pool specifically. -# ============================================================================== -{{- $nodePoolSpec := $spec.nodePool | default dict }} -{{- $nodePoolEnabled := true }} -{{- if hasKey $nodePoolSpec "enabled" }} - {{- $nodePoolEnabled = $nodePoolSpec.enabled }} -{{- end }} -{{- $nodePoolNodeClassName := $nodePoolSpec.nodeClassName | default "default" }} -{{- $nodePoolDisruption := $nodePoolSpec.disruption | default (dict "consolidationPolicy" "WhenEmptyOrUnderutilized" "consolidateAfter" "60s") }} - -# Default NVMe arm64 instance types -{{- $nvmeInstanceTypes := list "i4g.2xlarge" "i4g.4xlarge" "im4gn.2xlarge" }} - -# branches sub-pool (spot) -{{- $branchesSpec := $nodePoolSpec.branches | default dict }} -{{- $branchesEnabled := true }} -{{- if hasKey $branchesSpec "enabled" }} - {{- $branchesEnabled = $branchesSpec.enabled }} -{{- end }} -{{- $branchesLimits := $branchesSpec.limits | default (dict "cpu" "32" "memory" "128Gi") }} -{{- $branchesRequirements := $branchesSpec.requirements | default (list - (dict "key" "kubernetes.io/arch" "operator" "In" "values" (list "arm64")) - (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "spot")) - (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) -) }} - -# primary sub-pool (spot by default — Longhorn V2's replication absorbs -# preemption; losing one replica of three triggers a rebuild on a fresh -# node, not data loss) -{{- $primarySpec := $nodePoolSpec.primary | default dict }} -{{- $primaryEnabled := true }} -{{- if hasKey $primarySpec "enabled" }} - {{- $primaryEnabled = $primarySpec.enabled }} -{{- end }} -{{- $primaryLimits := $primarySpec.limits | default (dict "cpu" "32" "memory" "128Gi") }} -{{- $primaryRequirements := $primarySpec.requirements | default (list - (dict "key" "kubernetes.io/arch" "operator" "In" "values" (list "arm64")) - (dict "key" "karpenter.sh/capacity-type" "operator" "In" "values" (list "spot")) - (dict "key" "node.kubernetes.io/instance-type" "operator" "In" "values" $nvmeInstanceTypes) -) }} - -# Operator scheduling — operators (CNPG, Atlas, scale-to-zero) ride the primary -# pool when enabled (more reliable than spot for control-plane components) -{{- $nodePoolNodeSelector := dict }} -{{- $nodePoolTolerations := list }} -{{- if and $nodePoolEnabled $primaryEnabled }} - {{- $nodePoolNodeSelector = dict "workload-type" "psql" "sub-pool" "primary" }} - {{- $nodePoolTolerations = list - (dict "key" "psql" "value" "true" "effect" "NoSchedule") - (dict "key" "sub-pool" "value" "primary" "effect" "NoSchedule") - }} -{{- end }} - -# ============================================================================== -# Node prep — hugepages + nvme_tcp / vfio_pci / uio_pci_generic / ublk_drv -# kernel modules on NVMe nodes (Longhorn V2 prereqs); also annotates the -# K8s Node with node.longhorn.io/default-disks-config so Longhorn registers -# the local NVMe instance-store device as a V2 block-mode disk. -# Auto-gated: emitted when nodePool.enabled (no nodes → DS no-op anyway) -# ============================================================================== -{{- $nodePrepSpec := $spec.nodePrep | default dict }} -{{- $nodePrepEnabled := true }} -{{- if hasKey $nodePrepSpec "enabled" }} - {{- $nodePrepEnabled = $nodePrepSpec.enabled }} -{{- end }} -{{- $nodePrep := dict - "enabled" (and $nodePrepEnabled $nodePoolEnabled) - "hugepages" (dict - "count" (((($nodePrepSpec.hugepages | default dict)).count) | default 1024) - ) - "image" ($nodePrepSpec.image | default "alpine:3.20") -}} - -# ============================================================================== -# HA mode — stack-wide HA defaults injected into each platform component's -# Helm values (or our directly-composed Deployment specs) when enabled. -# Per-component overrides can still win via the component's values block. +# HA mode — stack-wide replica + topology-spread defaults injected into each +# platform component. Per-component values blocks can still override. # ============================================================================== {{- $haSpec := $spec.ha | default dict }} {{- $haEnabled := false }} @@ -181,6 +67,25 @@ {{- end }} {{- $atlas := $spec.atlasOperator | default dict }} +# ============================================================================== +# VolumeSnapshotClass — the stack's only storage-layer composition. +# PSQLBranch targets this VSC by name when forking primaries. Default driver +# is ebs.csi.aws.com (correct for EKS Auto Mode); override for other CSI +# providers via $spec.snapshotClass.driver. +# ============================================================================== +{{- $snapshotSpec := $spec.snapshotClass | default dict }} +{{- $snapshotEnabled := true }} +{{- if hasKey $snapshotSpec "enabled" }} + {{- $snapshotEnabled = $snapshotSpec.enabled }} +{{- end }} +{{- $snapshotClass := dict + "enabled" $snapshotEnabled + "name" ($snapshotSpec.name | default "psql") + "driver" ($snapshotSpec.driver | default "ebs.csi.aws.com") + "deletionPolicy" ($snapshotSpec.deletionPolicy | default "Delete") + "parameters" ($snapshotSpec.parameters | default dict) +}} + # ============================================================================== # Initialize $state # ============================================================================== @@ -192,27 +97,6 @@ "labels" $labels "helmProviderConfigRef" $helmProviderConfigRef "kubernetesProviderConfigRef" $k8sProviderConfigRef - "storage" $storage - "nodePool" (dict - "enabled" $nodePoolEnabled - "nodeClassName" $nodePoolNodeClassName - "disruption" $nodePoolDisruption - "nodeSelector" $nodePoolNodeSelector - "tolerations" $nodePoolTolerations - "branches" (dict - "enabled" $branchesEnabled - "name" (printf "%s-branches" $name) - "limits" $branchesLimits - "requirements" $branchesRequirements - ) - "primary" (dict - "enabled" $primaryEnabled - "name" (printf "%s-primary" $name) - "limits" $primaryLimits - "requirements" $primaryRequirements - ) - ) - "nodePrep" $nodePrep "ha" $ha "cnpg" (dict "name" ($cnpg.name | default "cloudnative-pg") @@ -232,6 +116,7 @@ "values" ($atlas.values | default dict) "overrideAllValues" ($atlas.overrideAllValues | default dict) ) + "snapshotClass" $snapshotClass "observed" (dict) "status" (dict) }} diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/render/010-state-status.yaml.gotmpl index ceb8273..a1b2a9b 100644 --- a/functions/render/010-state-status.yaml.gotmpl +++ b/functions/render/010-state-status.yaml.gotmpl @@ -1,6 +1,6 @@ # code: language=yaml # -# Extract observed state from all composed resources and compute status +# Extract observed state from composed resources and compute status # {{- $observed := $.observed.resources | default dict }} @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "nodepool-branches" "nodepool-primary" "node-prep" "longhorn" "storageclass-longhorn" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "volumesnapshotclass" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -30,11 +30,7 @@ "cnpg" (dict "ready" (get $checkReady "cnpg-operator")) "scaleToZeroPlugin" (dict "ready" (get $checkReady "cnpg-scale-to-zero")) "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) - "nodepoolBranches" (dict "ready" (get $checkReady "nodepool-branches")) - "nodepoolPrimary" (dict "ready" (get $checkReady "nodepool-primary")) - "nodePrep" (dict "ready" (get $checkReady "node-prep")) - "longhorn" (dict "ready" (get $checkReady "longhorn")) - "storageClass" (dict "ready" (get $checkReady "storageclass-longhorn")) + "snapshotClass" (dict "ready" (get $checkReady "volumesnapshotclass")) ) }} # ============================================================================== diff --git a/functions/render/140-nodepool-branches.yaml.gotmpl b/functions/render/140-nodepool-branches.yaml.gotmpl deleted file mode 100644 index 3f8f49a..0000000 --- a/functions/render/140-nodepool-branches.yaml.gotmpl +++ /dev/null @@ -1,53 +0,0 @@ -# code: language=yaml -# -# Karpenter NodePool: branches sub-pool (spot arm64 NVMe) -# -# Targets ephemeral PSQLBranch workloads. Spot capacity is acceptable here -# because branches are designed to be reproducible from their parent. -# -# Labels: workload-type=psql, sub-pool=branches -# Taints: psql=true:NoSchedule, sub-pool=branches:NoSchedule -# - -{{- if and $state.nodePool.enabled $state.nodePool.branches.enabled }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-nodepool-branches - annotations: - {{ setResourceNameAnnotation "nodepool-branches" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: karpenter.sh/v1 - kind: NodePool - metadata: - name: {{ $state.nodePool.branches.name }} - spec: - template: - metadata: - labels: - workload-type: psql - sub-pool: branches - spec: - nodeClassRef: - group: eks.amazonaws.com - kind: NodeClass - name: {{ $state.nodePool.nodeClassName }} - taints: - - key: psql - value: "true" - effect: NoSchedule - - key: sub-pool - value: branches - effect: NoSchedule - requirements: {{ $state.nodePool.branches.requirements | toJson }} - limits: {{ $state.nodePool.branches.limits | toJson }} - disruption: {{ $state.nodePool.disruption | toJson }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/145-nodepool-primary.yaml.gotmpl b/functions/render/145-nodepool-primary.yaml.gotmpl deleted file mode 100644 index 3a55306..0000000 --- a/functions/render/145-nodepool-primary.yaml.gotmpl +++ /dev/null @@ -1,93 +0,0 @@ -# code: language=yaml -# -# Karpenter NodePool: primary sub-pool (on-demand arm64 NVMe) -# -# Targets PSQLCluster primary serving DBs and the operators (CNPG, -# scale-to-zero, Atlas). On-demand because spot preemption of a Mayastor -# storage node would lose a replica — unacceptable for production data. -# -# Labels: workload-type=psql, sub-pool=primary -# Taints: psql=true:NoSchedule, sub-pool=primary:NoSchedule -# -# CNPG / Atlas / scale-to-zero schedule here via the nodeSelector + -# tolerations injected by their respective Helm release templates -# (see $state.nodePool.nodeSelector / .tolerations in state-init). -# - -{{- if and $state.nodePool.enabled $state.nodePool.primary.enabled }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-nodepool-primary - annotations: - {{ setResourceNameAnnotation "nodepool-primary" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: karpenter.sh/v1 - kind: NodePool - metadata: - name: {{ $state.nodePool.primary.name }} - spec: - template: - metadata: - labels: - workload-type: psql - sub-pool: primary - spec: - nodeClassRef: - group: eks.amazonaws.com - kind: NodeClass - name: {{ $state.nodePool.nodeClassName }} - taints: - - key: psql - value: "true" - effect: NoSchedule - - key: sub-pool - value: primary - effect: NoSchedule - requirements: {{ $state.nodePool.primary.requirements | toJson }} - limits: {{ $state.nodePool.primary.limits | toJson }} - disruption: {{ $state.nodePool.disruption | toJson }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} - -# ============================================================================== -# Usages: protect the primary NodePool from being deleted before the Helm -# releases that depend on it (CNPG operator, Atlas operator). If the NodePool -# vanishes first, operator pods lose their nodes and Helm cleanup hangs. -# ============================================================================== -{{- $pinnedReleases := dict - "cloudnative-pg" "cnpg" - "atlas-operator" "atlasOperator" -}} -{{- range $release, $obsKey := $pinnedReleases }} - {{- $releaseReady := (get (get $state.observed $obsKey | default dict) "ready") }} - {{- if and $state.observed.nodepoolPrimary.ready $releaseReady }} ---- -apiVersion: protection.crossplane.io/v1beta1 -kind: Usage -metadata: - name: {{ $state.name }}-delete-{{ $release }}-before-nodepool-primary - annotations: - {{ setResourceNameAnnotation (printf "usage-np-primary-%s" $release) }} - labels: {{ $state.labels | toJson }} -spec: - replayDeletion: true - of: - apiVersion: kubernetes.m.crossplane.io/v1alpha1 - kind: Object - resourceRef: - name: {{ $state.name }}-nodepool-primary - by: - apiVersion: helm.m.crossplane.io/v1beta1 - kind: Release - resourceRef: - name: {{ $release }} - {{- end }} -{{- end }} -{{- end }} diff --git a/functions/render/155-node-prep-daemonset.yaml.gotmpl b/functions/render/155-node-prep-daemonset.yaml.gotmpl deleted file mode 100644 index 92e2b68..0000000 --- a/functions/render/155-node-prep-daemonset.yaml.gotmpl +++ /dev/null @@ -1,258 +0,0 @@ -# code: language=yaml -# -# Node-prep DaemonSet for Longhorn V2 prerequisites -# -# Runs once per NVMe NodePool node and configures the host-level state -# Longhorn V2's SPDK data engine requires, then tells Longhorn to register -# the local NVMe instance-store device as a V2 block-mode disk by labeling -# + annotating the K8s Node. -# -# Composes: -# - ServiceAccount + ClusterRole + ClusterRoleBinding granting the DS -# permission to get/patch K8s Nodes (to set Longhorn disk labels and -# annotations). -# - DaemonSet that, per node: -# 1. Sysctls vm.nr_hugepages (required by SPDK) -# 2. modprobes nvme_tcp, vfio_pci, uio_pci_generic, ublk_drv -# (Longhorn V2 transports + UBLK ingress) -# 3. Detects /dev/nvme[1-9]n1 instance-store device by lsblk model -# 4. Labels the K8s Node `node.longhorn.io/create-default-disk=config` -# and annotates `node.longhorn.io/default-disks-config=[...]` so -# Longhorn auto-registers the device as a V2 block-mode disk -# tagged `psql` (matched by the StorageClass `diskSelector: psql`). -# -# After the init container finishes, a tiny pause container keeps the -# pod up so the DaemonSet reports Ready. -# -# Caveats: -# - On EKS Auto Mode (Bottlerocket OS), nvme_tcp is typically pre-loaded; -# vfio_pci / uio_pci_generic / ublk_drv may need the modprobe attempt; -# script falls back gracefully on each. -# - Auto Mode's NodeClass doesn't support userData, so runtime allocation -# via this DS is currently the only path for hugepages + non-default -# kernel modules. -# - -{{- if $state.nodePrep.enabled }} - -# ---- ServiceAccount for the DaemonSet ------------------------------------ ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-node-prep-sa - annotations: - {{ setResourceNameAnnotation "node-prep-sa" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: v1 - kind: ServiceAccount - metadata: - name: psql-node-prep - namespace: {{ $state.namespace }} - labels: {{ $state.labels | toJson }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} - -# ---- ClusterRole: patch labels + annotations on K8s Nodes ---------------- ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-node-prep-clusterrole - annotations: - {{ setResourceNameAnnotation "node-prep-clusterrole" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: psql-node-prep - labels: {{ $state.labels | toJson }} - rules: - - apiGroups: [""] - resources: ["nodes"] - verbs: ["get", "patch", "update"] - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} - ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-node-prep-clusterrolebinding - annotations: - {{ setResourceNameAnnotation "node-prep-clusterrolebinding" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: psql-node-prep - labels: {{ $state.labels | toJson }} - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: psql-node-prep - subjects: - - kind: ServiceAccount - name: psql-node-prep - namespace: {{ $state.namespace }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} - -# ---- DaemonSet ----------------------------------------------------------- ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-node-prep - annotations: - {{ setResourceNameAnnotation "node-prep" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: apps/v1 - kind: DaemonSet - metadata: - name: psql-node-prep - namespace: {{ $state.namespace }} - labels: - {{- range $k, $v := $state.labels }} - {{ $k }}: {{ $v | quote }} - {{- end }} - app: psql-node-prep - spec: - selector: - matchLabels: - app: psql-node-prep - template: - metadata: - labels: - app: psql-node-prep - spec: - serviceAccountName: psql-node-prep - hostPID: true - hostNetwork: true - nodeSelector: - workload-type: psql - tolerations: - - key: psql - value: "true" - effect: NoSchedule - - key: sub-pool - operator: Exists - effect: NoSchedule - initContainers: - - name: prep - image: {{ $state.nodePrep.image | quote }} - securityContext: - privileged: true - env: - - name: NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - command: - - /bin/sh - - -c - - | - set -e - # Toolchain (kubectl + lsblk via util-linux + curl for K8s API fallback) - if command -v apk >/dev/null 2>&1; then - apk add --no-cache util-linux curl kubectl - fi - - # ---- Hugepages (SPDK requires) ---- - echo "Setting vm.nr_hugepages={{ $state.nodePrep.hugepages.count }}" - if [ -w /proc/sys/vm/nr_hugepages ]; then - echo {{ $state.nodePrep.hugepages.count }} > /proc/sys/vm/nr_hugepages - else - echo "WARN: /proc/sys/vm/nr_hugepages not writable; allocate manually" - fi - - # ---- Kernel modules required by Longhorn V2 ---- - for mod in nvme_tcp vfio_pci uio_pci_generic ublk_drv; do - if lsmod 2>/dev/null | grep -q "^${mod}"; then - echo "${mod} already loaded" - else - echo "Attempting modprobe ${mod}" - modprobe ${mod} 2>/dev/null \ - || nsenter -t 1 -m modprobe ${mod} 2>/dev/null \ - || echo "WARN: could not load ${mod} (Bottlerocket may not ship it; verify on host)" - fi - done - - # ---- Register the local NVMe instance-store device as a - # Longhorn V2 block-mode disk via Node label + annotation - DEVICE=$(lsblk -dno NAME,TYPE,MODEL 2>/dev/null \ - | awk '$2=="disk" && /Amazon EC2 NVMe Instance Storage/ {print "/dev/"$1; exit}') - if [ -z "$DEVICE" ]; then - echo "WARN: no instance-store NVMe device on $NODE_NAME; skipping disk registration" - else - echo "Registering Longhorn V2 disk for $NODE_NAME ($DEVICE)" - DISKS_JSON='[{"path":"'"${DEVICE}"'","diskType":"block","name":"nvme-block-pool","tags":["psql"],"allowScheduling":true,"storageReserved":0}]' - kubectl label node "$NODE_NAME" \ - node.longhorn.io/create-default-disk=config --overwrite - kubectl annotate node "$NODE_NAME" \ - "node.longhorn.io/default-disks-config=${DISKS_JSON}" --overwrite - fi - - echo "node-prep complete" - volumeMounts: - - name: dev - mountPath: /dev - - name: sys - mountPath: /sys - - name: host-modules - mountPath: /lib/modules - readOnly: true - - name: proc - mountPath: /proc-host - containers: - - name: pause - image: {{ $state.nodePrep.image | quote }} - command: - - /bin/sh - - -c - - | - trap 'exit 0' TERM - while true; do sleep 3600 & wait $!; done - resources: - requests: - cpu: 10m - memory: 16Mi - limits: - cpu: 50m - memory: 32Mi - volumes: - - name: dev - hostPath: - path: /dev - - name: sys - hostPath: - path: /sys - - name: host-modules - hostPath: - path: /lib/modules - - name: proc - hostPath: - path: /proc - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/165-longhorn.yaml.gotmpl b/functions/render/165-longhorn.yaml.gotmpl deleted file mode 100644 index db8fe7c..0000000 --- a/functions/render/165-longhorn.yaml.gotmpl +++ /dev/null @@ -1,103 +0,0 @@ -# code: language=yaml -# -# Helm Release: Longhorn (V2 Data Engine — SPDK + NVMe-oF) -# -# Provides replicated CoW storage via Longhorn's V2 data engine, which uses -# SPDK userspace NVMe-oF instead of V1's kernel iSCSI. Comparable performance -# to OpenEBS Mayastor with a meaningfully lighter footprint (no bundled -# etcd/NATS/minio/loki/alloy — Longhorn keeps state in CRDs). -# -# Node-side prerequisites (provided by phase 3's NodePool config + node-prep DS): -# - 2MiB hugepages (`vm.nr_hugepages=1024` minimum) -# - kernel modules: `nvme_tcp`, `vfio_pci`, `uio_pci_generic`, `ublk_drv` -# - Local NVMe instance-store devices for V2 block-mode disks -# -# Status: Longhorn V2 is "Experimental" upstream as of Longhorn 1.10. -# Track promotion to GA via https://longhorn.io/docs//v2-data-engine/ -# -# Chart: https://charts.longhorn.io (chart name: longhorn) -# Upstream: https://github.com/longhorn/longhorn -# - -{{- $storage := $state.storage }} -{{- if $state.nodePool.enabled }} ---- -apiVersion: helm.m.crossplane.io/v1beta1 -kind: Release -metadata: - name: longhorn - annotations: - {{ setResourceNameAnnotation "longhorn" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - chart: - name: longhorn - repository: https://charts.longhorn.io - version: {{ $storage.chartVersion | quote }} - namespace: {{ $storage.namespace }} - {{- if $storage.overrideAllValues }} - values: - {{- toYaml $storage.overrideAllValues | nindent 6 }} - {{- else }} - {{- /* Build defaultSettings — V2 data engine + replication defaults. */}} - {{- $defaultSettings := dict - "v2DataEngine" true - "v2DataEngineHugepageLimit" "2048" - "defaultReplicaCount" $storage.replicationFactor - "defaultDataLocality" "best-effort" - "staleReplicaTimeout" 30 - "createDefaultDiskLabeledNodes" true - }} - - {{- /* Confine Longhorn components to NVMe nodes. Longhorn manager and - instance-manager are DaemonSets; csi components are Deployments. - All ride workload-type=psql nodes only. */}} - {{- $nodeSelector := dict }} - {{- $tolerations := list }} - {{- if $state.nodePool.enabled }} - {{- $nodeSelector = $state.nodePool.nodeSelector }} - {{- $tolerations = $state.nodePool.tolerations }} - {{- end }} - - {{- /* Helm chart structure: longhornManager, longhornDriver, longhornUI, - csi all accept tolerations + nodeSelector via top-level keys. */}} - {{- $chartDefaults := dict - "defaultSettings" $defaultSettings - "longhornManager" (dict - "tolerations" $tolerations - "nodeSelector" $nodeSelector - ) - "longhornDriver" (dict - "tolerations" $tolerations - "nodeSelector" $nodeSelector - ) - "longhornUI" (dict - "tolerations" $tolerations - "nodeSelector" $nodeSelector - ) - }} - - {{- if $state.ha.enabled }} - {{- /* HA: bump CSI controller + UI replicas. Longhorn manager runs as - a DaemonSet on every workload-type=psql node, so replica count - scales with the NodePool, not this knob. */}} - {{- $_ := set $chartDefaults "longhornUI" (mergeOverwrite (get $chartDefaults "longhornUI") (dict "replicas" $state.ha.replicas)) }} - {{- $_ := set $chartDefaults "csi" (dict - "attacherReplicaCount" $state.ha.replicas - "provisionerReplicaCount" $state.ha.replicas - "resizerReplicaCount" $state.ha.replicas - "snapshotterReplicaCount" $state.ha.replicas - ) }} - {{- end }} - - {{- $mergedValues := mergeOverwrite $chartDefaults ($storage.values | default dict) }} - values: - {{- toYaml $mergedValues | nindent 6 }} - {{- end }} - rollbackLimit: 3 - providerConfigRef: - name: {{ $state.helmProviderConfigRef.name }} - kind: {{ $state.helmProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/170-storageclass-longhorn.yaml.gotmpl b/functions/render/170-storageclass-longhorn.yaml.gotmpl deleted file mode 100644 index 1cafd54..0000000 --- a/functions/render/170-storageclass-longhorn.yaml.gotmpl +++ /dev/null @@ -1,68 +0,0 @@ -# code: language=yaml -# -# StorageClass + VolumeSnapshotClass — Longhorn V2 (replicated CoW) -# -# The stack's only StorageClass. Backed by Longhorn V2's SPDK data engine -# over NVMe-oF (TCP transport). Replicated by default -# (numberOfReplicas = $storage.replicationFactor); thin-provisioned by default. -# PSQLClusters that don't want CoW can target a different SC the cluster -# already provides; the stack doesn't compose a non-CoW SC. -# - -{{- $storage := $state.storage }} -{{- if $state.nodePool.enabled }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-storageclass-longhorn - annotations: - {{ setResourceNameAnnotation "storageclass-longhorn" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: storage.k8s.io/v1 - kind: StorageClass - metadata: - name: {{ $storage.storageClassName }} - labels: {{ $state.labels | toJson }} - provisioner: driver.longhorn.io - parameters: - dataEngine: "v2" - numberOfReplicas: "{{ $storage.replicationFactor }}" - staleReplicaTimeout: "30" - fsType: "ext4" - diskSelector: "psql" - reclaimPolicy: {{ $storage.reclaimPolicy }} - volumeBindingMode: {{ $storage.volumeBindingMode }} - allowVolumeExpansion: {{ $storage.allowVolumeExpansion }} - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} ---- -apiVersion: kubernetes.m.crossplane.io/v1alpha1 -kind: Object -metadata: - name: {{ $state.name }}-volumesnapshotclass-longhorn - annotations: - {{ setResourceNameAnnotation "volumesnapshotclass-longhorn" }} - labels: {{ $state.labels | toJson }} -spec: - managementPolicies: {{ $state.managementPolicies | toJson }} - forProvider: - manifest: - apiVersion: snapshot.storage.k8s.io/v1 - kind: VolumeSnapshotClass - metadata: - name: {{ $storage.storageClassName }} - labels: {{ $state.labels | toJson }} - driver: driver.longhorn.io - deletionPolicy: Delete - parameters: - type: snap - providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} -{{- end }} diff --git a/functions/render/170-volumesnapshotclass.yaml.gotmpl b/functions/render/170-volumesnapshotclass.yaml.gotmpl new file mode 100644 index 0000000..fab378c --- /dev/null +++ b/functions/render/170-volumesnapshotclass.yaml.gotmpl @@ -0,0 +1,45 @@ +# code: language=yaml +# +# VolumeSnapshotClass — the stack's only storage-layer composition. +# +# PSQLBranch references this VSC by name to fork PSQLCluster primaries via +# CSI snapshots. The stack does NOT compose a StorageClass — it relies on +# whatever the target cluster already provides (gp3 on EKS Auto Mode, +# standard on kind/k3d, etc.). +# +# Default driver: ebs.csi.aws.com (correct for EKS Auto Mode out of the box). +# Override $spec.snapshotClass.driver for non-AWS / non-EBS providers +# (e.g. driver.longhorn.io, hostpath.csi.k8s.io). +# + +{{- $snap := $state.snapshotClass }} +{{- if $snap.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-volumesnapshotclass + annotations: + {{ setResourceNameAnnotation "volumesnapshotclass" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshotClass + metadata: + name: {{ $snap.name }} + labels: {{ $state.labels | toJson }} + driver: {{ $snap.driver | quote }} + deletionPolicy: {{ $snap.deletionPolicy }} + {{- if $snap.parameters }} + parameters: + {{- range $k, $v := $snap.parameters }} + {{ $k }}: {{ $v | quote }} + {{- end }} + {{- end }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/render/200-cnpg-operator.yaml.gotmpl b/functions/render/200-cnpg-operator.yaml.gotmpl index 58a42fa..8ccecdc 100644 --- a/functions/render/200-cnpg-operator.yaml.gotmpl +++ b/functions/render/200-cnpg-operator.yaml.gotmpl @@ -33,12 +33,9 @@ spec: values: {{- toYaml $cnpg.overrideAllValues | nindent 6 }} {{- else }} - {{- /* Chart defaults */}} + {{- /* Chart defaults — Auto Mode handles scheduling, no nodeSelector/ + tolerations injected. */}} {{- $chartDefaults := dict }} - {{- if $state.nodePool.enabled }} - {{- $_ := set $chartDefaults "nodeSelector" $state.nodePool.nodeSelector }} - {{- $_ := set $chartDefaults "tolerations" $state.nodePool.tolerations }} - {{- end }} {{- if $state.ha.enabled }} {{- $_ := set $chartDefaults "replicaCount" $state.ha.replicas }} {{- if $state.ha.topologySpreadByZone }} diff --git a/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl b/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl index ebabca1..969016d 100644 --- a/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl +++ b/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl @@ -304,10 +304,6 @@ spec: app: scale-to-zero spec: serviceAccountName: cnpg-scale-to-zero-plugin - {{- if $state.nodePool.enabled }} - nodeSelector: {{ $state.nodePool.nodeSelector | toJson }} - tolerations: {{ $state.nodePool.tolerations | toJson }} - {{- end }} {{- if and $state.ha.enabled $state.ha.topologySpreadByZone }} topologySpreadConstraints: - maxSkew: 1 diff --git a/functions/render/220-atlas-operator.yaml.gotmpl b/functions/render/220-atlas-operator.yaml.gotmpl index 6b617d7..ab8f902 100644 --- a/functions/render/220-atlas-operator.yaml.gotmpl +++ b/functions/render/220-atlas-operator.yaml.gotmpl @@ -31,10 +31,6 @@ spec: {{- $chartDefaults := dict "prewarmDevDB" true }} - {{- if $state.nodePool.enabled }} - {{- $_ := set $chartDefaults "nodeSelector" $state.nodePool.nodeSelector }} - {{- $_ := set $chartDefaults "tolerations" $state.nodePool.tolerations }} - {{- end }} {{- if $state.ha.enabled }} {{- $_ := set $chartDefaults "replicaCount" $state.ha.replicas }} {{- if $state.ha.topologySpreadByZone }} diff --git a/tests/test-render/main.k b/tests/test-render/main.k index d4da64a..18adf80 100644 --- a/tests/test-render/main.k +++ b/tests/test-render/main.k @@ -1,6 +1,5 @@ import models.ai.com.ops.hops.v1alpha1 as stacksv1alpha1 import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 -import models.k8s.apimachinery.pkg.apis.meta.v1 as metav1 # ============================================================================== # Unit tests for PSQLStack XRD @@ -13,8 +12,8 @@ import models.k8s.apimachinery.pkg.apis.meta.v1 as metav1 _items = [ # ========================================================================== # Test 1: minimal claim renders the platform operators (CNPG, Atlas), the - # scale-to-zero plugin (default-on), Longhorn + StorageClass + node-prep + - # both NodePool sub-pools (nodePool default-on now — stack's whole point). + # scale-to-zero plugin (default-on), and the VolumeSnapshotClass + # (default driver: ebs.csi.aws.com, default name: "psql"). # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "minimal-renders-platform-stack" @@ -38,41 +37,16 @@ _items = [ kind = "Release" metadata.name = "atlas-operator" } - { - apiVersion = "helm.m.crossplane.io/v1beta1" - kind = "Release" - metadata.name = "longhorn" - } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "test-psql-storageclass-longhorn" - } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "test-psql-volumesnapshotclass-longhorn" + metadata.name = "test-psql-volumesnapshotclass" } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" metadata.name = "test-psql-s2z-deployment" } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "test-psql-nodepool-branches" - } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "test-psql-nodepool-primary" - } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "test-psql-node-prep" - } ] } } @@ -321,11 +295,8 @@ _items = [ scaleToZeroPlugin = {enabled = False} } } - # When disabled, no s2z-* Objects are composed. We assert the - # expected platform Releases are still present and ebs StorageClass - # is created — the assertion is a positive presence check; the - # absence of s2z-* resources is implicit (composition gates them - # on enabled=true). + # Positive presence: cnpg + atlas + VSC still composed. + # Absence of s2z-* resources is implicit (composition gates them). assertResources = [ { apiVersion = "helm.m.crossplane.io/v1beta1" @@ -335,114 +306,118 @@ _items = [ { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "no-s2z-storageclass-longhorn" + metadata.name = "no-s2z-volumesnapshotclass" } ] } } # ========================================================================== - # Test 9: nodePool.enabled=false skips Longhorn, StorageClass, and node-prep - # (the stack's storage requires the NodePool to have something to schedule - # on — disabling the NodePool reasonably skips everything that needs it). + # Test 9: ha.enabled=true sets replicaCount on CNPG and Atlas Helm releases. # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "nodepool-disabled-skips-storage" + metadata.name = "ha-enabled-injects-replica-count" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" timeoutSeconds = 60 validate = False xr = stacksv1alpha1.PSQLStack { - metadata.name = "np-off" + metadata.name = "ha-on" spec = { - clusterName = "my-cluster" - nodePool = {enabled = False} + clusterName = "prod" + ha = {enabled = True, replicas = 3} } } - # Positive presence: cnpg + atlas still composed assertResources = [ { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" metadata.name = "cloudnative-pg" + spec.forProvider.values.replicaCount = 3 } { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "Release" metadata.name = "atlas-operator" + spec.forProvider.values.replicaCount = 3 } ] } } # ========================================================================== - # Test 11: ha.enabled=true sets replicaCount + topology spread on CNPG and - # Atlas Helm releases, and on the directly-composed S2Z Deployment. + # Test 10: spec.snapshotClass settings flow through to the composed VSC. + # Default driver = ebs.csi.aws.com; override to verify it propagates. # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "ha-enabled-injects-replica-count-and-spread" + metadata.name = "snapshotclass-driver-flows-through" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" timeoutSeconds = 60 validate = False xr = stacksv1alpha1.PSQLStack { - metadata.name = "ha-on" + metadata.name = "vsc-tweaks" spec = { - clusterName = "prod" - ha = {enabled = True, replicas = 3} + clusterName = "my-cluster" + snapshotClass = { + driver = "driver.longhorn.io" + deletionPolicy = "Retain" + } } } assertResources = [ { - apiVersion = "helm.m.crossplane.io/v1beta1" - kind = "Release" - metadata.name = "cloudnative-pg" - spec.forProvider.values.replicaCount = 3 - } - { - apiVersion = "helm.m.crossplane.io/v1beta1" - kind = "Release" - metadata.name = "atlas-operator" - spec.forProvider.values.replicaCount = 3 + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "vsc-tweaks-volumesnapshotclass" + spec.forProvider.manifest = { + apiVersion = "snapshot.storage.k8s.io/v1" + kind = "VolumeSnapshotClass" + metadata = { + name = "psql" + labels = { + "hops.ops.com.ai/managed" = "true" + "hops.ops.com.ai/psql" = "vsc-tweaks" + } + } + driver = "driver.longhorn.io" + deletionPolicy = "Retain" + } } ] } } # ========================================================================== - # Test 10: spec.storage.replicationFactor flows through to the Longhorn - # StorageClass parameters (numberOfReplicas). + # Test 11: snapshotClass.enabled=false suppresses the VSC composition. # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "storage-settings-flow-through-to-sc" + metadata.name = "snapshotclass-can-be-disabled" spec = { compositionPath = "apis/psqlstacks/composition.yaml" xrdPath = "apis/psqlstacks/definition.yaml" timeoutSeconds = 60 validate = False xr = stacksv1alpha1.PSQLStack { - metadata.name = "storage-tweaks" + metadata.name = "no-vsc" spec = { clusterName = "my-cluster" - storage = { - replicationFactor = 1 - } + snapshotClass = {enabled = False} } } + # Cnpg + atlas still composed. No vsc-* Object expected (implicit). assertResources = [ { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "storage-tweaks-storageclass-longhorn" - spec.forProvider.manifest.parameters = { - dataEngine = "v2" - numberOfReplicas = "1" - staleReplicaTimeout = "30" - fsType = "ext4" - diskSelector = "psql" - } + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "cloudnative-pg" + } + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "atlas-operator" } ] } From 319048383780303d543ee016f362f764aaaff4b9 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Mon, 27 Apr 2026 15:43:49 -0500 Subject: [PATCH 19/44] fix: default snapshot driver to ebs.csi.eks.amazonaws.com EKS Auto Mode uses the managed `ebs.csi.eks.amazonaws.com` CSI driver, not the upstream `ebs.csi.aws.com`. Fix the default in state-init, XRD, README, and the minimal example so out-of-the-box installs on Auto Mode produce a working VolumeSnapshotClass without override. Self-managed EBS users can still override `spec.snapshotClass.driver` to `ebs.csi.aws.com`; non-AWS users override to their CSI driver name. Verified live on pat-local: psql VSC reconciles to driver ebs.csi.eks.amazonaws.com, XR Synced=True Ready=True, CNPG + Atlas + S2Z plugin all running cleanly on Auto Mode without dedicated nodes. --- README.md | 6 +++--- apis/psqlstacks/definition.yaml | 4 ++-- examples/psqlstacks/minimal.yaml | 2 +- functions/render/000-state-init.yaml.gotmpl | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index b59d7e1..739a385 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ The stack is intentionally **OS-agnostic and storage-agnostic**: - **No StorageClass.** PSQLClusters target whatever StorageClass the cluster already provides (`gp3` on EKS Auto Mode, `standard` on kind/k3d, etc.). - **No NodePool / node-prep.** Components run wherever the cluster's scheduler puts them. Auto Mode handles node provisioning end-to-end. -- **Single composed snapshot target.** The stack ships a `VolumeSnapshotClass` named `psql` so PSQLBranch can request snapshots without leaking driver-specific knowledge into PSQLBranch's spec. Default driver is `ebs.csi.aws.com` (EKS Auto Mode default); override for non-AWS clusters. +- **Single composed snapshot target.** The stack ships a `VolumeSnapshotClass` named `psql` so PSQLBranch can request snapshots without leaking driver-specific knowledge into PSQLBranch's spec. Default driver is `ebs.csi.eks.amazonaws.com` (EKS Auto Mode's managed EBS CSI driver); override for non-AWS clusters or self-managed EBS. If you need replicated CoW storage (true block-level branches with delta-only economics), that's a separate concern — see `aws-storage-stack` for self-managed nodes that can host Longhorn or similar. The default psql-stack stays on the AWS-blessed Auto Mode path. @@ -21,7 +21,7 @@ If you need replicated CoW storage (true block-level branches with delta-only ec | **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | | **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | | **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | -| **VolumeSnapshotClass** | on (`spec.snapshotClass.enabled: true`) | Named `psql` by default. Driver: `ebs.csi.aws.com`. PSQLBranch references this name. | +| **VolumeSnapshotClass** | on (`spec.snapshotClass.enabled: true`) | Named `psql` by default. Driver: `ebs.csi.eks.amazonaws.com`. PSQLBranch references this name. | | **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas + `topologySpreadConstraints` by zone on CNPG, Atlas, S2Z plugin. | ## Prerequisites @@ -141,7 +141,7 @@ spec: | **Snapshot class** | | | | | `snapshotClass.enabled` | bool | `true` | Compose the VolumeSnapshotClass | | `snapshotClass.name` | string | `psql` | VolumeSnapshotClass name (PSQLBranch references this) | -| `snapshotClass.driver` | string | `ebs.csi.aws.com` | CSI driver | +| `snapshotClass.driver` | string | `ebs.csi.eks.amazonaws.com` | CSI driver (Auto Mode default; override for self-managed EBS or non-AWS) | | `snapshotClass.deletionPolicy` | enum | `Delete` | `Delete` or `Retain` | | `snapshotClass.parameters` | object | — | Driver-specific parameters | diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 0491148..5b60ad7 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -165,9 +165,9 @@ spec: type: string default: psql driver: - description: CSI driver. Defaults to ebs.csi.aws.com (EKS Auto Mode default). + description: CSI driver. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode's managed EBS CSI driver). For self-managed EBS use ebs.csi.aws.com; for non-AWS providers, use the appropriate driver name. type: string - default: ebs.csi.aws.com + default: ebs.csi.eks.amazonaws.com deletionPolicy: description: Snapshot deletion policy. Defaults to Delete. type: string diff --git a/examples/psqlstacks/minimal.yaml b/examples/psqlstacks/minimal.yaml index 1bc55ad..88aaf5f 100644 --- a/examples/psqlstacks/minimal.yaml +++ b/examples/psqlstacks/minimal.yaml @@ -4,7 +4,7 @@ # - CloudNativePG operator (Helm) # - cnpg-i-scale-to-zero plugin (set of Objects, requires cert-manager) # - Atlas operator (Helm) -# - VolumeSnapshotClass `psql` (driver: ebs.csi.aws.com) +# - VolumeSnapshotClass `psql` (driver: ebs.csi.eks.amazonaws.com) # # PSQLClusters target the cluster's existing default StorageClass # (gp3 on EKS Auto Mode, standard on kind/k3d, etc.). PSQLBranch diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/render/000-state-init.yaml.gotmpl index 8863e6f..83b09a0 100644 --- a/functions/render/000-state-init.yaml.gotmpl +++ b/functions/render/000-state-init.yaml.gotmpl @@ -81,7 +81,7 @@ {{- $snapshotClass := dict "enabled" $snapshotEnabled "name" ($snapshotSpec.name | default "psql") - "driver" ($snapshotSpec.driver | default "ebs.csi.aws.com") + "driver" ($snapshotSpec.driver | default "ebs.csi.eks.amazonaws.com") "deletionPolicy" ($snapshotSpec.deletionPolicy | default "Delete") "parameters" ($snapshotSpec.parameters | default dict) }} From 38c2b7c1cad37e2150021c781ca93f0c794d6f67 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Mon, 27 Apr 2026 15:51:16 -0500 Subject: [PATCH 20/44] docs: flag snapshot-controller as a prerequisite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EKS Auto Mode ships the snapshot.storage.k8s.io CRDs but does NOT ship the snapshot-controller. Without a controller, our composed VolumeSnapshotClass is inert — PSQLBranch snapshots will sit forever without ever reaching ReadyToUse. Document it as a prerequisite (like cert-manager). The stack itself does not compose snapshot-controller — it's a foundational cluster concern that belongs in a separate stack (TBD: extend aws-cert-stack or create a focused snapshot-stack). Verified end-to-end on pat-local with the upstream kubernetes-csi/external-snapshotter v8.2.0 manifests: - PVC bound on ebs.csi.eks.amazonaws.com - VolumeSnapshot via the composed `psql` VSC reached readyToUse=true with a real EBS snapshot backing it. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 739a385..969bfd2 100644 --- a/README.md +++ b/README.md @@ -26,8 +26,8 @@ If you need replicated CoW storage (true block-level branches with delta-only ec ## Prerequisites -- **A working CSI driver + StorageClass** on the cluster. EKS Auto Mode provides `gp3` + `ebs.csi.aws.com` automatically. For kind/k3d, the bundled `standard` SC works. -- **VolumeSnapshot CRDs** (snapshot.storage.k8s.io). EKS Auto Mode includes the snapshot-controller; for self-managed clusters install it from [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter). +- **A working CSI driver + StorageClass** on the cluster. EKS Auto Mode provides `ebs.csi.eks.amazonaws.com` automatically. For kind/k3d, the bundled `standard` SC works. +- **VolumeSnapshot CRDs + snapshot-controller** (`snapshot.storage.k8s.io`). EKS Auto Mode ships the **CRDs only** — you must install the snapshot-controller separately (e.g. apply [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter) `rbac-snapshot-controller.yaml` + `setup-snapshot-controller.yaml`). Without a controller, the composed VolumeSnapshotClass is inert and PSQLBranch snapshots will never reach `ReadyToUse`. The stack itself does not compose snapshot-controller — it's a cluster-wide foundational concern that belongs in a separate stack. - **cert-manager** (only when `scaleToZeroPlugin.enabled` — the plugin uses cert-manager Issuer+Certificate for its gRPC TLS). Provided by [`aws-cert-stack`](../../aws/cert/). ## Stages From b7a522f2a679c20958758a1b7f92e86a25e0f77a Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Mon, 27 Apr 2026 23:48:10 -0500 Subject: [PATCH 21/44] docs: point snapshot-controller prereq at volume-snapshot-stack The dependency now exists as a real stack (xrs/stacks/k8s/volume-snapshot/, ghcr.io/hops-ops/volume-snapshot-stack). Replace the manual upstream-YAML install instructions with a pointer at the stack. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 969bfd2..5706272 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ If you need replicated CoW storage (true block-level branches with delta-only ec ## Prerequisites - **A working CSI driver + StorageClass** on the cluster. EKS Auto Mode provides `ebs.csi.eks.amazonaws.com` automatically. For kind/k3d, the bundled `standard` SC works. -- **VolumeSnapshot CRDs + snapshot-controller** (`snapshot.storage.k8s.io`). EKS Auto Mode ships the **CRDs only** — you must install the snapshot-controller separately (e.g. apply [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter) `rbac-snapshot-controller.yaml` + `setup-snapshot-controller.yaml`). Without a controller, the composed VolumeSnapshotClass is inert and PSQLBranch snapshots will never reach `ReadyToUse`. The stack itself does not compose snapshot-controller — it's a cluster-wide foundational concern that belongs in a separate stack. +- **VolumeSnapshot CRDs + snapshot-controller** (`snapshot.storage.k8s.io`). EKS Auto Mode ships the snapshot CRDs but **not** the cluster-wide snapshot-controller. Without one, the composed VolumeSnapshotClass is inert and PSQLBranch snapshots will never reach `ReadyToUse`. Install [`volume-snapshot-stack`](../volume-snapshot/) (also in this org) — it composes the upstream snapshot-controller via the piraeus-charts Helm chart and is the canonical CRD installer for the cluster. - **cert-manager** (only when `scaleToZeroPlugin.enabled` — the plugin uses cert-manager Issuer+Certificate for its gRPC TLS). Provided by [`aws-cert-stack`](../../aws/cert/). ## Stages From d4340db66e010c8e91cdb8697416566d2b202938 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Tue, 28 Apr 2026 15:24:23 -0500 Subject: [PATCH 22/44] feat: merge psql-cluster + psql-branch APIs into psql-stack package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Merges PSQLCluster and PSQLBranch XRDs (previously standalone repos under hops-ops/psql-cluster and hops-ops/psql-branch) into this package. One Configuration package, one release cadence, one e2e flow. Changes: - apis/{psqlclusters,psqlbranches}/ — XRDs and compositions copied in - examples/{psqlclusters,psqlbranches}/ — example manifests copied in - functions/render/ → functions/stack/ — renamed to make room for siblings - functions/{cluster,branch}/ — new function packages, gotmpls copied from the standalone repos. Composition functionRefs updated: psqlstacks → hops-ops-psql-stackstack psqlclusters → hops-ops-psql-stackcluster psqlbranches → hops-ops-psql-stackbranch - tests/test-{stack,cluster,branch}/ — render tests renamed (was test-render-*) - tests/e2etest-psql/main.k — unified e2e covering all three XRs at Synced; TODO upgrade to Ready integration after volume-snapshot-stack v0.1.0 - .github/workflows/on-pr.yaml + on-push-main.yaml — switched to multi-API workflow signature, pinned at @feat/multi-api-support for testing - Makefile — EXAMPLES list extended; render/validate logic still single-API (follow-up to make per-example api_path work locally) Workflow change being tested: unbounded-tech/workflows-crossplane@feat/multi-api-support (validate.yaml now resolves api_path per example with fallback to inputs.api_path) --- .github/workflows/on-pr.yaml | 16 +- .github/workflows/on-push-main.yaml | 14 +- Makefile | 11 +- apis/psqlbranches/composition.yaml | 16 + apis/psqlbranches/definition.yaml | 191 +++++++ apis/psqlclusters/composition.yaml | 16 + apis/psqlclusters/definition.yaml | 208 ++++++++ apis/psqlstacks/composition.yaml | 4 +- examples/psqlbranches/cross-namespace.yaml | 18 + examples/psqlbranches/preview-with-ttl.yaml | 22 + examples/psqlbranches/same-namespace.yaml | 13 + examples/psqlclusters/minimal.yaml | 12 + examples/psqlclusters/standard.yaml | 35 ++ functions/branch/000-state-init.yaml.gotmpl | 135 +++++ functions/branch/010-state-status.yaml.gotmpl | 39 ++ .../branch/100-source-snapshot.yaml.gotmpl | 43 ++ .../branch/110-branch-snapshot.yaml.gotmpl | 45 ++ functions/branch/200-cnpg-cluster.yaml.gotmpl | 98 ++++ functions/branch/999-status.yaml.gotmpl | 14 + functions/cluster/000-state-init.yaml.gotmpl | 171 ++++++ .../cluster/010-state-status.yaml.gotmpl | 46 ++ .../cluster/100-external-secret.yaml.gotmpl | 56 ++ .../cluster/200-cnpg-cluster.yaml.gotmpl | 108 ++++ functions/cluster/999-status.yaml.gotmpl | 22 + .../000-state-init.yaml.gotmpl | 0 .../010-state-status.yaml.gotmpl | 0 .../170-volumesnapshotclass.yaml.gotmpl | 0 .../200-cnpg-operator.yaml.gotmpl | 0 .../210-cnpg-scale-to-zero.yaml.gotmpl | 0 .../220-atlas-operator.yaml.gotmpl | 0 .../{render => stack}/999-status.yaml.gotmpl | 0 tests/e2etest-psql/main.k | 101 +++- tests/{test-render => test-branch}/kcl.mod | 0 tests/test-branch/main.k | 329 ++++++++++++ .../ai/com/ops/hops/v1alpha1/psqlbranch.k | 481 +++++++++++++++++ .../dev/meta/v1alpha1/compositiontest.k | 113 ++++ .../io/upbound/dev/meta/v1alpha1/e2etest.k | 167 ++++++ .../upbound/dev/meta/v1alpha1/operationtest.k | 98 ++++ .../io/upbound/dev/meta/v1alpha1/project.k | 320 +++++++++++ .../io/upbound/dev/meta/v2alpha1/project.k | 325 ++++++++++++ .../pkg/apis/meta/v1/object_meta.k | 97 ++++ .../pkg/apis/meta/v1/owner_reference.k | 41 ++ tests/test-branch/model/kcl.mod | 4 + tests/test-cluster/kcl.mod | 6 + tests/test-cluster/main.k | 357 +++++++++++++ .../ai/com/ops/hops/v1alpha1/psqlcluster.k | 496 ++++++++++++++++++ .../dev/meta/v1alpha1/compositiontest.k | 113 ++++ .../io/upbound/dev/meta/v1alpha1/e2etest.k | 167 ++++++ .../upbound/dev/meta/v1alpha1/operationtest.k | 98 ++++ .../io/upbound/dev/meta/v1alpha1/project.k | 320 +++++++++++ .../io/upbound/dev/meta/v2alpha1/project.k | 325 ++++++++++++ .../pkg/apis/meta/v1/object_meta.k | 97 ++++ .../pkg/apis/meta/v1/owner_reference.k | 41 ++ tests/test-cluster/model/kcl.mod | 4 + tests/test-stack/kcl.mod | 6 + tests/{test-render => test-stack}/main.k | 0 tests/{test-render => test-stack}/model | 0 57 files changed, 5429 insertions(+), 30 deletions(-) create mode 100644 apis/psqlbranches/composition.yaml create mode 100644 apis/psqlbranches/definition.yaml create mode 100644 apis/psqlclusters/composition.yaml create mode 100644 apis/psqlclusters/definition.yaml create mode 100644 examples/psqlbranches/cross-namespace.yaml create mode 100644 examples/psqlbranches/preview-with-ttl.yaml create mode 100644 examples/psqlbranches/same-namespace.yaml create mode 100644 examples/psqlclusters/minimal.yaml create mode 100644 examples/psqlclusters/standard.yaml create mode 100644 functions/branch/000-state-init.yaml.gotmpl create mode 100644 functions/branch/010-state-status.yaml.gotmpl create mode 100644 functions/branch/100-source-snapshot.yaml.gotmpl create mode 100644 functions/branch/110-branch-snapshot.yaml.gotmpl create mode 100644 functions/branch/200-cnpg-cluster.yaml.gotmpl create mode 100644 functions/branch/999-status.yaml.gotmpl create mode 100644 functions/cluster/000-state-init.yaml.gotmpl create mode 100644 functions/cluster/010-state-status.yaml.gotmpl create mode 100644 functions/cluster/100-external-secret.yaml.gotmpl create mode 100644 functions/cluster/200-cnpg-cluster.yaml.gotmpl create mode 100644 functions/cluster/999-status.yaml.gotmpl rename functions/{render => stack}/000-state-init.yaml.gotmpl (100%) rename functions/{render => stack}/010-state-status.yaml.gotmpl (100%) rename functions/{render => stack}/170-volumesnapshotclass.yaml.gotmpl (100%) rename functions/{render => stack}/200-cnpg-operator.yaml.gotmpl (100%) rename functions/{render => stack}/210-cnpg-scale-to-zero.yaml.gotmpl (100%) rename functions/{render => stack}/220-atlas-operator.yaml.gotmpl (100%) rename functions/{render => stack}/999-status.yaml.gotmpl (100%) rename tests/{test-render => test-branch}/kcl.mod (100%) create mode 100644 tests/test-branch/main.k create mode 100644 tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k create mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k create mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k create mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k create mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k create mode 100644 tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k create mode 100644 tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k create mode 100644 tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k create mode 100644 tests/test-branch/model/kcl.mod create mode 100644 tests/test-cluster/kcl.mod create mode 100644 tests/test-cluster/main.k create mode 100644 tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k create mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k create mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k create mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k create mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k create mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k create mode 100644 tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k create mode 100644 tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k create mode 100644 tests/test-cluster/model/kcl.mod create mode 100644 tests/test-stack/kcl.mod rename tests/{test-render => test-stack}/main.k (100%) rename tests/{test-render => test-stack}/model (100%) diff --git a/.github/workflows/on-pr.yaml b/.github/workflows/on-pr.yaml index 65429f9..44b939c 100644 --- a/.github/workflows/on-pr.yaml +++ b/.github/workflows/on-pr.yaml @@ -27,28 +27,30 @@ permissions: jobs: validate: - uses: unbounded-tech/workflows-crossplane/.github/workflows/validate.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/validate.yaml@feat/multi-api-support with: examples: | [ - { "example": "examples/psqlstacks/minimal.yaml" }, - { "example": "examples/psqlstacks/standard.yaml" } + { "example": "examples/psqlstacks/minimal.yaml", "api_path": "apis/psqlstacks" }, + { "example": "examples/psqlstacks/standard.yaml", "api_path": "apis/psqlstacks" }, + { "example": "examples/psqlclusters/minimal.yaml", "api_path": "apis/psqlclusters" }, + { "example": "examples/psqlclusters/standard.yaml","api_path": "apis/psqlclusters" }, + { "example": "examples/psqlbranches/same-namespace.yaml", "api_path": "apis/psqlbranches" } ] - api_path: apis/psqlstacks error_on_missing_schemas: true test: - uses: unbounded-tech/workflows-crossplane/.github/workflows/test.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/test.yaml@feat/multi-api-support e2e: - uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@feat/multi-api-support publish: needs: - validate - test - e2e - uses: unbounded-tech/workflows-crossplane/.github/workflows/publish.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/publish.yaml@feat/multi-api-support secrets: inherit with: tag: pr-${{ github.event.pull_request.number }}-${{ github.sha }} diff --git a/.github/workflows/on-push-main.yaml b/.github/workflows/on-push-main.yaml index 063182b..c70333d 100644 --- a/.github/workflows/on-push-main.yaml +++ b/.github/workflows/on-push-main.yaml @@ -23,21 +23,23 @@ permissions: jobs: validate: - uses: unbounded-tech/workflows-crossplane/.github/workflows/validate.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/validate.yaml@feat/multi-api-support with: examples: | [ - { "example": "examples/psqlstacks/minimal.yaml" }, - { "example": "examples/psqlstacks/standard.yaml" } + { "example": "examples/psqlstacks/minimal.yaml", "api_path": "apis/psqlstacks" }, + { "example": "examples/psqlstacks/standard.yaml", "api_path": "apis/psqlstacks" }, + { "example": "examples/psqlclusters/minimal.yaml", "api_path": "apis/psqlclusters" }, + { "example": "examples/psqlclusters/standard.yaml","api_path": "apis/psqlclusters" }, + { "example": "examples/psqlbranches/same-namespace.yaml", "api_path": "apis/psqlbranches" } ] - api_path: apis/psqlstacks error_on_missing_schemas: true test: - uses: unbounded-tech/workflows-crossplane/.github/workflows/test.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/test.yaml@feat/multi-api-support e2e: - uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@v2.20.0 + uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@feat/multi-api-support version-and-tag: name: Version and Tag diff --git a/Makefile b/Makefile index 8e68960..12e9926 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,7 @@ SHELL := /bin/bash PACKAGE ?= psql-stack +# Default XRD_DIR for legacy single-API targets; multi-API targets derive per-example. XRD_DIR := apis/psqlstacks COMPOSITION := $(XRD_DIR)/composition.yaml DEFINITION := $(XRD_DIR)/definition.yaml @@ -8,6 +9,10 @@ EXAMPLE_DEFAULT := examples/psqlstacks/standard.yaml RENDER_TESTS := $(wildcard tests/test-*) E2E_TESTS := $(wildcard tests/e2etest-*) +# Multi-API support: examples//.yaml maps to apis//. +# Helper macro: api_dir_for(example_path) → apis/ +api-dir = apis/$(word 2,$(subst /, ,$(1))) + clean: rm -rf _output rm -rf .up @@ -17,9 +22,13 @@ build: # Examples list - mirrors GitHub Actions workflow # Format: example_path::observed_resources_path (observed_resources_path is optional) +# api_path is derived from example_path via the api-dir macro (examples//... → apis//). EXAMPLES := \ examples/psqlstacks/minimal.yaml:: \ - examples/psqlstacks/standard.yaml:: + examples/psqlstacks/standard.yaml:: \ + examples/psqlclusters/minimal.yaml:: \ + examples/psqlclusters/standard.yaml:: \ + examples/psqlbranches/same-namespace.yaml:: # Render all examples (parallel execution, output shown per-job when complete) render\:all: diff --git a/apis/psqlbranches/composition.yaml b/apis/psqlbranches/composition.yaml new file mode 100644 index 0000000..846a4b8 --- /dev/null +++ b/apis/psqlbranches/composition.yaml @@ -0,0 +1,16 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: Composition +metadata: + name: psqlbranches.hops.ops.com.ai +spec: + compositeTypeRef: + apiVersion: hops.ops.com.ai/v1alpha1 + kind: PSQLBranch + mode: Pipeline + pipeline: + - functionRef: + name: hops-ops-psql-stackbranch + step: branch + - functionRef: + name: crossplane-contrib-function-auto-ready + step: crossplane-contrib-function-auto-ready diff --git a/apis/psqlbranches/definition.yaml b/apis/psqlbranches/definition.yaml new file mode 100644 index 0000000..a62e024 --- /dev/null +++ b/apis/psqlbranches/definition.yaml @@ -0,0 +1,191 @@ +apiVersion: apiextensions.crossplane.io/v2 +kind: CompositeResourceDefinition +metadata: + name: psqlbranches.hops.ops.com.ai +spec: + group: hops.ops.com.ai + names: + kind: PSQLBranch + plural: psqlbranches + scope: Namespaced + versions: + - name: v1alpha1 + referenceable: true + served: true + schema: + openAPIV3Schema: + description: | + PSQLBranch is an ephemeral CoW fork of a PSQLCluster, designed for + preview environments (one branch per PR) and short-lived data + branches. Composes a CSI VolumeSnapshot of the source's PVC plus a + new CNPG Cluster bootstrapped via `bootstrap.recovery.volumeSnapshots`. + + Cross-namespace forks are supported (preview-pr-N namespace forking a + source in team-app namespace) via a bridging VolumeSnapshotContent — + the render template composes a VolumeSnapshot in the source namespace + AND a static-bound VolumeSnapshot in the branch namespace, then the + new Cluster references the branch-ns VolumeSnapshot for recovery. + + Same-namespace forks collapse this into a single VolumeSnapshot. + + Required prerequisites on the target cluster: psql-stack (CNPG + operator), volume-snapshot-stack (snapshot-controller + CRDs), and + a source PSQLCluster with `branching.enabled: true`. + type: object + properties: + spec: + description: PSQLBranchSpec defines the desired state. + type: object + properties: + clusterName: + description: Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. + type: string + labels: + description: Custom labels merged with stack defaults; applied to every composed resource. + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true + managementPolicies: + description: Crossplane managementPolicies. Defaults to ["*"]. + type: array + items: + type: string + default: + - "*" + kubernetesProviderConfigRef: + description: Reference to the Kubernetes ProviderConfig used to apply composed resources. Defaults to clusterName. + type: object + properties: + name: + type: string + kind: + type: string + enum: + - ProviderConfig + - ClusterProviderConfig + + source: + description: | + Source PSQLCluster to fork. Cross-namespace supported — leave + `namespace` empty for same-namespace branching, or set it to + the team's namespace for preview-PR-style forks. + type: object + properties: + name: + description: Source PSQLCluster name (required). + type: string + namespace: + description: Source PSQLCluster namespace. Empty = same as branch. + type: string + default: "" + pvcName: + description: | + Source PVC name to snapshot. Empty = derive from CNPG + naming convention (`-1` for the primary + instance). + type: string + default: "" + snapshotClassName: + description: VolumeSnapshotClass to use for the snapshot. Defaults to "psql" (composed by psql-stack). + type: string + default: psql + required: + - name + + branch: + description: Branch cluster sizing. + type: object + properties: + instances: + description: Branch instance count. Defaults to 1 (HA on branches is unusual). + type: integer + default: 1 + storage: + description: Branch PVC sizing. Empty fields inherit from the source's storage on recovery. + type: object + properties: + size: + description: Branch PVC size, e.g. "10Gi". Empty = match source. + type: string + default: "" + class: + description: Branch StorageClass. Empty = inherit from source. + type: string + default: "" + + postgresql: + description: Postgres version on the branch (must match source for snapshot recovery to succeed). + type: object + properties: + version: + type: string + default: "17" + + scaleToZero: + description: | + Auto-hibernate the branch after `idleTimeout` of no client + activity. Default-on with aggressive timeout — preview-PR + branches should hibernate fast to keep cost flat. + type: object + properties: + enabled: + type: boolean + default: true + idleTimeout: + description: Idle duration before hibernate. Defaults to "10m" (more aggressive than PSQLCluster's 30m). + type: string + default: 10m + + ttl: + description: | + Recorded intent for branch expiration. v0 does NOT enforce + the TTL automatically — pair with an external cleanup + mechanism (Kyverno cleanupPolicies, ArgoCD ApplicationSet + with pullRequest generator that deletes on PR close, etc.). + Status reports `expiresAt` (creation + after) for tooling. + type: object + properties: + enabled: + type: boolean + default: false + after: + description: Duration after which the branch should be deleted. Defaults to "168h" (7 days). + type: string + default: 168h + + cnpg: + description: Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not covered by the toggles above. + type: object + properties: + values: + description: Merged into the Cluster.spec. + type: object + x-kubernetes-preserve-unknown-fields: true + overrideAllValues: + description: Replaces the entire Cluster.spec. + type: object + x-kubernetes-preserve-unknown-fields: true + required: + - clusterName + - source + status: + description: PSQLBranchStatus defines the observed state. + type: object + properties: + ready: + type: boolean + bootstrapPhase: + description: Current branch CNPG Cluster `.status.phase`. + type: string + bootstrapMethod: + description: Bootstrap method actually used. v0 always "volumeSnapshot". + type: string + sourceSnapshotContent: + description: Name of the cluster-scoped VolumeSnapshotContent backing this branch (for cross-ns bridging visibility). + type: string + expiresAt: + description: Computed deletion deadline when ttl.enabled is true. + type: string + required: + - spec diff --git a/apis/psqlclusters/composition.yaml b/apis/psqlclusters/composition.yaml new file mode 100644 index 0000000..d122120 --- /dev/null +++ b/apis/psqlclusters/composition.yaml @@ -0,0 +1,16 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: Composition +metadata: + name: psqlclusters.hops.ops.com.ai +spec: + compositeTypeRef: + apiVersion: hops.ops.com.ai/v1alpha1 + kind: PSQLCluster + mode: Pipeline + pipeline: + - functionRef: + name: hops-ops-psql-stackcluster + step: cluster + - functionRef: + name: crossplane-contrib-function-auto-ready + step: crossplane-contrib-function-auto-ready diff --git a/apis/psqlclusters/definition.yaml b/apis/psqlclusters/definition.yaml new file mode 100644 index 0000000..94c3e8e --- /dev/null +++ b/apis/psqlclusters/definition.yaml @@ -0,0 +1,208 @@ +apiVersion: apiextensions.crossplane.io/v2 +kind: CompositeResourceDefinition +metadata: + name: psqlclusters.hops.ops.com.ai +spec: + group: hops.ops.com.ai + names: + kind: PSQLCluster + plural: psqlclusters + scope: Namespaced + versions: + - name: v1alpha1 + referenceable: true + served: true + schema: + openAPIV3Schema: + description: | + PSQLCluster is the per-app Postgres primitive. Composes a CloudNativePG + `Cluster` plus the wiring (annotations, plugin entries, labels) required + for the platform integrations (scale-to-zero, branching, monitoring, HA). + + Companion to `psql-stack` (which provides the CNPG operator + S2Z plugin + substrate; required as a target-cluster prerequisite) and to `psql-branch` + (which composes ephemeral CoW forks of a PSQLCluster via VolumeSnapshot). + + Intent-first surface: integration toggles (`scaleToZero.enabled`, + `branching.enabled`, `monitoring.enabled`, `ha.enabled`) translate to + CNPG mechanics inside the render template — consumer manifests don't + reach for `cnpg.overrideAllValues` unless their use case is genuinely + novel. + type: object + properties: + spec: + description: PSQLClusterSpec defines the desired state. + type: object + properties: + clusterName: + description: Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. + type: string + labels: + description: Custom labels merged with stack defaults; applied to every composed resource. + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true + managementPolicies: + description: Crossplane managementPolicies applied to every composed resource. Defaults to ["*"]. + type: array + items: + type: string + default: + - "*" + kubernetesProviderConfigRef: + description: Reference to the Kubernetes ProviderConfig used to apply the Cluster CR + ExternalSecret. Defaults to clusterName. + type: object + properties: + name: + type: string + kind: + type: string + enum: + - ProviderConfig + - ClusterProviderConfig + + # ============================================================ + # Sizing + # ============================================================ + instances: + description: Number of Postgres instances. Default 1 (single-node). HA mode bumps to ha.replicas. + type: integer + default: 1 + storage: + description: Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (gp3 default on EKS Auto Mode) — bumping `size` upward grows the volume in place with no downtime. + type: object + properties: + size: + description: PVC size, e.g. "10Gi". Required. + type: string + class: + description: StorageClass name. Empty = cluster default (gp3 on Auto Mode). + type: string + default: "" + required: + - size + + # ============================================================ + # Postgres + # ============================================================ + postgresql: + description: Postgres version + tuning parameters. + type: object + properties: + version: + description: Postgres major version. Defaults to "17". + type: string + default: "17" + parameters: + description: postgres.conf parameters (e.g. `shared_buffers`, `max_connections`). + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true + + # ============================================================ + # Credentials — typed intent + # ============================================================ + credentials: + description: Where Postgres credentials are sourced from. + type: object + properties: + superuser: + type: object + properties: + managedBy: + description: ClusterSecretStore name for sourcing the superuser password from a secrets backend (default ESO ClusterSecretStore convention). Set "" to skip ExternalSecret composition (consumer must provide a K8s Secret named `` with `username` + `password` keys). + type: string + default: hops-aws-secrets-manager + secretName: + description: K8s Secret name receiving the credentials. Defaults to "-superuser". + type: string + default: "" + remoteKey: + description: Key in the remote secrets backend (AWS Secrets Manager) holding the superuser password. Defaults to "/superuser". + type: string + default: "" + + # ============================================================ + # Cross-stack integration toggles + # ============================================================ + scaleToZero: + description: Auto-hibernate the cluster after `idleTimeout` of no client activity. Requires the cnpg-i-scale-to-zero plugin on the target cluster (provided by psql-stack). When `ha.enabled` is also true, the cluster will not actually scale to zero (HA implies always-available); a Warning condition is emitted on the XR but the annotation is rendered for tooling that reads it. + type: object + properties: + enabled: + type: boolean + default: false + idleTimeout: + description: How long the cluster must be idle before hibernating. Defaults to "30m". + type: string + default: 30m + + branching: + description: Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). + type: object + properties: + enabled: + type: boolean + default: true + snapshotClassName: + description: VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. + type: string + default: psql + + monitoring: + description: Add Prometheus scrape configuration. CNPG operator handles the actual PodMonitor creation when `monitoring.enablePodMonitor` is set on the Cluster CR. + type: object + properties: + enabled: + type: boolean + default: true + + # ============================================================ + # HA + # ============================================================ + ha: + description: Stack-wide HA. Bumps instances to `replicas` (default 3) and adds zonal topology spread. + type: object + properties: + enabled: + type: boolean + default: false + replicas: + type: integer + default: 3 + topologySpreadByZone: + type: boolean + default: true + + # ============================================================ + # Escape hatches + # ============================================================ + cnpg: + description: Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. + type: object + properties: + values: + description: Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. + type: object + x-kubernetes-preserve-unknown-fields: true + overrideAllValues: + description: Replaces the entire Cluster.spec — total escape hatch. + type: object + x-kubernetes-preserve-unknown-fields: true + required: + - clusterName + - storage + status: + description: PSQLClusterStatus defines the observed state. + type: object + properties: + ready: + description: Overall readiness — true once the CNPG Cluster reaches `Cluster in healthy state` AND the ExternalSecret (if composed) is SecretSynced. + type: boolean + clusterPhase: + description: Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). + type: string + required: + - spec diff --git a/apis/psqlstacks/composition.yaml b/apis/psqlstacks/composition.yaml index fd35e11..ce963cc 100644 --- a/apis/psqlstacks/composition.yaml +++ b/apis/psqlstacks/composition.yaml @@ -9,8 +9,8 @@ spec: mode: Pipeline pipeline: - functionRef: - name: hops-ops-psql-stackrender - step: render + name: hops-ops-psql-stackstack + step: stack - functionRef: name: crossplane-contrib-function-auto-ready step: crossplane-contrib-function-auto-ready diff --git a/examples/psqlbranches/cross-namespace.yaml b/examples/psqlbranches/cross-namespace.yaml new file mode 100644 index 0000000..bf1e903 --- /dev/null +++ b/examples/psqlbranches/cross-namespace.yaml @@ -0,0 +1,18 @@ +# Cross-namespace branch — preview branch in `preview-pr-142` namespace +# forking a source PSQLCluster in `team-app` namespace. +# +# Composition: 1 VolumeSnapshot in `team-app` (of source PVC) + 1 bridging +# VolumeSnapshot in `preview-pr-142` (bound to the same VolumeSnapshotContent) +# + 1 CNPG Cluster in `preview-pr-142` (bootstrapped from the branch-ns +# VolumeSnapshot). +# +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLBranch +metadata: + name: pr-142 + namespace: preview-pr-142 +spec: + clusterName: production-cluster + source: + name: my-app + namespace: team-app # cross-namespace: branch ns differs from source ns diff --git a/examples/psqlbranches/preview-with-ttl.yaml b/examples/psqlbranches/preview-with-ttl.yaml new file mode 100644 index 0000000..4c669ef --- /dev/null +++ b/examples/psqlbranches/preview-with-ttl.yaml @@ -0,0 +1,22 @@ +# Preview-PR posture — aggressive S2Z + 7-day TTL recorded for external +# cleanup tooling (Kyverno cleanupPolicies, ArgoCD sync windows, etc.). +# +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLBranch +metadata: + name: pr-142 + namespace: preview-pr-142 +spec: + clusterName: production-cluster + labels: + pr: "142" + team: payments + source: + name: orders + namespace: payments + scaleToZero: + enabled: true + idleTimeout: 5m # very aggressive — preview envs hibernate quickly + ttl: + enabled: true + after: 168h # 7d diff --git a/examples/psqlbranches/same-namespace.yaml b/examples/psqlbranches/same-namespace.yaml new file mode 100644 index 0000000..a0361ba --- /dev/null +++ b/examples/psqlbranches/same-namespace.yaml @@ -0,0 +1,13 @@ +# Same-namespace branch — fork a PSQLCluster in the same namespace. +# Composition collapses to: 1 VolumeSnapshot (referencing source PVC directly) +# + 1 CNPG Cluster bootstrapped from it. +# +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLBranch +metadata: + name: pr-142 + namespace: default +spec: + clusterName: my-cluster + source: + name: my-app # source PSQLCluster in same namespace diff --git a/examples/psqlclusters/minimal.yaml b/examples/psqlclusters/minimal.yaml new file mode 100644 index 0000000..4579d89 --- /dev/null +++ b/examples/psqlclusters/minimal.yaml @@ -0,0 +1,12 @@ +# Minimal claim: single Postgres instance, ESO-managed superuser, branching on, +# monitoring on. Sub-10-line claim for typical app workloads. +# +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLCluster +metadata: + name: my-app + namespace: default +spec: + clusterName: my-cluster + storage: + size: 10Gi diff --git a/examples/psqlclusters/standard.yaml b/examples/psqlclusters/standard.yaml new file mode 100644 index 0000000..8ef59d3 --- /dev/null +++ b/examples/psqlclusters/standard.yaml @@ -0,0 +1,35 @@ +# Standard production posture: HA across zones, scaleToZero opted in for +# off-hours savings, custom postgres tuning, team labels. +# +# Note: when both ha.enabled and scaleToZero.enabled are true, the cluster +# won't actually scale to zero (HA implies always-available); a warning +# Condition flags this on the XR status. The annotation is still rendered +# for tooling that reads it. +# +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLCluster +metadata: + name: orders + namespace: orders +spec: + clusterName: production-cluster + labels: + team: payments + instances: 3 + storage: + size: 100Gi + postgresql: + version: "17" + parameters: + shared_buffers: "1GB" + effective_cache_size: "3GB" + ha: + enabled: true + replicas: 3 + topologySpreadByZone: true + scaleToZero: + enabled: false + branching: + enabled: true + monitoring: + enabled: true diff --git a/functions/branch/000-state-init.yaml.gotmpl b/functions/branch/000-state-init.yaml.gotmpl new file mode 100644 index 0000000..dc7d545 --- /dev/null +++ b/functions/branch/000-state-init.yaml.gotmpl @@ -0,0 +1,135 @@ +# code: language=yaml +# +# Initialize $state with spec defaults +# + +{{- $xr := getCompositeResource . }} +{{- $spec := $xr.spec | default dict }} +{{- $metadata := $xr.metadata | default dict }} + +# ============================================================================== +# Core defaults +# ============================================================================== +{{- $name := $metadata.name }} +{{- $namespace := $metadata.namespace | default "default" }} +{{- $clusterName := $spec.clusterName }} +{{- $managementPolicies := $spec.managementPolicies | default (list "*") }} + +{{- $defaultLabels := dict + "hops.ops.com.ai/managed" "true" + "hops.ops.com.ai/psql-branch" $name +}} +{{- $labels := merge $defaultLabels ($spec.labels | default dict) }} + +{{- $k8sProviderConfigRef := $spec.kubernetesProviderConfigRef | default dict }} +{{- $k8sProviderConfigRef = dict + "name" ($k8sProviderConfigRef.name | default $clusterName) + "kind" ($k8sProviderConfigRef.kind | default "ProviderConfig") +}} + +# ============================================================================== +# Source — the PSQLCluster being forked +# Default pvcName: "-1" (CNPG's primary instance naming). +# Default snapshotClassName: "psql" (composed by psql-stack). +# ============================================================================== +{{- $sourceSpec := $spec.source }} +{{- $sourceName := $sourceSpec.name }} +{{- $sourceNamespace := $sourceSpec.namespace | default "" }} +{{- if not $sourceNamespace }} + {{- $sourceNamespace = $namespace }} +{{- end }} +{{- $sourcePvcName := $sourceSpec.pvcName | default "" }} +{{- if not $sourcePvcName }} + {{- $sourcePvcName = printf "%s-1" $sourceName }} +{{- end }} +{{- $source := dict + "name" $sourceName + "namespace" $sourceNamespace + "pvcName" $sourcePvcName + "snapshotClassName" ($sourceSpec.snapshotClassName | default "psql") +}} + +# Cross-namespace bridging gate — render the source-ns VolumeSnapshot only when +# the source is in a different namespace from the branch. +{{- $crossNamespace := ne $sourceNamespace $namespace }} + +# ============================================================================== +# Branch sizing +# ============================================================================== +{{- $branchSpec := $spec.branch | default dict }} +{{- $branchStorageSpec := $branchSpec.storage | default dict }} +# CNPG requires spec.storage.size on the Cluster CR — default to "10Gi" when +# unspecified. Users with larger sources (>10Gi) MUST set this explicitly to +# at least the source PVC size; CNPG/EBS can't shrink during recovery. +{{- $branchSize := $branchStorageSpec.size | default "" }} +{{- if not $branchSize }} + {{- $branchSize = "10Gi" }} +{{- end }} +{{- $branch := dict + "instances" ($branchSpec.instances | default 1) + "storage" (dict + "size" $branchSize + "class" ($branchStorageSpec.class | default "") + ) +}} + +# ============================================================================== +# Postgres version (must match source for snapshot recovery to succeed) +# ============================================================================== +{{- $pgSpec := $spec.postgresql | default dict }} +{{- $postgresql := dict + "version" ($pgSpec.version | default "17") +}} + +# ============================================================================== +# Integration toggles +# ============================================================================== +{{- $s2zSpec := $spec.scaleToZero | default dict }} +{{- $s2zEnabled := true }} +{{- if hasKey $s2zSpec "enabled" }} + {{- $s2zEnabled = $s2zSpec.enabled }} +{{- end }} +{{- $scaleToZero := dict + "enabled" $s2zEnabled + "idleTimeout" ($s2zSpec.idleTimeout | default "10m") +}} + +{{- $ttlSpec := $spec.ttl | default dict }} +{{- $ttlEnabled := false }} +{{- if hasKey $ttlSpec "enabled" }} + {{- $ttlEnabled = $ttlSpec.enabled }} +{{- end }} +{{- $ttl := dict + "enabled" $ttlEnabled + "after" ($ttlSpec.after | default "168h") +}} + +# ============================================================================== +# Escape hatch +# ============================================================================== +{{- $cnpgSpec := $spec.cnpg | default dict }} +{{- $cnpg := dict + "values" ($cnpgSpec.values | default dict) + "overrideAllValues" ($cnpgSpec.overrideAllValues | default dict) +}} + +# ============================================================================== +# Initialize $state +# ============================================================================== +{{- $state := dict + "name" $name + "namespace" $namespace + "clusterName" $clusterName + "managementPolicies" $managementPolicies + "labels" $labels + "kubernetesProviderConfigRef" $k8sProviderConfigRef + "source" $source + "crossNamespace" $crossNamespace + "branch" $branch + "postgresql" $postgresql + "scaleToZero" $scaleToZero + "ttl" $ttl + "cnpg" $cnpg + "observed" (dict) + "status" (dict) +}} diff --git a/functions/branch/010-state-status.yaml.gotmpl b/functions/branch/010-state-status.yaml.gotmpl new file mode 100644 index 0000000..ea788a9 --- /dev/null +++ b/functions/branch/010-state-status.yaml.gotmpl @@ -0,0 +1,39 @@ +# code: language=yaml +# +# Extract observed state from composed resources +# + +{{- $observed := $.observed.resources | default dict }} + +# Branch CNPG Cluster phase — wraps the same provider-kubernetes Object pattern +# as PSQLCluster. +{{- $clusterEntry := get $observed "cnpg-cluster" | default dict }} +{{- $clusterResource := $clusterEntry.resource | default dict }} +{{- $clusterAtProvider := (($clusterResource.status | default dict).atProvider | default dict) }} +{{- $clusterManifest := $clusterAtProvider.manifest | default dict }} +{{- $clusterStatus := $clusterManifest.status | default dict }} +{{- $clusterPhase := $clusterStatus.phase | default "" }} + +# Read the bound VolumeSnapshotContent name from the branch-ns VolumeSnapshot +# (or the source-ns one when same-namespace). +{{- $snapEntry := get $observed "branch-snapshot" | default dict }} +{{- if not (get $observed "branch-snapshot") }} + {{- $snapEntry = get $observed "source-snapshot" | default dict }} +{{- end }} +{{- $snapResource := $snapEntry.resource | default dict }} +{{- $snapAtProvider := (($snapResource.status | default dict).atProvider | default dict) }} +{{- $snapManifest := $snapAtProvider.manifest | default dict }} +{{- $snapStatus := $snapManifest.status | default dict }} +{{- $snapContent := $snapStatus.boundVolumeSnapshotContentName | default "" }} + +{{- $state = set $state "observed" (dict + "cluster" (dict "phase" $clusterPhase) + "snapshotContent" $snapContent +) }} + +{{- $state = set $state "status" (dict + "ready" false + "bootstrapPhase" $clusterPhase + "bootstrapMethod" "volumeSnapshot" + "sourceSnapshotContent" $snapContent +) }} diff --git a/functions/branch/100-source-snapshot.yaml.gotmpl b/functions/branch/100-source-snapshot.yaml.gotmpl new file mode 100644 index 0000000..46be663 --- /dev/null +++ b/functions/branch/100-source-snapshot.yaml.gotmpl @@ -0,0 +1,43 @@ +# code: language=yaml +# +# Source-namespace VolumeSnapshot — cross-namespace branching only. +# +# When the source PSQLCluster is in a different namespace from the branch, +# we need a VolumeSnapshot in the source's namespace (because +# VolumeSnapshot.spec.source.persistentVolumeClaimName is same-namespace +# only). The snapshot-controller binds this to a cluster-scoped +# VolumeSnapshotContent, which the branch-ns VolumeSnapshot then references +# statically (in 110-branch-snapshot). +# +# For same-namespace branching, this template is skipped — the branch-ns +# snapshot references the source PVC directly. +# + +{{- $source := $state.source }} +{{- if $state.crossNamespace }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-source-snapshot + annotations: + {{ setResourceNameAnnotation "source-snapshot" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshot + metadata: + name: {{ $state.name }}-src + namespace: {{ $source.namespace }} + labels: {{ $state.labels | toJson }} + spec: + volumeSnapshotClassName: {{ $source.snapshotClassName }} + source: + persistentVolumeClaimName: {{ $source.pvcName }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/branch/110-branch-snapshot.yaml.gotmpl b/functions/branch/110-branch-snapshot.yaml.gotmpl new file mode 100644 index 0000000..25dbab2 --- /dev/null +++ b/functions/branch/110-branch-snapshot.yaml.gotmpl @@ -0,0 +1,45 @@ +# code: language=yaml +# +# Branch-namespace VolumeSnapshot — bound to either the source PVC directly +# (same-namespace branching) or to the VolumeSnapshotContent created from +# the source-namespace VolumeSnapshot (cross-namespace branching). +# +# The branch's CNPG Cluster references this VolumeSnapshot for +# `bootstrap.recovery.volumeSnapshots.storage.name` — recovery requires +# the VolumeSnapshot to be in the same namespace as the new Cluster. +# + +{{- $source := $state.source }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-branch-snapshot + annotations: + {{ setResourceNameAnnotation "branch-snapshot" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshot + metadata: + name: {{ $state.name }}-snap + namespace: {{ $state.namespace }} + labels: {{ $state.labels | toJson }} + spec: + volumeSnapshotClassName: {{ $source.snapshotClassName }} + source: + {{- if $state.crossNamespace }} + # Cross-ns branching — bind to the VolumeSnapshotContent created from + # the source-ns VolumeSnapshot. The content name is observed on the + # source-ns Object's status; static binding works once that exists. + volumeSnapshotContentName: {{ $state.observed.snapshotContent | quote }} + {{- else }} + # Same-ns branching — reference the source PVC directly. + persistentVolumeClaimName: {{ $source.pvcName }} + {{- end }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} diff --git a/functions/branch/200-cnpg-cluster.yaml.gotmpl b/functions/branch/200-cnpg-cluster.yaml.gotmpl new file mode 100644 index 0000000..7329e5d --- /dev/null +++ b/functions/branch/200-cnpg-cluster.yaml.gotmpl @@ -0,0 +1,98 @@ +# code: language=yaml +# +# Branch CNPG Cluster — bootstraps from the branch-namespace VolumeSnapshot. +# +# CNPG bootstrap.recovery.volumeSnapshots restores the entire Postgres data +# directory from the snapshot. Postgres roles + their hashed passwords are +# preserved, so the branch has the same `app` user as the source — but the +# K8s Secret holding that password is NOT auto-replicated to the branch +# namespace. App pods that need to connect should either: +# - Have a copy of the source's K8s Secret in the branch ns (manual or via +# ESO with the same remoteKey), or +# - Use a SecretReflector to mirror it +# This is documented in README. +# + +{{- $cnpg := $state.cnpg }} + +{{- $clusterAnnotations := dict }} +{{- if $state.scaleToZero.enabled }} + {{- $_ := set $clusterAnnotations "cnpg-i-scale-to-zero.xata.io/idle-timeout" $state.scaleToZero.idleTimeout }} +{{- end }} +{{- if $state.ttl.enabled }} + {{- $_ := set $clusterAnnotations "hops.ops.com.ai/ttl" $state.ttl.after }} +{{- end }} + +# Branch-only labels — distinguish from PSQLCluster's `branching-source` label. +{{- $clusterLabels := merge $state.labels (dict + "hops.ops.com.ai/branch-of" $state.source.name +) }} + +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-cnpg-cluster + annotations: + {{ setResourceNameAnnotation "cnpg-cluster" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: postgresql.cnpg.io/v1 + kind: Cluster + metadata: + name: {{ $state.name }} + namespace: {{ $state.namespace }} + {{- if $clusterAnnotations }} + annotations: + {{- range $k, $v := $clusterAnnotations }} + {{ $k | quote }}: {{ $v | quote }} + {{- end }} + {{- end }} + labels: + {{- range $k, $v := $clusterLabels }} + {{ $k }}: {{ $v | quote }} + {{- end }} + {{- if $cnpg.overrideAllValues }} + spec: + {{- toYaml $cnpg.overrideAllValues | nindent 8 }} + {{- else }} + spec: + {{- $clusterSpec := dict + "instances" $state.branch.instances + "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) + "bootstrap" (dict + "recovery" (dict + "volumeSnapshots" (dict + "storage" (dict + "name" (printf "%s-snap" $state.name) + "kind" "VolumeSnapshot" + "apiGroup" "snapshot.storage.k8s.io" + ) + ) + ) + ) + }} + {{- /* Storage — size always set (state-init defaults to 10Gi; consumer + must override for larger sources). class only set when explicitly + provided. */}} + {{- $storage := dict "size" $state.branch.storage.size }} + {{- if $state.branch.storage.class }} + {{- $_ := set $storage "storageClass" $state.branch.storage.class }} + {{- end }} + {{- $_ := set $clusterSpec "storage" $storage }} + {{- if $state.scaleToZero.enabled }} + {{- $_ := set $clusterSpec "plugins" (list (dict + "name" "cnpg-i-scale-to-zero.xata.io" + "enabled" true + )) }} + {{- end }} + {{- $_ := set $clusterSpec "monitoring" (dict "enablePodMonitor" true) }} + {{- $mergedClusterSpec := mergeOverwrite $clusterSpec ($cnpg.values | default dict) }} + {{- toYaml $mergedClusterSpec | nindent 8 }} + {{- end }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} diff --git a/functions/branch/999-status.yaml.gotmpl b/functions/branch/999-status.yaml.gotmpl new file mode 100644 index 0000000..5900a57 --- /dev/null +++ b/functions/branch/999-status.yaml.gotmpl @@ -0,0 +1,14 @@ +# code: language=yaml +# +# Output XR status +# + +{{- $xr := getCompositeResource . }} +--- +apiVersion: {{ $xr.apiVersion }} +kind: {{ $xr.kind }} +status: + ready: {{ $state.status.ready }} + bootstrapPhase: {{ $state.status.bootstrapPhase | quote }} + bootstrapMethod: {{ $state.status.bootstrapMethod | quote }} + sourceSnapshotContent: {{ $state.status.sourceSnapshotContent | quote }} diff --git a/functions/cluster/000-state-init.yaml.gotmpl b/functions/cluster/000-state-init.yaml.gotmpl new file mode 100644 index 0000000..a9e03a7 --- /dev/null +++ b/functions/cluster/000-state-init.yaml.gotmpl @@ -0,0 +1,171 @@ +# code: language=yaml +# +# Initialize $state with spec defaults +# + +{{- $xr := getCompositeResource . }} +{{- $spec := $xr.spec | default dict }} +{{- $metadata := $xr.metadata | default dict }} + +# ============================================================================== +# Core defaults +# ============================================================================== +{{- $name := $metadata.name }} +{{- $namespace := $metadata.namespace | default "default" }} +{{- $clusterName := $spec.clusterName }} +{{- $managementPolicies := $spec.managementPolicies | default (list "*") }} + +# Labels +{{- $defaultLabels := dict + "hops.ops.com.ai/managed" "true" + "hops.ops.com.ai/psql-cluster" $name +}} +{{- $labels := merge $defaultLabels ($spec.labels | default dict) }} + +# Kubernetes provider config (defaults to clusterName) +{{- $k8sProviderConfigRef := $spec.kubernetesProviderConfigRef | default dict }} +{{- $k8sProviderConfigRef = dict + "name" ($k8sProviderConfigRef.name | default $clusterName) + "kind" ($k8sProviderConfigRef.kind | default "ProviderConfig") +}} + +# ============================================================================== +# HA — flips instances to ha.replicas + zonal topology spread. +# When ha.enabled and scaleToZero.enabled are both true, the cluster won't +# actually scale to zero (HA implies always-available); a Warning condition +# gets emitted on the XR but the annotation is still rendered for tooling. +# ============================================================================== +{{- $haSpec := $spec.ha | default dict }} +{{- $haEnabled := false }} +{{- if hasKey $haSpec "enabled" }} + {{- $haEnabled = $haSpec.enabled }} +{{- end }} +{{- $haReplicas := $haSpec.replicas | default 3 }} +{{- $haSpreadByZone := true }} +{{- if hasKey $haSpec "topologySpreadByZone" }} + {{- $haSpreadByZone = $haSpec.topologySpreadByZone }} +{{- end }} +{{- $ha := dict + "enabled" $haEnabled + "replicas" $haReplicas + "topologySpreadByZone" $haSpreadByZone +}} + +# Resolve effective instance count: HA wins +{{- $instances := $spec.instances | default 1 }} +{{- if $haEnabled }} + {{- $instances = $haReplicas }} +{{- end }} + +# ============================================================================== +# Storage +# ============================================================================== +{{- $storageSpec := $spec.storage }} +{{- $storage := dict + "size" $storageSpec.size + "class" ($storageSpec.class | default "") +}} + +# ============================================================================== +# Postgres +# ============================================================================== +{{- $pgSpec := $spec.postgresql | default dict }} +{{- $postgresql := dict + "version" ($pgSpec.version | default "17") + "parameters" ($pgSpec.parameters | default dict) +}} + +# ============================================================================== +# Credentials +# ============================================================================== +{{- $credSpec := ($spec.credentials | default dict).superuser | default dict }} +{{- $credManagedBy := "hops-aws-secrets-manager" }} +{{- if hasKey $credSpec "managedBy" }} + {{- $credManagedBy = $credSpec.managedBy }} +{{- end }} +{{- $credSecretName := $credSpec.secretName }} +{{- if not $credSecretName }} + {{- $credSecretName = printf "%s-superuser" $name }} +{{- end }} +{{- $credRemoteKey := $credSpec.remoteKey }} +{{- if not $credRemoteKey }} + {{- $credRemoteKey = printf "%s/superuser" $name }} +{{- end }} +{{- $credentials := dict + "superuser" (dict + "managedBy" $credManagedBy + "secretName" $credSecretName + "remoteKey" $credRemoteKey + ) +}} + +# ============================================================================== +# Integration toggles +# ============================================================================== +{{- $s2zSpec := $spec.scaleToZero | default dict }} +{{- $s2zEnabled := false }} +{{- if hasKey $s2zSpec "enabled" }} + {{- $s2zEnabled = $s2zSpec.enabled }} +{{- end }} +{{- $scaleToZero := dict + "enabled" $s2zEnabled + "idleTimeout" ($s2zSpec.idleTimeout | default "30m") +}} + +{{- $branchSpec := $spec.branching | default dict }} +{{- $branchEnabled := true }} +{{- if hasKey $branchSpec "enabled" }} + {{- $branchEnabled = $branchSpec.enabled }} +{{- end }} +{{- $branching := dict + "enabled" $branchEnabled + "snapshotClassName" ($branchSpec.snapshotClassName | default "psql") +}} + +{{- $monSpec := $spec.monitoring | default dict }} +{{- $monEnabled := true }} +{{- if hasKey $monSpec "enabled" }} + {{- $monEnabled = $monSpec.enabled }} +{{- end }} +{{- $monitoring := dict "enabled" $monEnabled }} + +# ============================================================================== +# Escape hatch +# ============================================================================== +{{- $cnpgSpec := $spec.cnpg | default dict }} +{{- $cnpg := dict + "values" ($cnpgSpec.values | default dict) + "overrideAllValues" ($cnpgSpec.overrideAllValues | default dict) +}} + +# ============================================================================== +# Initialize $state +# ============================================================================== +{{- $state := dict + "name" $name + "namespace" $namespace + "clusterName" $clusterName + "managementPolicies" $managementPolicies + "labels" $labels + "kubernetesProviderConfigRef" $k8sProviderConfigRef + "instances" $instances + "storage" $storage + "postgresql" $postgresql + "credentials" $credentials + "scaleToZero" $scaleToZero + "branching" $branching + "monitoring" $monitoring + "ha" $ha + "cnpg" $cnpg + "observed" (dict) + "status" (dict) + "warnings" (list) +}} + +# ============================================================================== +# Validate cross-toggle combinations +# ============================================================================== +{{- if and $haEnabled $s2zEnabled }} + {{- $warnings := append $state.warnings "scaleToZero+ha both enabled: HA implies always-available, the cluster will not actually scale to zero" }} + {{- $state = set $state "warnings" $warnings }} +{{- end }} diff --git a/functions/cluster/010-state-status.yaml.gotmpl b/functions/cluster/010-state-status.yaml.gotmpl new file mode 100644 index 0000000..d66a668 --- /dev/null +++ b/functions/cluster/010-state-status.yaml.gotmpl @@ -0,0 +1,46 @@ +# code: language=yaml +# +# Extract observed state from composed resources +# + +{{- $observed := $.observed.resources | default dict }} + +# ============================================================================== +# Cluster status — read CNPG Cluster's `.status.phase` from the wrapped Object's +# `.status.atProvider.manifest.status.phase`. The Object's Ready=true means the +# manifest applied; the wrapped Cluster's phase tells us if Postgres is healthy. +# ============================================================================== +{{- $clusterEntry := get $observed "cnpg-cluster" | default dict }} +{{- $clusterResource := $clusterEntry.resource | default dict }} +{{- $clusterAtProvider := (($clusterResource.status | default dict).atProvider | default dict) }} +{{- $clusterManifest := $clusterAtProvider.manifest | default dict }} +{{- $clusterStatus := $clusterManifest.status | default dict }} +{{- $clusterPhase := $clusterStatus.phase | default "" }} +{{- $clusterReady := false }} +{{- if eq $clusterPhase "Cluster in healthy state" }} + {{- $clusterReady = true }} +{{- end }} + +# ============================================================================== +# ExternalSecret status — Object Ready=true means the ExternalSecret CR exists. +# We don't inspect the wrapped status further; SecretSynced is fast enough. +# ============================================================================== +{{- $esEntry := get $observed "external-secret" | default dict }} +{{- $esResource := $esEntry.resource | default dict }} +{{- $esConditions := (($esResource.status | default dict).conditions | default list) }} +{{- $esReady := false }} +{{- range $esConditions }} + {{- if and (eq .type "Ready") (eq .status "True") }} + {{- $esReady = true }} + {{- end }} +{{- end }} + +{{- $state = set $state "observed" (dict + "cluster" (dict "ready" $clusterReady "phase" $clusterPhase) + "externalSecret" (dict "ready" $esReady) +) }} + +{{- $state = set $state "status" (dict + "ready" false + "clusterPhase" $clusterPhase +) }} diff --git a/functions/cluster/100-external-secret.yaml.gotmpl b/functions/cluster/100-external-secret.yaml.gotmpl new file mode 100644 index 0000000..a09ab08 --- /dev/null +++ b/functions/cluster/100-external-secret.yaml.gotmpl @@ -0,0 +1,56 @@ +# code: language=yaml +# +# ExternalSecret — sources the Postgres superuser password from the platform's +# ClusterSecretStore (default: hops-aws-secrets-manager) and writes it to a +# K8s Secret that the CNPG Cluster will reference for bootstrap. +# +# Skipped when credentials.superuser.managedBy is empty — consumer must +# pre-create the K8s Secret themselves in that case. +# + +{{- $cred := $state.credentials.superuser }} +{{- if $cred.managedBy }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-external-secret + annotations: + {{ setResourceNameAnnotation "external-secret" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: external-secrets.io/v1beta1 + kind: ExternalSecret + metadata: + name: {{ $cred.secretName }} + namespace: {{ $state.namespace }} + labels: {{ $state.labels | toJson }} + spec: + refreshInterval: 1h + secretStoreRef: + name: {{ $cred.managedBy }} + kind: ClusterSecretStore + target: + name: {{ $cred.secretName }} + creationPolicy: Owner + template: + type: kubernetes.io/basic-auth + data: + username: "{{ "{{" }} .username {{ "}}" }}" + password: "{{ "{{" }} .password {{ "}}" }}" + data: + - secretKey: username + remoteRef: + key: {{ $cred.remoteKey | quote }} + property: username + - secretKey: password + remoteRef: + key: {{ $cred.remoteKey | quote }} + property: password + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/functions/cluster/200-cnpg-cluster.yaml.gotmpl b/functions/cluster/200-cnpg-cluster.yaml.gotmpl new file mode 100644 index 0000000..0e5d284 --- /dev/null +++ b/functions/cluster/200-cnpg-cluster.yaml.gotmpl @@ -0,0 +1,108 @@ +# code: language=yaml +# +# CNPG Cluster CR — the core composed resource. +# +# Wrapped in a Crossplane Object (provider-kubernetes) so we can apply it +# without provider-helm and so its lifecycle is owned by the PSQLCluster XR. +# +# Translates the intent-first integration toggles into CNPG mechanics: +# - scaleToZero.enabled → annotation + plugins entry +# - branching.enabled → labels for PSQLBranch consumption +# - monitoring.enabled → spec.monitoring.enablePodMonitor +# - ha.enabled → spec.instances + spec.affinity.topologyKey +# + +{{- $cnpg := $state.cnpg }} + +# Annotations on the Cluster CR (for plugin tooling) +{{- $clusterAnnotations := dict }} +{{- if $state.scaleToZero.enabled }} + {{- $_ := set $clusterAnnotations "cnpg-i-scale-to-zero.xata.io/idle-timeout" $state.scaleToZero.idleTimeout }} +{{- end }} + +# Labels — stack defaults + branching markers (when enabled) read by PSQLBranch +{{- $clusterLabels := merge $state.labels (dict) }} +{{- if $state.branching.enabled }} + {{- $_ := set $clusterLabels "hops.ops.com.ai/branching-source" "true" }} + {{- $_ := set $clusterLabels "hops.ops.com.ai/snapshot-class" $state.branching.snapshotClassName }} +{{- end }} + +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-cnpg-cluster + annotations: + {{ setResourceNameAnnotation "cnpg-cluster" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: postgresql.cnpg.io/v1 + kind: Cluster + metadata: + name: {{ $state.name }} + namespace: {{ $state.namespace }} + {{- if $clusterAnnotations }} + annotations: + {{- range $k, $v := $clusterAnnotations }} + {{ $k | quote }}: {{ $v | quote }} + {{- end }} + {{- end }} + labels: + {{- range $k, $v := $clusterLabels }} + {{ $k }}: {{ $v | quote }} + {{- end }} + {{- if $cnpg.overrideAllValues }} + spec: + {{- toYaml $cnpg.overrideAllValues | nindent 8 }} + {{- else }} + spec: + {{- /* Build the Cluster.spec from intent-first state, then mergeOverwrite + with $cnpg.values for fine-grained passthrough overrides. */}} + {{- $clusterSpec := dict + "instances" $state.instances + "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) + "bootstrap" (dict + "initdb" (dict + "database" "app" + "owner" "app" + "secret" (dict "name" $state.credentials.superuser.secretName) + ) + ) + "storage" (dict + "size" $state.storage.size + ) + "monitoring" (dict + "enablePodMonitor" $state.monitoring.enabled + ) + }} + {{- if $state.storage.class }} + {{- $_ := set $clusterSpec.storage "storageClass" $state.storage.class }} + {{- end }} + {{- if $state.postgresql.parameters }} + {{- $_ := set $clusterSpec "postgresql" (dict "parameters" $state.postgresql.parameters) }} + {{- end }} + {{- if $state.scaleToZero.enabled }} + {{- $_ := set $clusterSpec "plugins" (list (dict + "name" "cnpg-i-scale-to-zero.xata.io" + "enabled" true + )) }} + {{- end }} + {{- if $state.ha.enabled }} + {{- $affinity := dict + "enablePodAntiAffinity" true + "podAntiAffinityType" "preferred" + }} + {{- if $state.ha.topologySpreadByZone }} + {{- $_ := set $affinity "topologyKey" "topology.kubernetes.io/zone" }} + {{- end }} + {{- $_ := set $clusterSpec "affinity" $affinity }} + {{- end }} + {{- $mergedClusterSpec := mergeOverwrite $clusterSpec ($cnpg.values | default dict) }} + {{- toYaml $mergedClusterSpec | nindent 8 }} + {{- end }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} diff --git a/functions/cluster/999-status.yaml.gotmpl b/functions/cluster/999-status.yaml.gotmpl new file mode 100644 index 0000000..08d7187 --- /dev/null +++ b/functions/cluster/999-status.yaml.gotmpl @@ -0,0 +1,22 @@ +# code: language=yaml +# +# Output XR status + Warning conditions for cross-toggle conflicts. +# + +{{- $xr := getCompositeResource . }} +--- +apiVersion: {{ $xr.apiVersion }} +kind: {{ $xr.kind }} +status: + ready: {{ $state.status.ready }} + clusterPhase: {{ $state.status.clusterPhase | quote }} + {{- if $state.warnings }} + conditions: + {{- range $i, $msg := $state.warnings }} + - type: ConfigurationWarning + status: "True" + reason: ConflictingToggles + message: {{ $msg | quote }} + lastTransitionTime: "1970-01-01T00:00:00Z" + {{- end }} + {{- end }} diff --git a/functions/render/000-state-init.yaml.gotmpl b/functions/stack/000-state-init.yaml.gotmpl similarity index 100% rename from functions/render/000-state-init.yaml.gotmpl rename to functions/stack/000-state-init.yaml.gotmpl diff --git a/functions/render/010-state-status.yaml.gotmpl b/functions/stack/010-state-status.yaml.gotmpl similarity index 100% rename from functions/render/010-state-status.yaml.gotmpl rename to functions/stack/010-state-status.yaml.gotmpl diff --git a/functions/render/170-volumesnapshotclass.yaml.gotmpl b/functions/stack/170-volumesnapshotclass.yaml.gotmpl similarity index 100% rename from functions/render/170-volumesnapshotclass.yaml.gotmpl rename to functions/stack/170-volumesnapshotclass.yaml.gotmpl diff --git a/functions/render/200-cnpg-operator.yaml.gotmpl b/functions/stack/200-cnpg-operator.yaml.gotmpl similarity index 100% rename from functions/render/200-cnpg-operator.yaml.gotmpl rename to functions/stack/200-cnpg-operator.yaml.gotmpl diff --git a/functions/render/210-cnpg-scale-to-zero.yaml.gotmpl b/functions/stack/210-cnpg-scale-to-zero.yaml.gotmpl similarity index 100% rename from functions/render/210-cnpg-scale-to-zero.yaml.gotmpl rename to functions/stack/210-cnpg-scale-to-zero.yaml.gotmpl diff --git a/functions/render/220-atlas-operator.yaml.gotmpl b/functions/stack/220-atlas-operator.yaml.gotmpl similarity index 100% rename from functions/render/220-atlas-operator.yaml.gotmpl rename to functions/stack/220-atlas-operator.yaml.gotmpl diff --git a/functions/render/999-status.yaml.gotmpl b/functions/stack/999-status.yaml.gotmpl similarity index 100% rename from functions/render/999-status.yaml.gotmpl rename to functions/stack/999-status.yaml.gotmpl diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index bececbf..ec04191 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -1,13 +1,27 @@ import datetime import math -import models.ai.com.ops.hops.v1alpha1 as stacksv1alpha1 +import models.ai.com.ops.hops.v1alpha1 as hopsv1alpha1 import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 # ============================================================================== -# E2E Test for Psql (Kubernetes-only) +# Unified E2E Test for psql-stack package # -# Both components run in-cluster: Helm releases only. -# No cloud credentials needed - uses InjectedIdentity for Helm provider. +# Exercises all three XRs in the package together: +# - PSQLStack (installs CNPG operator + atlas-operator via Helm) +# - PSQLCluster (per-app Postgres XR; composes a CNPG Cluster CR + ExternalSecret) +# - PSQLBranch (ephemeral fork; composes VolumeSnapshot + bootstrapped CNPG Cluster) +# +# Bar: defaultConditions = ["Synced"]. We verify each XR composes its children +# correctly. We do NOT wait for the wrapped CNPG Cluster / VolumeSnapshot to +# reconcile — that requires CNPG + snapshot-controller to be live in-cluster. +# +# TODO (after volume-snapshot-stack v0.1.0 releases): upgrade this test to +# defaultConditions = ["Ready"] using the observe e2e pattern: +# - initResources: install volume-snapshot-stack Configuration package +# - extraResources: VolumeSnapshotStack XR (deploys snapshot-controller) +# - bump timeoutSeconds to ~5400 (PSQLStack Helm install + CNPG bootstrap + +# real PVC + snapshot + bootstrap-from-snapshot is a long chain) +# Tracked in tasks/merge-psql-client-apis-into-stack progress log. # # Run: make e2e # Run without cleanup: up test run tests/e2etest-psql --e2e --skip-control-plane-cleanup @@ -15,6 +29,8 @@ import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 _now = str(int(math.floor(datetime.ticks()))) _test_name = "e2e-psql-" + _now +_cluster_name = "e2e-psql-cluster-" + _now +_branch_name = "e2e-psql-branch-" + _now _items = [ metav1alpha1.E2ETest { @@ -23,18 +39,18 @@ _items = [ crossplane = { autoUpgrade.channel = "Rapid" } - defaultConditions = ["Ready"] - timeoutSeconds = 1800 # 30 minutes - 2 Helm charts + CRDs - cleanupTimeoutSeconds = 900 # 15 minutes + defaultConditions = ["Synced"] + timeoutSeconds = 1800 # 30 min — Helm install + composition apply + cleanupTimeoutSeconds = 900 # 15 min skipDelete = False # ============================================================== - # extraResources: RBAC + Helm ProviderConfig - # These are NOT deleted during cleanup + # extraResources: RBAC + Provider configs + # NOT deleted during cleanup # ============================================================== extraResources = [ - # Grant cluster-admin to all SAs in crossplane-system namespace - # Required for Helm provider to create namespaces and install charts + # cluster-admin to all SAs in crossplane-system so Helm + Kubernetes + # providers can apply wrapped manifests freely. { apiVersion = "rbac.authorization.k8s.io/v1" kind = "ClusterRoleBinding" @@ -52,7 +68,7 @@ _items = [ } ] } - # Helm ProviderConfig - uses in-cluster identity + # Helm ProviderConfig — for PSQLStack (installs CNPG + atlas-operator) { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "ProviderConfig" @@ -66,14 +82,29 @@ _items = [ } } } + # Kubernetes ProviderConfig — for PSQLCluster + PSQLBranch + { + apiVersion = "kubernetes.m.crossplane.io/v1beta1" + kind = "ProviderConfig" + metadata = { + name = "default" + namespace = "default" + } + spec = { + credentials = { + source = "InjectedIdentity" + } + } + } ] # ============================================================== - # manifests: Psql XR + # manifests: all three XRs applied together # Deleted during cleanup (unless --skip-delete) # ============================================================== manifests = [ - stacksv1alpha1.PSQLStack { + # 1. PSQLStack — Helm-installs CNPG operator + atlas-operator + VolumeSnapshotClass + hopsv1alpha1.PSQLStack { metadata = { name = _test_name namespace = "default" @@ -101,6 +132,48 @@ _items = [ } } } + # 2. PSQLCluster — per-app Postgres XR (composition only; CNPG CRDs + # aren't reconciled in kind without the operator being live) + hopsv1alpha1.PSQLCluster { + metadata = { + name = _cluster_name + namespace = "default" + } + spec = { + clusterName = "default" + labels = { + "hops.ops.com.ai/e2etest" = "true" + "hops.ops.com.ai/test-run" = _now + } + kubernetesProviderConfigRef = { + name = "default" + kind = "ProviderConfig" + } + storage = {size = "1Gi"} + # Skip ExternalSecret in e2e — no ESO ClusterSecretStore in kind harness. + credentials.superuser.managedBy = "" + } + } + # 3. PSQLBranch — ephemeral fork (composition only; VolumeSnapshot + # + recovery bootstrap not exercised without snapshot-controller) + hopsv1alpha1.PSQLBranch { + metadata = { + name = _branch_name + namespace = "default" + } + spec = { + clusterName = "default" + labels = { + "hops.ops.com.ai/e2etest" = "true" + "hops.ops.com.ai/test-run" = _now + } + kubernetesProviderConfigRef = { + name = "default" + kind = "ProviderConfig" + } + source = {name = _cluster_name} + } + } ] } } diff --git a/tests/test-render/kcl.mod b/tests/test-branch/kcl.mod similarity index 100% rename from tests/test-render/kcl.mod rename to tests/test-branch/kcl.mod diff --git a/tests/test-branch/main.k b/tests/test-branch/main.k new file mode 100644 index 0000000..9d2500e --- /dev/null +++ b/tests/test-branch/main.k @@ -0,0 +1,329 @@ +import models.ai.com.ops.hops.v1alpha1 as stacksv1alpha1 +import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 + +# ============================================================================== +# Unit tests for PSQLBranch XRD +# +# Verify: cross-ns vs same-ns composition, scaleToZero defaults, TTL annotation, +# overrideAllValues escape hatch, providerConfig defaults. +# ============================================================================== + +_items = [ + # ========================================================================== + # Test 1: same-namespace branch — only branch-snapshot composed (no source- + # snapshot bridging), references source PVC directly. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "same-namespace-branch-direct-pvc-reference" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-same", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src-app"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-same-branch-snapshot" + spec.forProvider.manifest = { + apiVersion = "snapshot.storage.k8s.io/v1" + kind = "VolumeSnapshot" + metadata = { + name = "br-same-snap" + namespace = "default" + } + spec = { + volumeSnapshotClassName = "psql" + source.persistentVolumeClaimName = "src-app-1" + } + } + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-same-cnpg-cluster" + spec.forProvider.manifest = { + apiVersion = "postgresql.cnpg.io/v1" + kind = "Cluster" + metadata = { + name = "br-same" + namespace = "default" + labels = { + "hops.ops.com.ai/managed" = "true" + "hops.ops.com.ai/psql-branch" = "br-same" + "hops.ops.com.ai/branch-of" = "src-app" + } + } + spec = { + instances = 1 + imageName = "ghcr.io/cloudnative-pg/postgresql:17" + bootstrap.recovery.volumeSnapshots.storage = { + name = "br-same-snap" + kind = "VolumeSnapshot" + apiGroup = "snapshot.storage.k8s.io" + } + } + } + } + ] + } + } + + # ========================================================================== + # Test 2: cross-namespace branch — source-snapshot in source ns + branch- + # snapshot in branch ns + Cluster in branch ns. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "cross-namespace-branch-bridges-via-content" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-x", namespace = "preview-pr-1"} + spec = { + clusterName = "prod" + source = {name = "src-app", namespace = "team-app"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-x-source-snapshot" + spec.forProvider.manifest = { + apiVersion = "snapshot.storage.k8s.io/v1" + kind = "VolumeSnapshot" + metadata = { + name = "br-x-src" + namespace = "team-app" + } + spec.source.persistentVolumeClaimName = "src-app-1" + } + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-x-branch-snapshot" + spec.forProvider.manifest.metadata = { + name = "br-x-snap" + namespace = "preview-pr-1" + } + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-x-cnpg-cluster" + spec.forProvider.manifest.metadata = { + name = "br-x" + namespace = "preview-pr-1" + } + } + ] + } + } + + # ========================================================================== + # Test 3: scaleToZero default-on; renders annotation + plugin entry. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "scaletozero-default-on-renders-annotation-and-plugin" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-s2z", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-s2z-cnpg-cluster" + spec.forProvider.manifest = { + metadata.annotations = { + "cnpg-i-scale-to-zero.xata.io/idle-timeout" = "10m" + } + spec.plugins = [{ + name = "cnpg-i-scale-to-zero.xata.io" + enabled = True + }] + } + } + ] + } + } + + # ========================================================================== + # Test 4: scaleToZero.enabled=false suppresses annotation + plugin. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "scaletozero-disabled-strips-plugin" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-no-s2z", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src"} + scaleToZero = {enabled = False} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-no-s2z-cnpg-cluster" + spec.forProvider.manifest.metadata = { + name = "br-no-s2z" + namespace = "default" + } + } + ] + } + } + + # ========================================================================== + # Test 5: ttl.enabled=true adds the hops/ttl annotation on the Cluster CR. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "ttl-enabled-adds-annotation" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-ttl", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src"} + ttl = {enabled = True, after = "24h"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-ttl-cnpg-cluster" + spec.forProvider.manifest.metadata.annotations = { + "cnpg-i-scale-to-zero.xata.io/idle-timeout" = "10m" + "hops.ops.com.ai/ttl" = "24h" + } + } + ] + } + } + + # ========================================================================== + # Test 6: custom snapshotClassName flows through to the VolumeSnapshot. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "custom-snapshot-class-flows-through" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-custom-vsc", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src", snapshotClassName = "longhorn-snap"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-custom-vsc-branch-snapshot" + spec.forProvider.manifest.spec.volumeSnapshotClassName = "longhorn-snap" + } + ] + } + } + + # ========================================================================== + # Test 7: cnpg.overrideAllValues replaces the entire Cluster.spec. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "cnpg-override-all-values-replaces-cluster-spec" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-override", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src"} + cnpg.overrideAllValues = { + instances = 5 + imageName = "custom/postgres:99" + } + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-override-cnpg-cluster" + spec.forProvider.manifest.spec = { + instances = 5 + imageName = "custom/postgres:99" + } + } + ] + } + } + + # ========================================================================== + # Test 8: kubernetesProviderConfigRef defaults to clusterName. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "k8s-provider-config-defaults-to-cluster-name" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-pc", namespace = "default"} + spec = { + clusterName = "prod-cluster" + source = {name = "src"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-pc-cnpg-cluster" + spec.providerConfigRef = { + name = "prod-cluster" + kind = "ProviderConfig" + } + } + ] + } + } +] + +items = _items diff --git a/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k b/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k new file mode 100644 index 0000000..2b4276e --- /dev/null +++ b/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k @@ -0,0 +1,481 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema PSQLBranch: + r""" + PSQLBranch is an ephemeral CoW fork of a PSQLCluster, designed for + preview environments (one branch per PR) and short-lived data + branches. Composes a CSI VolumeSnapshot of the source's PVC plus a + new CNPG Cluster bootstrapped via `bootstrap.recovery.volumeSnapshots`. + + Cross-namespace forks are supported (preview-pr-N namespace forking a + source in team-app namespace) via a bridging VolumeSnapshotContent — + the render template composes a VolumeSnapshot in the source namespace + AND a static-bound VolumeSnapshot in the branch namespace, then the + new Cluster references the branch-ns VolumeSnapshot for recovery. + + Same-namespace forks collapse this into a single VolumeSnapshot. + + Required prerequisites on the target cluster: psql-stack (CNPG + operator), volume-snapshot-stack (snapshot-controller + CRDs), and + a source PSQLCluster with `branching.enabled: true`. + + + Attributes + ---------- + apiVersion : str, default is "hops.ops.com.ai/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "PSQLBranch", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : HopsOpsComAiV1alpha1PSQLBranchSpec, default is Undefined, required + spec + status : HopsOpsComAiV1alpha1PSQLBranchStatus, default is Undefined, optional + status + """ + + + apiVersion: "hops.ops.com.ai/v1alpha1" = "hops.ops.com.ai/v1alpha1" + + kind: "PSQLBranch" = "PSQLBranch" + + metadata?: v1.ObjectMeta + + spec: HopsOpsComAiV1alpha1PSQLBranchSpec + + status?: HopsOpsComAiV1alpha1PSQLBranchStatus + + +schema HopsOpsComAiV1alpha1PSQLBranchSpec: + r""" + PSQLBranchSpec defines the desired state. + + Attributes + ---------- + branch : HopsOpsComAiV1alpha1PSQLBranchSpecBranch, default is Undefined, optional + branch + clusterName : str, default is Undefined, required + Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. + cnpg : HopsOpsComAiV1alpha1PSQLBranchSpecCnpg, default is Undefined, optional + cnpg + crossplane : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplane, default is Undefined, optional + crossplane + kubernetesProviderConfigRef : HopsOpsComAiV1alpha1PSQLBranchSpecKubernetesProviderConfigRef, default is Undefined, optional + kubernetes provider config ref + labels : {str:str}, default is Undefined, optional + Custom labels merged with stack defaults; applied to every composed resource. + managementPolicies : [str], default is ["*"], optional + Crossplane managementPolicies. Defaults to ["*"]. + postgresql : HopsOpsComAiV1alpha1PSQLBranchSpecPostgresql, default is Undefined, optional + postgresql + scaleToZero : HopsOpsComAiV1alpha1PSQLBranchSpecScaleToZero, default is Undefined, optional + scale to zero + source : HopsOpsComAiV1alpha1PSQLBranchSpecSource, default is Undefined, required + source + ttl : HopsOpsComAiV1alpha1PSQLBranchSpecTTL, default is Undefined, optional + ttl + """ + + + branch?: HopsOpsComAiV1alpha1PSQLBranchSpecBranch + + clusterName: str + + cnpg?: HopsOpsComAiV1alpha1PSQLBranchSpecCnpg + + crossplane?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplane + + kubernetesProviderConfigRef?: HopsOpsComAiV1alpha1PSQLBranchSpecKubernetesProviderConfigRef + + labels?: {str:str} + + managementPolicies?: [str] = ["*"] + + postgresql?: HopsOpsComAiV1alpha1PSQLBranchSpecPostgresql + + scaleToZero?: HopsOpsComAiV1alpha1PSQLBranchSpecScaleToZero + + source: HopsOpsComAiV1alpha1PSQLBranchSpecSource + + ttl?: HopsOpsComAiV1alpha1PSQLBranchSpecTTL + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecBranch: + r""" + Branch cluster sizing. + + Attributes + ---------- + instances : int, default is 1, optional + Branch instance count. Defaults to 1 (HA on branches is unusual). + storage : HopsOpsComAiV1alpha1PSQLBranchSpecBranchStorage, default is Undefined, optional + storage + """ + + + instances?: int = 1 + + storage?: HopsOpsComAiV1alpha1PSQLBranchSpecBranchStorage + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecBranchStorage: + r""" + Branch PVC sizing. Empty fields inherit from the source's storage on recovery. + + Attributes + ---------- + class : str, default is Undefined, optional + Branch StorageClass. Empty = inherit from source. + size : str, default is Undefined, optional + Branch PVC size, e.g. "10Gi". Empty = match source. + """ + + + class?: str = "" + + size?: str = "" + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCnpg: + r""" + Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not covered by the toggles above. + + Attributes + ---------- + overrideAllValues : any, default is Undefined, optional + Replaces the entire Cluster.spec. + values : any, default is Undefined, optional + Merged into the Cluster.spec. + """ + + + overrideAllValues?: any + + values?: any + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplane: + r""" + Configures how Crossplane will reconcile this composite resource + + Attributes + ---------- + compositionRef : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRef, default is Undefined, optional + composition ref + compositionRevisionRef : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionRef, default is Undefined, optional + composition revision ref + compositionRevisionSelector : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionSelector, default is Undefined, optional + composition revision selector + compositionSelector : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionSelector, default is Undefined, optional + composition selector + compositionUpdatePolicy : str, default is Undefined, optional + composition update policy + resourceRefs : [HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneResourceRefsItems0], default is Undefined, optional + resource refs + """ + + + compositionRef?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRef + + compositionRevisionRef?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionRef + + compositionRevisionSelector?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionSelector + + compositionSelector?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionSelector + + compositionUpdatePolicy?: "Automatic" | "Manual" + + resourceRefs?: [HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneResourceRefsItems0] + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRef: + r""" + hops ops com ai v1alpha1 p SQL branch spec crossplane composition ref + + Attributes + ---------- + name : str, default is Undefined, required + name + """ + + + name: str + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionRef: + r""" + hops ops com ai v1alpha1 p SQL branch spec crossplane composition revision ref + + Attributes + ---------- + name : str, default is Undefined, required + name + """ + + + name: str + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionSelector: + r""" + hops ops com ai v1alpha1 p SQL branch spec crossplane composition revision selector + + Attributes + ---------- + matchLabels : {str:str}, default is Undefined, required + match labels + """ + + + matchLabels: {str:str} + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionSelector: + r""" + hops ops com ai v1alpha1 p SQL branch spec crossplane composition selector + + Attributes + ---------- + matchLabels : {str:str}, default is Undefined, required + match labels + """ + + + matchLabels: {str:str} + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneResourceRefsItems0: + r""" + hops ops com ai v1alpha1 p SQL branch spec crossplane resource refs items0 + + Attributes + ---------- + apiVersion : str, default is Undefined, required + api version + kind : str, default is Undefined, required + kind + name : str, default is Undefined, optional + name + """ + + + apiVersion: str + + kind: str + + name?: str + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecKubernetesProviderConfigRef: + r""" + Reference to the Kubernetes ProviderConfig used to apply composed resources. Defaults to clusterName. + + Attributes + ---------- + kind : str, default is Undefined, optional + kind + name : str, default is Undefined, optional + name + """ + + + kind?: "ProviderConfig" | "ClusterProviderConfig" + + name?: str + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecPostgresql: + r""" + Postgres version on the branch (must match source for snapshot recovery to succeed). + + Attributes + ---------- + version : str, default is "17", optional + version + """ + + + version?: str = "17" + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecScaleToZero: + r""" + Auto-hibernate the branch after `idleTimeout` of no client + activity. Default-on with aggressive timeout — preview-PR + branches should hibernate fast to keep cost flat. + + + Attributes + ---------- + enabled : bool, default is True, optional + enabled + idleTimeout : str, default is "10m", optional + Idle duration before hibernate. Defaults to "10m" (more aggressive than PSQLCluster's 30m). + """ + + + enabled?: bool = True + + idleTimeout?: str = "10m" + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecSource: + r""" + Source PSQLCluster to fork. Cross-namespace supported — leave + `namespace` empty for same-namespace branching, or set it to + the team's namespace for preview-PR-style forks. + + + Attributes + ---------- + name : str, default is Undefined, required + Source PSQLCluster name (required). + namespace : str, default is Undefined, optional + Source PSQLCluster namespace. Empty = same as branch. + pvcName : str, default is Undefined, optional + Source PVC name to snapshot. Empty = derive from CNPG + naming convention (`-1` for the primary + instance). + + snapshotClassName : str, default is "psql", optional + VolumeSnapshotClass to use for the snapshot. Defaults to "psql" (composed by psql-stack). + """ + + + name: str + + namespace?: str = "" + + pvcName?: str = "" + + snapshotClassName?: str = "psql" + + +schema HopsOpsComAiV1alpha1PSQLBranchSpecTTL: + r""" + Recorded intent for branch expiration. v0 does NOT enforce + the TTL automatically — pair with an external cleanup + mechanism (Kyverno cleanupPolicies, ArgoCD ApplicationSet + with pullRequest generator that deletes on PR close, etc.). + Status reports `expiresAt` (creation + after) for tooling. + + + Attributes + ---------- + after : str, default is "168h", optional + Duration after which the branch should be deleted. Defaults to "168h" (7 days). + enabled : bool, default is Undefined, optional + enabled + """ + + + after?: str = "168h" + + enabled?: bool = False + + +schema HopsOpsComAiV1alpha1PSQLBranchStatus: + r""" + PSQLBranchStatus defines the observed state. + + Attributes + ---------- + bootstrapMethod : str, default is Undefined, optional + Bootstrap method actually used. v0 always "volumeSnapshot". + bootstrapPhase : str, default is Undefined, optional + Current branch CNPG Cluster `.status.phase`. + conditions : [HopsOpsComAiV1alpha1PSQLBranchStatusConditionsItems0], default is Undefined, optional + Conditions of the resource. + crossplane : HopsOpsComAiV1alpha1PSQLBranchStatusCrossplane, default is Undefined, optional + crossplane + expiresAt : str, default is Undefined, optional + Computed deletion deadline when ttl.enabled is true. + ready : bool, default is Undefined, optional + ready + sourceSnapshotContent : str, default is Undefined, optional + Name of the cluster-scoped VolumeSnapshotContent backing this branch (for cross-ns bridging visibility). + """ + + + bootstrapMethod?: str + + bootstrapPhase?: str + + conditions?: [HopsOpsComAiV1alpha1PSQLBranchStatusConditionsItems0] + + crossplane?: HopsOpsComAiV1alpha1PSQLBranchStatusCrossplane + + expiresAt?: str + + ready?: bool + + sourceSnapshotContent?: str + + +schema HopsOpsComAiV1alpha1PSQLBranchStatusConditionsItems0: + r""" + hops ops com ai v1alpha1 p SQL branch status conditions items0 + + Attributes + ---------- + lastTransitionTime : str, default is Undefined, required + last transition time + message : str, default is Undefined, optional + message + observedGeneration : int, default is Undefined, optional + observed generation + reason : str, default is Undefined, required + reason + status : str, default is Undefined, required + status + $type : str, default is Undefined, required + type + """ + + + lastTransitionTime: str + + message?: str + + observedGeneration?: int + + reason: str + + status: str + + $type: str + + +schema HopsOpsComAiV1alpha1PSQLBranchStatusCrossplane: + r""" + Indicates how Crossplane is reconciling this composite resource + + Attributes + ---------- + connectionDetails : HopsOpsComAiV1alpha1PSQLBranchStatusCrossplaneConnectionDetails, default is Undefined, optional + connection details + """ + + + connectionDetails?: HopsOpsComAiV1alpha1PSQLBranchStatusCrossplaneConnectionDetails + + +schema HopsOpsComAiV1alpha1PSQLBranchStatusCrossplaneConnectionDetails: + r""" + hops ops com ai v1alpha1 p SQL branch status crossplane connection details + + Attributes + ---------- + lastPublishedTime : str, default is Undefined, optional + last published time + """ + + + lastPublishedTime?: str + + diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k new file mode 100644 index 0000000..82fd889 --- /dev/null +++ b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k @@ -0,0 +1,113 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema CompositionTest: + r""" + CompositionTest defines the schema for the CompositionTest custom resource. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "CompositionTest", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV1alpha1CompositionTestSpec, default is Undefined, required + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "CompositionTest" = "CompositionTest" + + metadata?: v1.ObjectMeta + + spec: MetaDevUpboundIoV1alpha1CompositionTestSpec + + +schema MetaDevUpboundIoV1alpha1CompositionTestSpec: + r""" + CompositionTestSpec defines the specification for the CompositionTest custom resource. + + Attributes + ---------- + assertResources : [any], default is Undefined, optional + AssertResources defines assertions to validate resources after test completion. + Optional. + composition : any, default is Undefined, optional + Composition specifies the composition definition inline. + Optional. + compositionPath : str, default is Undefined, optional + Composition specifies the composition definition path. + Optional. + context : {str:any}, default is Undefined, optional + Context specifies context for the Function Pipeline inline as key-value pairs. + Keys are context keys, values are JSON data. + Optional. + extraResources : [any], default is Undefined, optional + ExtraResources specifies additional resources inline. + Optional. + functionCredentialsPath : str, default is Undefined, optional + FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. + Optional. + observedResources : [any], default is Undefined, optional + ObservedResources specifies additional observed resources inline. + Optional. + timeoutSeconds : int, default is 30, required + Timeout for the test in seconds + Required. Default is 30s. + validate : bool, default is Undefined, optional + Validate indicates whether to validate managed resources against schemas. + Optional. + xr : any, default is Undefined, optional + XR specifies the composite resource (XR) inline. + Mutually exclusive with XRPath. At least one of XR or XRPath must be specified. + xrPath : str, default is Undefined, optional + XRPath specifies the composite resource (XR) path. + Mutually exclusive with XR. At least one of XR or XRPath must be specified. + xrd : any, default is Undefined, optional + XRD specifies the XRD definition inline. + Optional. + xrdPath : str, default is Undefined, optional + XRD specifies the XRD definition path. + Optional. + """ + + + assertResources?: [any] + + composition?: any + + compositionPath?: str + + context?: {str:any} + + extraResources?: [any] + + functionCredentialsPath?: str + + observedResources?: [any] + + timeoutSeconds: int = 30 + + validate?: bool + + xr?: any + + xrPath?: str + + xrd?: any + + xrdPath?: str + + + check: + timeoutSeconds >= 1 + + diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k new file mode 100644 index 0000000..bccda70 --- /dev/null +++ b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k @@ -0,0 +1,167 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema E2ETest: + r""" + E2ETest defines the schema for the E2ETest custom resource used for e2e + testing of Crossplane configurations in controlplanes. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "E2ETest", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV1alpha1E2ETestSpec, default is Undefined, required + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "E2ETest" = "E2ETest" + + metadata?: v1.ObjectMeta + + spec: MetaDevUpboundIoV1alpha1E2ETestSpec + + +schema MetaDevUpboundIoV1alpha1E2ETestSpec: + r""" + E2ETestSpec defines the specification for e2e testing of Crossplane + configurations. It orchestrates the complete test lifecycle including setting + up controlplane, applying test resources in the correct order (InitResources + → Configuration → ExtraResources → Manifests), validating conditions, and + handling cleanup. This spec allows you to define e2e tests that verify your + Crossplane compositions, providers, and managed resources work correctly + together in a real controlplane environment. + + Attributes + ---------- + cleanupTimeoutSeconds : int, default is 600, optional + CleanupTimeoutSeconds defines the maximum duration in seconds for cleanup + operations after the test completes. This timeout applies to the deletion + of test resources and any associated managed resources. If not specified, + defaults to 600 seconds (10 minutes). Consider increasing this value for + tests with many resources or complex deletion dependencies. + crossplane : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane, default is Undefined, required + crossplane + defaultConditions : [str], default is Undefined, optional + DefaultConditions specifies the expected conditions that should be met + after the manifests are applied. These are validation checks that verify + the resources are functioning correctly. Each condition is a string + expression that will be evaluated against the deployed resources. Common + conditions include checking resource status for readiness + extraResources : [any], default is Undefined, optional + ExtraResources specifies additional Kubernetes resources that should be + created or updated after the configuration has been successfully applied. + These resources may depend on the primary configuration being in place. + Common use cases include ConfigMaps, Secrets, providerConfigs. Each + resource must be a valid Kubernetes object. + initResources : [any], default is Undefined, optional + InitResources specifies Kubernetes resources that must be created or + updated before the configuration is applied. These are typically + prerequisite resources that the configuration depends on. Common use + cases include ImageConfigs, DeploymentRuntimeConfigs, or any foundational + resources required for the configuration to work. Each resource must be a + valid Kubernetes object. + manifests : [any], default is Undefined, required + Manifests contains the Kubernetes resources that will be applied as part + of this e2e test. These are the primary resources being tested - they + will be created in the controlplane and then validated against the + conditions specified in DefaultConditions. Each manifest must be a valid + Kubernetes object. At least one manifest is required. Examples include + Claims, Composite Resources or any Kubernetes resource you want to test. + skipDelete : bool, default is Undefined, optional + If true, skip resource deletion after test + timeoutSeconds : int, default is Undefined, optional + TimeoutSeconds defines the maximum duration in seconds that the test is + allowed to run before being marked as failed. This includes time for + resource creation, condition checks, and any reconciliation processes. If + not specified, a default timeout will be used. Consider setting higher + values for tests involving complex resources or those requiring multiple + reconciliation cycles. + """ + + + cleanupTimeoutSeconds?: int = 600 + + crossplane: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane + + defaultConditions?: [str] + + extraResources?: [any] + + initResources?: [any] + + manifests: [any] + + skipDelete?: bool + + timeoutSeconds?: int + + + check: + cleanupTimeoutSeconds >= 1 if cleanupTimeoutSeconds not in [None, Undefined] + len(defaultConditions) >= 1 if defaultConditions + len(manifests) >= 1 + timeoutSeconds >= 1 if timeoutSeconds not in [None, Undefined] + + +schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane: + r""" + Crossplane specifies the Crossplane configuration and settings required + for this test. This includes the version of Universal Crossplane to + install, and optional auto-upgrade settings. The configuration defined + here will be used to set up the controlplane before applying the test + manifests. + + Attributes + ---------- + autoUpgrade : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade, default is Undefined, optional + auto upgrade + state : str, default is "Running", optional + State defines the state for crossplane and provider workloads. We support + the following states where 'Running' is the default: + - Running: Starts/Scales up all crossplane and provider workloads in the ControlPlane + - Paused: Pauses/Scales down all crossplane and provider workloads in the ControlPlane + version : str, default is Undefined, optional + Version is the version of Universal Crossplane to install. + """ + + + autoUpgrade?: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade + + state?: "Running" | "Paused" = "Running" + + version?: str + + +schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade: + r""" + AutoUpgrades defines the auto upgrade configuration for Crossplane. + + Attributes + ---------- + channel : str, default is "Stable", optional + Channel defines the upgrade channels for Crossplane. We support the following channels where 'Stable' is the + default: + - None: disables auto-upgrades and keeps the control plane at its current version of Crossplane. + - Patch: automatically upgrades the control plane to the latest supported patch version when it + becomes available while keeping the minor version the same. + - Stable: automatically upgrades the control plane to the latest supported patch release on minor + version N-1, where N is the latest supported minor version. + - Rapid: automatically upgrades the cluster to the latest supported patch release on the latest + supported minor version. + """ + + + channel?: "None" | "Patch" | "Stable" | "Rapid" = "Stable" + + diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k new file mode 100644 index 0000000..f9e8c95 --- /dev/null +++ b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k @@ -0,0 +1,98 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema OperationTest: + r""" + OperationTest defines the schema for the OperationTest custom resource. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "OperationTest", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, required + metadata + spec : MetaDevUpboundIoV1alpha1OperationTestSpec, default is Undefined, required + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "OperationTest" = "OperationTest" + + metadata: v1.ObjectMeta + + spec: MetaDevUpboundIoV1alpha1OperationTestSpec + + +schema MetaDevUpboundIoV1alpha1OperationTestSpec: + r""" + OperationTestSpec defines the specification for the OperationTest custom resource. + + Attributes + ---------- + assertResources : [any], default is Undefined, optional + AssertResources defines assertions to validate resources after test completion. + Optional. + context : {str:any}, default is Undefined, optional + Context specifies context for the Function Pipeline inline as key-value pairs. + Keys are context keys, values are JSON data. + Optional. + functionCredentialsPath : str, default is Undefined, optional + FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. + Optional. + operation : any, default is Undefined, required + Operation specifies the Operation definition inline. + Optional. + operationPath : str, default is Undefined, optional + OperationPath specifies the XRD definition path. + Optional. + requiredResources : [any], default is Undefined, optional + RequiredResources specifies additional required resources inline. + Optional. + requiredResourcesPath : str, default is Undefined, optional + RequiredResourcesPath specifies a path to required resources file. + Optional. + timeoutSeconds : int, default is 30, required + Timeout for the test in seconds + Required. Default is 30s. + watchedResource : any, default is Undefined, optional + WatchedResource specifies additional watched resource inline. + Optional. + watchedResourcePath : str, default is Undefined, optional + WatchedResourcePath specifies a path to watched resource file. + Optional. + """ + + + assertResources?: [any] + + context?: {str:any} + + functionCredentialsPath?: str + + operation: any + + operationPath?: str + + requiredResources?: [any] + + requiredResourcesPath?: str + + timeoutSeconds: int = 30 + + watchedResource?: any + + watchedResourcePath?: str + + + check: + timeoutSeconds >= 1 + + diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k new file mode 100644 index 0000000..334e249 --- /dev/null +++ b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k @@ -0,0 +1,320 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema Project: + r""" + Project defines an Upbound Project, which can be built into a Crossplane + Configuration. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "Project", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV1alpha1ProjectSpec, default is Undefined, optional + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "Project" = "Project" + + metadata?: v1.ObjectMeta + + spec?: MetaDevUpboundIoV1alpha1ProjectSpec + + +schema MetaDevUpboundIoV1alpha1ProjectSpec: + r""" + ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes + resource there is no Status, only Spec. + + Attributes + ---------- + apiDependencies : [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional + APIDependencies are the API dependencies for this project. + NOTE: This is an experimental feature and is subject to change. + architectures : [str], default is Undefined, optional + architectures + crossplane : MetaDevUpboundIoV1alpha1ProjectSpecCrossplane, default is Undefined, optional + crossplane + dependsOn : [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0], default is Undefined, optional + depends on + description : str, default is Undefined, optional + description + imageConfig : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0], default is Undefined, optional + image config + license : str, default is Undefined, optional + license + maintainer : str, default is Undefined, optional + maintainer + paths : MetaDevUpboundIoV1alpha1ProjectSpecPaths, default is Undefined, optional + paths + readme : str, default is Undefined, optional + readme + repository : str, default is Undefined, required + repository + source : str, default is Undefined, optional + source + """ + + + apiDependencies?: [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0] + + architectures?: [str] + + crossplane?: MetaDevUpboundIoV1alpha1ProjectSpecCrossplane + + dependsOn?: [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0] + + description?: str + + imageConfig?: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0] + + license?: str + + maintainer?: str + + paths?: MetaDevUpboundIoV1alpha1ProjectSpecPaths + + readme?: str + + repository: str + + source?: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0: + r""" + APIDependencies defines a reference to an external API dependency. + NOTE: This is an experimental feature and is subject to change. + + Attributes + ---------- + git : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional + git + http : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional + http + k8s : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional + k8s + $type : str, default is Undefined, required + Type defines the type of API dependency. + """ + + + git?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git + + http?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP + + k8s?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s + + $type: "k8s" | "crd" + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git: + r""" + Git defines the git repository source for the API dependency. + + Attributes + ---------- + path : str, default is Undefined, optional + Path is the path within the repository to the API definition. + ref : str, default is Undefined, optional + Ref is the git reference (branch, tag, or commit SHA). + repository : str, default is Undefined, required + Repository is the git repository URL. + """ + + + path?: str + + ref?: str + + repository: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP: + r""" + HTTP defines the HTTP source for the API dependency. + + Attributes + ---------- + url : str, default is Undefined, required + URL is the HTTP/HTTPS URL to fetch the API dependency from. + """ + + + url: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s: + r""" + K8s defines the Kubernetes API version for the dependency. + + Attributes + ---------- + version : str, default is Undefined, required + Version is the Kubernetes API version (e.g., "v1.33.0"). + """ + + + version: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecCrossplane: + r""" + CrossplaneConstraints specifies a packages compatibility with Crossplane versions. + + Attributes + ---------- + version : str, default is Undefined, required + Semantic version constraints of Crossplane that package is compatible with. + """ + + + version: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0: + r""" + Dependency is a dependency on another package. A dependency can be of an + arbitrary API version and kind, but Crossplane expects package dependencies + to behave like a Crossplane package. Specifically it expects to be able to + create the dependency and set its spec.package field to a package OCI + reference. + + Attributes + ---------- + apiVersion : str, default is Undefined, optional + APIVersion of the dependency. + configuration : str, default is Undefined, optional + Configuration is the name of a Configuration package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + function : str, default is Undefined, optional + Function is the name of a Function package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + kind : str, default is Undefined, optional + Kind of the dependency. + package : str, default is Undefined, optional + Package OCI reference of the dependency. Only used when apiVersion and + kind are set. + Must be a fully qualified image name, including the registry, + repository, and tag. For example, "registry.example.com/repo/package:tag". + provider : str, default is Undefined, optional + Provider is the name of a Provider package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion and kind instead. + version : str, default is Undefined, required + Version is the semantic version constraints of the dependency image. + """ + + + apiVersion?: str + + configuration?: str + + function?: str + + kind?: str + + package?: str + + provider?: str + + version: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0: + r""" + ImageConfig defines a set of rules for matching and rewriting images. + + Attributes + ---------- + matchImages : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required + MatchImages is a list of image matching rules that should be satisfied. + rewriteImage : MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required + rewrite image + """ + + + matchImages: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0] + + rewriteImage: MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage + + +schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0: + r""" + ImageMatch defines a rule for matching image. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix that should be matched. + $type : str, default is "Prefix", optional + Type is the type of match. + """ + + + prefix: str + + $type?: "Prefix" = "Prefix" + + +schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage: + r""" + RewriteImage defines how a matched image should be rewritten. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix to use when rewriting the image. + """ + + + prefix: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecPaths: + r""" + ProjectPaths configures the locations of various parts of the project, for + use at build time. + + Attributes + ---------- + apis : str, default is Undefined, optional + APIs is the directory holding the project's apis. If not + specified, it defaults to `apis/`. + examples : str, default is Undefined, optional + Examples is the directory holding the project's examples. If not + specified, it defaults to `examples/`. + functions : str, default is Undefined, optional + Functions is the directory holding the project's functions. If not + specified, it defaults to `functions/`. + tests : str, default is Undefined, optional + Tests is the directory holding the project's tests. If not + specified, it defaults to `tests/`. + """ + + + apis?: str + + examples?: str + + functions?: str + + tests?: str + + diff --git a/tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k b/tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k new file mode 100644 index 0000000..5eb6272 --- /dev/null +++ b/tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k @@ -0,0 +1,325 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema Project: + r""" + Project defines an Upbound Project, which can be built into a Crossplane + Configuration. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v2alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "Project", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV2alpha1ProjectSpec, default is Undefined, optional + spec + """ + + + apiVersion: "meta.dev.upbound.io/v2alpha1" = "meta.dev.upbound.io/v2alpha1" + + kind: "Project" = "Project" + + metadata?: v1.ObjectMeta + + spec?: MetaDevUpboundIoV2alpha1ProjectSpec + + +schema MetaDevUpboundIoV2alpha1ProjectSpec: + r""" + ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes + resource there is no Status, only Spec. + + Attributes + ---------- + apiDependencies : [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional + APIDependencies are the API dependencies for this project. + NOTE: This is an experimental feature and is subject to change. + architectures : [str], default is Undefined, optional + architectures + crossplane : MetaDevUpboundIoV2alpha1ProjectSpecCrossplane, default is Undefined, optional + crossplane + dependsOn : [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0], default is Undefined, optional + depends on + description : str, default is Undefined, optional + description + imageConfig : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0], default is Undefined, optional + image config + license : str, default is Undefined, optional + license + maintainer : str, default is Undefined, optional + maintainer + paths : MetaDevUpboundIoV2alpha1ProjectSpecPaths, default is Undefined, optional + paths + readme : str, default is Undefined, optional + readme + repository : str, default is Undefined, required + repository + source : str, default is Undefined, optional + source + """ + + + apiDependencies?: [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0] + + architectures?: [str] + + crossplane?: MetaDevUpboundIoV2alpha1ProjectSpecCrossplane + + dependsOn?: [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0] + + description?: str + + imageConfig?: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0] + + license?: str + + maintainer?: str + + paths?: MetaDevUpboundIoV2alpha1ProjectSpecPaths + + readme?: str + + repository: str + + source?: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0: + r""" + APIDependencies defines a reference to an external API dependency. + NOTE: This is an experimental feature and is subject to change. + + Attributes + ---------- + git : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional + git + http : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional + http + k8s : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional + k8s + $type : str, default is Undefined, required + Type defines the type of API dependency. + """ + + + git?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git + + http?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP + + k8s?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s + + $type: "k8s" | "crd" + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git: + r""" + Git defines the git repository source for the API dependency. + + Attributes + ---------- + path : str, default is Undefined, optional + Path is the path within the repository to the API definition. + ref : str, default is Undefined, optional + Ref is the git reference (branch, tag, or commit SHA). + repository : str, default is Undefined, required + Repository is the git repository URL. + """ + + + path?: str + + ref?: str + + repository: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP: + r""" + HTTP defines the HTTP source for the API dependency. + + Attributes + ---------- + url : str, default is Undefined, required + URL is the HTTP/HTTPS URL to fetch the API dependency from. + """ + + + url: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s: + r""" + K8s defines the Kubernetes API version for the dependency. + + Attributes + ---------- + version : str, default is Undefined, required + Version is the Kubernetes API version (e.g., "v1.33.0"). + """ + + + version: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecCrossplane: + r""" + CrossplaneConstraints specifies a packages compatibility with Crossplane versions. + + Attributes + ---------- + version : str, default is Undefined, required + Semantic version constraints of Crossplane that package is compatible with. + """ + + + version: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0: + r""" + Dependency is a dependency on another package. A dependency can be of an + arbitrary API version and kind, but Crossplane expects package dependencies + to behave like a Crossplane package. Specifically it expects to be able to + create the dependency and set its spec.package field to a package OCI + reference. + + Attributes + ---------- + apiVersion : str, default is Undefined, optional + APIVersion of the dependency. + configuration : str, default is Undefined, optional + Configuration is the name of a Configuration package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + function : str, default is Undefined, optional + Function is the name of a Function package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + kind : str, default is Undefined, optional + Kind of the dependency. + package : str, default is Undefined, optional + Package OCI reference of the dependency. Only used when apiVersion and + kind are set. + Must be a fully qualified image name, including the registry, + repository, and tag. For example, "registry.example.com/repo/package:tag". + provider : str, default is Undefined, optional + Provider is the name of a Provider package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion and kind instead. + version : str, default is Undefined, required + Version is the semantic version constraints of the dependency image. + """ + + + apiVersion?: str + + configuration?: str + + function?: str + + kind?: str + + package?: str + + provider?: str + + version: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0: + r""" + ImageConfig defines a set of rules for matching and rewriting images. + + Attributes + ---------- + matchImages : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required + MatchImages is a list of image matching rules that should be satisfied. + rewriteImage : MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required + rewrite image + """ + + + matchImages: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0] + + rewriteImage: MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage + + +schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0: + r""" + ImageMatch defines a rule for matching image. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix that should be matched. + $type : str, default is "Prefix", optional + Type is the type of match. + """ + + + prefix: str + + $type?: "Prefix" = "Prefix" + + +schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage: + r""" + RewriteImage defines how a matched image should be rewritten. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix to use when rewriting the image. + """ + + + prefix: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecPaths: + r""" + ProjectPaths configures the locations of various parts of the project, for + use at build time. + + Attributes + ---------- + apis : str, default is Undefined, optional + APIs is the directory holding the project's apis. If not + specified, it defaults to `apis/`. + examples : str, default is Undefined, optional + Examples is the directory holding the project's examples. If not + specified, it defaults to `examples/`. + functions : str, default is Undefined, optional + Functions is the directory holding the project's functions. If not + specified, it defaults to `functions/`. + operations : str, default is Undefined, optional + Operations is the directory holding the project's operations. If not + specified, it defaults to `operations/`. + tests : str, default is Undefined, optional + Tests is the directory holding the project's tests. If not + specified, it defaults to `tests/`. + """ + + + apis?: str + + examples?: str + + functions?: str + + operations?: str + + tests?: str + + diff --git a/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k b/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k new file mode 100644 index 0000000..0705904 --- /dev/null +++ b/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k @@ -0,0 +1,97 @@ +""" +This is the object_meta module in k8s.apimachinery.pkg.apis.meta.v1 package. +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" + + +schema ObjectMeta: + r""" + ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create. + + Attributes + ---------- + annotations : {str:str}, default is Undefined, optional + Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations + clusterName : str, default is Undefined, optional + The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request. + creationTimestamp : str, default is Undefined, optional + CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. + + Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata + deletionGracePeriodSeconds : int, default is Undefined, optional + Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. + deletionTimestamp : str, default is Undefined, optional + DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested. + + Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata + finalizers : [str], default is Undefined, optional + Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. + generateName : str, default is Undefined, optional + GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server. + + If this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). + + Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency + generation : int, default is Undefined, optional + A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. + labels : {str:str}, default is Undefined, optional + Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels + managedFields : [ManagedFieldsEntry], default is Undefined, optional + ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. + name : str, default is Undefined, optional + Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names + namespace : str, default is Undefined, optional + Namespace defines the space within each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty. + + Must be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces + ownerReferences : [OwnerReference], default is Undefined, optional + List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. + resourceVersion : str, default is Undefined, optional + An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources. + + Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + selfLink : str, default is Undefined, optional + SelfLink is a URL representing this object. Populated by the system. Read-only. + + DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release. + uid : str, default is Undefined, optional + UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations. + + Populated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids + """ + + + annotations?: {str:str} + + clusterName?: str + + creationTimestamp?: str + + deletionGracePeriodSeconds?: int + + deletionTimestamp?: str + + finalizers?: [str] + + generateName?: str + + generation?: int + + labels?: {str:str} + + managedFields?: any + + name?: str + + namespace?: str + + ownerReferences?: [OwnerReference] + + resourceVersion?: str + + selfLink?: str + + uid?: str + + diff --git a/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k b/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k new file mode 100644 index 0000000..3854a5c --- /dev/null +++ b/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k @@ -0,0 +1,41 @@ +""" +This is the owner_reference module in k8s.apimachinery.pkg.apis.meta.v1 package. +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" + + +schema OwnerReference: + r""" + OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field. + + Attributes + ---------- + apiVersion : str, default is Undefined, required + API version of the referent. + blockOwnerDeletion : bool, default is Undefined, optional + If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. + controller : bool, default is Undefined, optional + If true, this reference points to the managing controller. + kind : str, default is Undefined, required + Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + name : str, default is Undefined, required + Name of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#names + uid : str, default is Undefined, required + UID of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#uids + """ + + + apiVersion: str + + blockOwnerDeletion?: bool + + controller?: bool + + kind: str + + name: str + + uid: str + + diff --git a/tests/test-branch/model/kcl.mod b/tests/test-branch/model/kcl.mod new file mode 100644 index 0000000..9df41d7 --- /dev/null +++ b/tests/test-branch/model/kcl.mod @@ -0,0 +1,4 @@ +[package] +name = "models" +edition = "v0.11.2" +version = "0.0.1" diff --git a/tests/test-cluster/kcl.mod b/tests/test-cluster/kcl.mod new file mode 100644 index 0000000..e2b40d0 --- /dev/null +++ b/tests/test-cluster/kcl.mod @@ -0,0 +1,6 @@ +[package] +name = "test-render" +version = "0.0.1" + +[dependencies] +models = { path = "./model" } diff --git a/tests/test-cluster/main.k b/tests/test-cluster/main.k new file mode 100644 index 0000000..0e3c8b6 --- /dev/null +++ b/tests/test-cluster/main.k @@ -0,0 +1,357 @@ +import models.ai.com.ops.hops.v1alpha1 as stacksv1alpha1 +import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 + +# ============================================================================== +# Unit tests for PSQLCluster XRD +# +# Philosophy: test YOUR API, not CNPG's. Verify spec fields produce expected +# wiring (intent → mechanics) inside the composed Cluster + ExternalSecret. +# ============================================================================== + +_items = [ + # ========================================================================== + # Test 1: minimal claim renders Cluster + ExternalSecret with sensible + # defaults. Branching labels on by default, monitoring on, no plugins, no + # affinity, no custom storage class. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "minimal-renders-cluster-and-secret" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "test-app", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "10Gi"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-app-cnpg-cluster" + spec.forProvider.manifest = { + apiVersion = "postgresql.cnpg.io/v1" + kind = "Cluster" + metadata = { + name = "test-app" + namespace = "default" + labels = { + "hops.ops.com.ai/managed" = "true" + "hops.ops.com.ai/psql-cluster" = "test-app" + "hops.ops.com.ai/branching-source" = "true" + "hops.ops.com.ai/snapshot-class" = "psql" + } + } + spec = { + instances = 1 + imageName = "ghcr.io/cloudnative-pg/postgresql:17" + bootstrap.initdb = { + database = "app" + owner = "app" + secret.name = "test-app-superuser" + } + storage.size = "10Gi" + monitoring.enablePodMonitor = True + } + } + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-app-external-secret" + spec.forProvider.manifest = { + apiVersion = "external-secrets.io/v1beta1" + kind = "ExternalSecret" + metadata.name = "test-app-superuser" + spec = { + refreshInterval = "1h" + secretStoreRef = { + name = "hops-aws-secrets-manager" + kind = "ClusterSecretStore" + } + target.name = "test-app-superuser" + } + } + } + ] + } + } + + # ========================================================================== + # Test 2: ha.enabled=true bumps instances + adds zonal anti-affinity. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "ha-enabled-bumps-instances-and-affinity" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "ha-app", namespace = "default"} + spec = { + clusterName = "prod" + storage = {size = "20Gi"} + ha = {enabled = True, replicas = 3, topologySpreadByZone = True} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "ha-app-cnpg-cluster" + spec.forProvider.manifest.spec = { + instances = 3 + affinity = { + enablePodAntiAffinity = True + podAntiAffinityType = "preferred" + topologyKey = "topology.kubernetes.io/zone" + } + } + } + ] + } + } + + # ========================================================================== + # Test 3: scaleToZero.enabled=true adds annotation + plugins entry. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "scaletozero-enabled-adds-annotation-and-plugin" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "s2z-app", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi"} + scaleToZero = {enabled = True, idleTimeout = "10m"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "s2z-app-cnpg-cluster" + spec.forProvider.manifest = { + metadata.annotations = { + "cnpg-i-scale-to-zero.xata.io/idle-timeout" = "10m" + } + spec.plugins = [{ + name = "cnpg-i-scale-to-zero.xata.io" + enabled = True + }] + } + } + ] + } + } + + # ========================================================================== + # Test 4: branching.enabled=false strips the branching-source labels. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "branching-disabled-strips-labels" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "no-branch", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi"} + branching = {enabled = False} + } + } + # Stack labels remain; branching-* labels do not. + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "no-branch-cnpg-cluster" + spec.forProvider.manifest.metadata.labels = { + "hops.ops.com.ai/managed" = "true" + "hops.ops.com.ai/psql-cluster" = "no-branch" + } + } + ] + } + } + + # ========================================================================== + # Test 5: storage.class set passes through to spec.storage.storageClass. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "storage-class-passes-through" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "sc-app", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi", class = "gp3"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "sc-app-cnpg-cluster" + spec.forProvider.manifest.spec.storage = { + size = "5Gi" + storageClass = "gp3" + } + } + ] + } + } + + # ========================================================================== + # Test 6: postgresql.parameters pass through. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "postgresql-parameters-pass-through" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "tuned-app", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi"} + postgresql = { + version = "17" + parameters = { + shared_buffers = "256MB" + max_connections = "200" + } + } + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "tuned-app-cnpg-cluster" + spec.forProvider.manifest.spec.postgresql.parameters = { + shared_buffers = "256MB" + max_connections = "200" + } + } + ] + } + } + + # ========================================================================== + # Test 7: credentials.superuser.managedBy="" suppresses ExternalSecret. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "no-managedby-skips-external-secret" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "byo-secret", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi"} + credentials.superuser.managedBy = "" + } + } + # Cluster still composed; ExternalSecret absent. + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "byo-secret-cnpg-cluster" + } + ] + } + } + + # ========================================================================== + # Test 8: cnpg.overrideAllValues replaces the whole Cluster.spec. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "cnpg-override-replaces-cluster-spec" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "override-app", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi"} # required, but overridden by cnpg.overrideAllValues + cnpg.overrideAllValues = { + instances = 5 + imageName = "custom/postgres:99" + storage = {size = "999Gi"} + } + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "override-app-cnpg-cluster" + spec.forProvider.manifest.spec = { + instances = 5 + imageName = "custom/postgres:99" + storage.size = "999Gi" + } + } + ] + } + } + + # ========================================================================== + # Test 9: kubernetesProviderConfigRef defaults to clusterName. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "k8s-provider-config-defaults-to-cluster-name" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "p-app", namespace = "default"} + spec = { + clusterName = "prod-cluster" + storage = {size = "5Gi"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "p-app-cnpg-cluster" + spec.providerConfigRef = { + name = "prod-cluster" + kind = "ProviderConfig" + } + } + ] + } + } +] + +items = _items diff --git a/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k b/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k new file mode 100644 index 0000000..9d113d6 --- /dev/null +++ b/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k @@ -0,0 +1,496 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema PSQLCluster: + r""" + PSQLCluster is the per-app Postgres primitive. Composes a CloudNativePG + `Cluster` plus the wiring (annotations, plugin entries, labels) required + for the platform integrations (scale-to-zero, branching, monitoring, HA). + + Companion to `psql-stack` (which provides the CNPG operator + S2Z plugin + substrate; required as a target-cluster prerequisite) and to `psql-branch` + (which composes ephemeral CoW forks of a PSQLCluster via VolumeSnapshot). + + Intent-first surface: integration toggles (`scaleToZero.enabled`, + `branching.enabled`, `monitoring.enabled`, `ha.enabled`) translate to + CNPG mechanics inside the render template — consumer manifests don't + reach for `cnpg.overrideAllValues` unless their use case is genuinely + novel. + + + Attributes + ---------- + apiVersion : str, default is "hops.ops.com.ai/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "PSQLCluster", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : HopsOpsComAiV1alpha1PSQLClusterSpec, default is Undefined, required + spec + status : HopsOpsComAiV1alpha1PSQLClusterStatus, default is Undefined, optional + status + """ + + + apiVersion: "hops.ops.com.ai/v1alpha1" = "hops.ops.com.ai/v1alpha1" + + kind: "PSQLCluster" = "PSQLCluster" + + metadata?: v1.ObjectMeta + + spec: HopsOpsComAiV1alpha1PSQLClusterSpec + + status?: HopsOpsComAiV1alpha1PSQLClusterStatus + + +schema HopsOpsComAiV1alpha1PSQLClusterSpec: + r""" + PSQLClusterSpec defines the desired state. + + Attributes + ---------- + branching : HopsOpsComAiV1alpha1PSQLClusterSpecBranching, default is Undefined, optional + branching + clusterName : str, default is Undefined, required + Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. + cnpg : HopsOpsComAiV1alpha1PSQLClusterSpecCnpg, default is Undefined, optional + cnpg + credentials : HopsOpsComAiV1alpha1PSQLClusterSpecCredentials, default is Undefined, optional + credentials + crossplane : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane, default is Undefined, optional + crossplane + ha : HopsOpsComAiV1alpha1PSQLClusterSpecHa, default is Undefined, optional + ha + instances : int, default is 1, optional + Number of Postgres instances. Default 1 (single-node). HA mode bumps to ha.replicas. + kubernetesProviderConfigRef : HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef, default is Undefined, optional + kubernetes provider config ref + labels : {str:str}, default is Undefined, optional + Custom labels merged with stack defaults; applied to every composed resource. + managementPolicies : [str], default is ["*"], optional + Crossplane managementPolicies applied to every composed resource. Defaults to ["*"]. + monitoring : HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring, default is Undefined, optional + monitoring + postgresql : HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql, default is Undefined, optional + postgresql + scaleToZero : HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero, default is Undefined, optional + scale to zero + storage : HopsOpsComAiV1alpha1PSQLClusterSpecStorage, default is Undefined, required + storage + """ + + + branching?: HopsOpsComAiV1alpha1PSQLClusterSpecBranching + + clusterName: str + + cnpg?: HopsOpsComAiV1alpha1PSQLClusterSpecCnpg + + credentials?: HopsOpsComAiV1alpha1PSQLClusterSpecCredentials + + crossplane?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane + + ha?: HopsOpsComAiV1alpha1PSQLClusterSpecHa + + instances?: int = 1 + + kubernetesProviderConfigRef?: HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef + + labels?: {str:str} + + managementPolicies?: [str] = ["*"] + + monitoring?: HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring + + postgresql?: HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql + + scaleToZero?: HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero + + storage: HopsOpsComAiV1alpha1PSQLClusterSpecStorage + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecBranching: + r""" + Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). + + Attributes + ---------- + enabled : bool, default is True, optional + enabled + snapshotClassName : str, default is "psql", optional + VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. + """ + + + enabled?: bool = True + + snapshotClassName?: str = "psql" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCnpg: + r""" + Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. + + Attributes + ---------- + overrideAllValues : any, default is Undefined, optional + Replaces the entire Cluster.spec — total escape hatch. + values : any, default is Undefined, optional + Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. + """ + + + overrideAllValues?: any + + values?: any + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCredentials: + r""" + Where Postgres credentials are sourced from. + + Attributes + ---------- + superuser : HopsOpsComAiV1alpha1PSQLClusterSpecCredentialsSuperuser, default is Undefined, optional + superuser + """ + + + superuser?: HopsOpsComAiV1alpha1PSQLClusterSpecCredentialsSuperuser + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCredentialsSuperuser: + r""" + hops ops com ai v1alpha1 p SQL cluster spec credentials superuser + + Attributes + ---------- + managedBy : str, default is "hops-aws-secrets-manager", optional + ClusterSecretStore name for sourcing the superuser password from a secrets backend (default ESO ClusterSecretStore convention). Set "" to skip ExternalSecret composition (consumer must provide a K8s Secret named `` with `username` + `password` keys). + remoteKey : str, default is Undefined, optional + Key in the remote secrets backend (AWS Secrets Manager) holding the superuser password. Defaults to "/superuser". + secretName : str, default is Undefined, optional + K8s Secret name receiving the credentials. Defaults to "-superuser". + """ + + + managedBy?: str = "hops-aws-secrets-manager" + + remoteKey?: str = "" + + secretName?: str = "" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane: + r""" + Configures how Crossplane will reconcile this composite resource + + Attributes + ---------- + compositionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef, default is Undefined, optional + composition ref + compositionRevisionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef, default is Undefined, optional + composition revision ref + compositionRevisionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector, default is Undefined, optional + composition revision selector + compositionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector, default is Undefined, optional + composition selector + compositionUpdatePolicy : str, default is Undefined, optional + composition update policy + resourceRefs : [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0], default is Undefined, optional + resource refs + """ + + + compositionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef + + compositionRevisionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef + + compositionRevisionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector + + compositionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector + + compositionUpdatePolicy?: "Automatic" | "Manual" + + resourceRefs?: [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0] + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition ref + + Attributes + ---------- + name : str, default is Undefined, required + name + """ + + + name: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision ref + + Attributes + ---------- + name : str, default is Undefined, required + name + """ + + + name: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision selector + + Attributes + ---------- + matchLabels : {str:str}, default is Undefined, required + match labels + """ + + + matchLabels: {str:str} + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition selector + + Attributes + ---------- + matchLabels : {str:str}, default is Undefined, required + match labels + """ + + + matchLabels: {str:str} + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane resource refs items0 + + Attributes + ---------- + apiVersion : str, default is Undefined, required + api version + kind : str, default is Undefined, required + kind + name : str, default is Undefined, optional + name + """ + + + apiVersion: str + + kind: str + + name?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecHa: + r""" + Stack-wide HA. Bumps instances to `replicas` (default 3) and adds zonal topology spread. + + Attributes + ---------- + enabled : bool, default is Undefined, optional + enabled + replicas : int, default is 3, optional + replicas + topologySpreadByZone : bool, default is True, optional + topology spread by zone + """ + + + enabled?: bool = False + + replicas?: int = 3 + + topologySpreadByZone?: bool = True + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef: + r""" + Reference to the Kubernetes ProviderConfig used to apply the Cluster CR + ExternalSecret. Defaults to clusterName. + + Attributes + ---------- + kind : str, default is Undefined, optional + kind + name : str, default is Undefined, optional + name + """ + + + kind?: "ProviderConfig" | "ClusterProviderConfig" + + name?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring: + r""" + Add Prometheus scrape configuration. CNPG operator handles the actual PodMonitor creation when `monitoring.enablePodMonitor` is set on the Cluster CR. + + Attributes + ---------- + enabled : bool, default is True, optional + enabled + """ + + + enabled?: bool = True + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql: + r""" + Postgres version + tuning parameters. + + Attributes + ---------- + parameters : {str:str}, default is Undefined, optional + postgres.conf parameters (e.g. `shared_buffers`, `max_connections`). + version : str, default is "17", optional + Postgres major version. Defaults to "17". + """ + + + parameters?: {str:str} + + version?: str = "17" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero: + r""" + Auto-hibernate the cluster after `idleTimeout` of no client activity. Requires the cnpg-i-scale-to-zero plugin on the target cluster (provided by psql-stack). When `ha.enabled` is also true, the cluster will not actually scale to zero (HA implies always-available); a Warning condition is emitted on the XR but the annotation is rendered for tooling that reads it. + + Attributes + ---------- + enabled : bool, default is Undefined, optional + enabled + idleTimeout : str, default is "30m", optional + How long the cluster must be idle before hibernating. Defaults to "30m". + """ + + + enabled?: bool = False + + idleTimeout?: str = "30m" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecStorage: + r""" + Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (gp3 default on EKS Auto Mode) — bumping `size` upward grows the volume in place with no downtime. + + Attributes + ---------- + class : str, default is Undefined, optional + StorageClass name. Empty = cluster default (gp3 on Auto Mode). + size : str, default is Undefined, required + PVC size, e.g. "10Gi". Required. + """ + + + class?: str = "" + + size: str + + +schema HopsOpsComAiV1alpha1PSQLClusterStatus: + r""" + PSQLClusterStatus defines the observed state. + + Attributes + ---------- + clusterPhase : str, default is Undefined, optional + Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). + conditions : [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0], default is Undefined, optional + Conditions of the resource. + crossplane : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane, default is Undefined, optional + crossplane + ready : bool, default is Undefined, optional + Overall readiness — true once the CNPG Cluster reaches `Cluster in healthy state` AND the ExternalSecret (if composed) is SecretSynced. + """ + + + clusterPhase?: str + + conditions?: [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0] + + crossplane?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane + + ready?: bool + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0: + r""" + hops ops com ai v1alpha1 p SQL cluster status conditions items0 + + Attributes + ---------- + lastTransitionTime : str, default is Undefined, required + last transition time + message : str, default is Undefined, optional + message + observedGeneration : int, default is Undefined, optional + observed generation + reason : str, default is Undefined, required + reason + status : str, default is Undefined, required + status + $type : str, default is Undefined, required + type + """ + + + lastTransitionTime: str + + message?: str + + observedGeneration?: int + + reason: str + + status: str + + $type: str + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane: + r""" + Indicates how Crossplane is reconciling this composite resource + + Attributes + ---------- + connectionDetails : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails, default is Undefined, optional + connection details + """ + + + connectionDetails?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails: + r""" + hops ops com ai v1alpha1 p SQL cluster status crossplane connection details + + Attributes + ---------- + lastPublishedTime : str, default is Undefined, optional + last published time + """ + + + lastPublishedTime?: str + + diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k new file mode 100644 index 0000000..82fd889 --- /dev/null +++ b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k @@ -0,0 +1,113 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema CompositionTest: + r""" + CompositionTest defines the schema for the CompositionTest custom resource. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "CompositionTest", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV1alpha1CompositionTestSpec, default is Undefined, required + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "CompositionTest" = "CompositionTest" + + metadata?: v1.ObjectMeta + + spec: MetaDevUpboundIoV1alpha1CompositionTestSpec + + +schema MetaDevUpboundIoV1alpha1CompositionTestSpec: + r""" + CompositionTestSpec defines the specification for the CompositionTest custom resource. + + Attributes + ---------- + assertResources : [any], default is Undefined, optional + AssertResources defines assertions to validate resources after test completion. + Optional. + composition : any, default is Undefined, optional + Composition specifies the composition definition inline. + Optional. + compositionPath : str, default is Undefined, optional + Composition specifies the composition definition path. + Optional. + context : {str:any}, default is Undefined, optional + Context specifies context for the Function Pipeline inline as key-value pairs. + Keys are context keys, values are JSON data. + Optional. + extraResources : [any], default is Undefined, optional + ExtraResources specifies additional resources inline. + Optional. + functionCredentialsPath : str, default is Undefined, optional + FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. + Optional. + observedResources : [any], default is Undefined, optional + ObservedResources specifies additional observed resources inline. + Optional. + timeoutSeconds : int, default is 30, required + Timeout for the test in seconds + Required. Default is 30s. + validate : bool, default is Undefined, optional + Validate indicates whether to validate managed resources against schemas. + Optional. + xr : any, default is Undefined, optional + XR specifies the composite resource (XR) inline. + Mutually exclusive with XRPath. At least one of XR or XRPath must be specified. + xrPath : str, default is Undefined, optional + XRPath specifies the composite resource (XR) path. + Mutually exclusive with XR. At least one of XR or XRPath must be specified. + xrd : any, default is Undefined, optional + XRD specifies the XRD definition inline. + Optional. + xrdPath : str, default is Undefined, optional + XRD specifies the XRD definition path. + Optional. + """ + + + assertResources?: [any] + + composition?: any + + compositionPath?: str + + context?: {str:any} + + extraResources?: [any] + + functionCredentialsPath?: str + + observedResources?: [any] + + timeoutSeconds: int = 30 + + validate?: bool + + xr?: any + + xrPath?: str + + xrd?: any + + xrdPath?: str + + + check: + timeoutSeconds >= 1 + + diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k new file mode 100644 index 0000000..bccda70 --- /dev/null +++ b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k @@ -0,0 +1,167 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema E2ETest: + r""" + E2ETest defines the schema for the E2ETest custom resource used for e2e + testing of Crossplane configurations in controlplanes. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "E2ETest", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV1alpha1E2ETestSpec, default is Undefined, required + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "E2ETest" = "E2ETest" + + metadata?: v1.ObjectMeta + + spec: MetaDevUpboundIoV1alpha1E2ETestSpec + + +schema MetaDevUpboundIoV1alpha1E2ETestSpec: + r""" + E2ETestSpec defines the specification for e2e testing of Crossplane + configurations. It orchestrates the complete test lifecycle including setting + up controlplane, applying test resources in the correct order (InitResources + → Configuration → ExtraResources → Manifests), validating conditions, and + handling cleanup. This spec allows you to define e2e tests that verify your + Crossplane compositions, providers, and managed resources work correctly + together in a real controlplane environment. + + Attributes + ---------- + cleanupTimeoutSeconds : int, default is 600, optional + CleanupTimeoutSeconds defines the maximum duration in seconds for cleanup + operations after the test completes. This timeout applies to the deletion + of test resources and any associated managed resources. If not specified, + defaults to 600 seconds (10 minutes). Consider increasing this value for + tests with many resources or complex deletion dependencies. + crossplane : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane, default is Undefined, required + crossplane + defaultConditions : [str], default is Undefined, optional + DefaultConditions specifies the expected conditions that should be met + after the manifests are applied. These are validation checks that verify + the resources are functioning correctly. Each condition is a string + expression that will be evaluated against the deployed resources. Common + conditions include checking resource status for readiness + extraResources : [any], default is Undefined, optional + ExtraResources specifies additional Kubernetes resources that should be + created or updated after the configuration has been successfully applied. + These resources may depend on the primary configuration being in place. + Common use cases include ConfigMaps, Secrets, providerConfigs. Each + resource must be a valid Kubernetes object. + initResources : [any], default is Undefined, optional + InitResources specifies Kubernetes resources that must be created or + updated before the configuration is applied. These are typically + prerequisite resources that the configuration depends on. Common use + cases include ImageConfigs, DeploymentRuntimeConfigs, or any foundational + resources required for the configuration to work. Each resource must be a + valid Kubernetes object. + manifests : [any], default is Undefined, required + Manifests contains the Kubernetes resources that will be applied as part + of this e2e test. These are the primary resources being tested - they + will be created in the controlplane and then validated against the + conditions specified in DefaultConditions. Each manifest must be a valid + Kubernetes object. At least one manifest is required. Examples include + Claims, Composite Resources or any Kubernetes resource you want to test. + skipDelete : bool, default is Undefined, optional + If true, skip resource deletion after test + timeoutSeconds : int, default is Undefined, optional + TimeoutSeconds defines the maximum duration in seconds that the test is + allowed to run before being marked as failed. This includes time for + resource creation, condition checks, and any reconciliation processes. If + not specified, a default timeout will be used. Consider setting higher + values for tests involving complex resources or those requiring multiple + reconciliation cycles. + """ + + + cleanupTimeoutSeconds?: int = 600 + + crossplane: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane + + defaultConditions?: [str] + + extraResources?: [any] + + initResources?: [any] + + manifests: [any] + + skipDelete?: bool + + timeoutSeconds?: int + + + check: + cleanupTimeoutSeconds >= 1 if cleanupTimeoutSeconds not in [None, Undefined] + len(defaultConditions) >= 1 if defaultConditions + len(manifests) >= 1 + timeoutSeconds >= 1 if timeoutSeconds not in [None, Undefined] + + +schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane: + r""" + Crossplane specifies the Crossplane configuration and settings required + for this test. This includes the version of Universal Crossplane to + install, and optional auto-upgrade settings. The configuration defined + here will be used to set up the controlplane before applying the test + manifests. + + Attributes + ---------- + autoUpgrade : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade, default is Undefined, optional + auto upgrade + state : str, default is "Running", optional + State defines the state for crossplane and provider workloads. We support + the following states where 'Running' is the default: + - Running: Starts/Scales up all crossplane and provider workloads in the ControlPlane + - Paused: Pauses/Scales down all crossplane and provider workloads in the ControlPlane + version : str, default is Undefined, optional + Version is the version of Universal Crossplane to install. + """ + + + autoUpgrade?: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade + + state?: "Running" | "Paused" = "Running" + + version?: str + + +schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade: + r""" + AutoUpgrades defines the auto upgrade configuration for Crossplane. + + Attributes + ---------- + channel : str, default is "Stable", optional + Channel defines the upgrade channels for Crossplane. We support the following channels where 'Stable' is the + default: + - None: disables auto-upgrades and keeps the control plane at its current version of Crossplane. + - Patch: automatically upgrades the control plane to the latest supported patch version when it + becomes available while keeping the minor version the same. + - Stable: automatically upgrades the control plane to the latest supported patch release on minor + version N-1, where N is the latest supported minor version. + - Rapid: automatically upgrades the cluster to the latest supported patch release on the latest + supported minor version. + """ + + + channel?: "None" | "Patch" | "Stable" | "Rapid" = "Stable" + + diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k new file mode 100644 index 0000000..f9e8c95 --- /dev/null +++ b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k @@ -0,0 +1,98 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema OperationTest: + r""" + OperationTest defines the schema for the OperationTest custom resource. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "OperationTest", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, required + metadata + spec : MetaDevUpboundIoV1alpha1OperationTestSpec, default is Undefined, required + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "OperationTest" = "OperationTest" + + metadata: v1.ObjectMeta + + spec: MetaDevUpboundIoV1alpha1OperationTestSpec + + +schema MetaDevUpboundIoV1alpha1OperationTestSpec: + r""" + OperationTestSpec defines the specification for the OperationTest custom resource. + + Attributes + ---------- + assertResources : [any], default is Undefined, optional + AssertResources defines assertions to validate resources after test completion. + Optional. + context : {str:any}, default is Undefined, optional + Context specifies context for the Function Pipeline inline as key-value pairs. + Keys are context keys, values are JSON data. + Optional. + functionCredentialsPath : str, default is Undefined, optional + FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. + Optional. + operation : any, default is Undefined, required + Operation specifies the Operation definition inline. + Optional. + operationPath : str, default is Undefined, optional + OperationPath specifies the XRD definition path. + Optional. + requiredResources : [any], default is Undefined, optional + RequiredResources specifies additional required resources inline. + Optional. + requiredResourcesPath : str, default is Undefined, optional + RequiredResourcesPath specifies a path to required resources file. + Optional. + timeoutSeconds : int, default is 30, required + Timeout for the test in seconds + Required. Default is 30s. + watchedResource : any, default is Undefined, optional + WatchedResource specifies additional watched resource inline. + Optional. + watchedResourcePath : str, default is Undefined, optional + WatchedResourcePath specifies a path to watched resource file. + Optional. + """ + + + assertResources?: [any] + + context?: {str:any} + + functionCredentialsPath?: str + + operation: any + + operationPath?: str + + requiredResources?: [any] + + requiredResourcesPath?: str + + timeoutSeconds: int = 30 + + watchedResource?: any + + watchedResourcePath?: str + + + check: + timeoutSeconds >= 1 + + diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k new file mode 100644 index 0000000..334e249 --- /dev/null +++ b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k @@ -0,0 +1,320 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema Project: + r""" + Project defines an Upbound Project, which can be built into a Crossplane + Configuration. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "Project", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV1alpha1ProjectSpec, default is Undefined, optional + spec + """ + + + apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" + + kind: "Project" = "Project" + + metadata?: v1.ObjectMeta + + spec?: MetaDevUpboundIoV1alpha1ProjectSpec + + +schema MetaDevUpboundIoV1alpha1ProjectSpec: + r""" + ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes + resource there is no Status, only Spec. + + Attributes + ---------- + apiDependencies : [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional + APIDependencies are the API dependencies for this project. + NOTE: This is an experimental feature and is subject to change. + architectures : [str], default is Undefined, optional + architectures + crossplane : MetaDevUpboundIoV1alpha1ProjectSpecCrossplane, default is Undefined, optional + crossplane + dependsOn : [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0], default is Undefined, optional + depends on + description : str, default is Undefined, optional + description + imageConfig : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0], default is Undefined, optional + image config + license : str, default is Undefined, optional + license + maintainer : str, default is Undefined, optional + maintainer + paths : MetaDevUpboundIoV1alpha1ProjectSpecPaths, default is Undefined, optional + paths + readme : str, default is Undefined, optional + readme + repository : str, default is Undefined, required + repository + source : str, default is Undefined, optional + source + """ + + + apiDependencies?: [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0] + + architectures?: [str] + + crossplane?: MetaDevUpboundIoV1alpha1ProjectSpecCrossplane + + dependsOn?: [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0] + + description?: str + + imageConfig?: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0] + + license?: str + + maintainer?: str + + paths?: MetaDevUpboundIoV1alpha1ProjectSpecPaths + + readme?: str + + repository: str + + source?: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0: + r""" + APIDependencies defines a reference to an external API dependency. + NOTE: This is an experimental feature and is subject to change. + + Attributes + ---------- + git : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional + git + http : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional + http + k8s : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional + k8s + $type : str, default is Undefined, required + Type defines the type of API dependency. + """ + + + git?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git + + http?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP + + k8s?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s + + $type: "k8s" | "crd" + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git: + r""" + Git defines the git repository source for the API dependency. + + Attributes + ---------- + path : str, default is Undefined, optional + Path is the path within the repository to the API definition. + ref : str, default is Undefined, optional + Ref is the git reference (branch, tag, or commit SHA). + repository : str, default is Undefined, required + Repository is the git repository URL. + """ + + + path?: str + + ref?: str + + repository: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP: + r""" + HTTP defines the HTTP source for the API dependency. + + Attributes + ---------- + url : str, default is Undefined, required + URL is the HTTP/HTTPS URL to fetch the API dependency from. + """ + + + url: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s: + r""" + K8s defines the Kubernetes API version for the dependency. + + Attributes + ---------- + version : str, default is Undefined, required + Version is the Kubernetes API version (e.g., "v1.33.0"). + """ + + + version: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecCrossplane: + r""" + CrossplaneConstraints specifies a packages compatibility with Crossplane versions. + + Attributes + ---------- + version : str, default is Undefined, required + Semantic version constraints of Crossplane that package is compatible with. + """ + + + version: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0: + r""" + Dependency is a dependency on another package. A dependency can be of an + arbitrary API version and kind, but Crossplane expects package dependencies + to behave like a Crossplane package. Specifically it expects to be able to + create the dependency and set its spec.package field to a package OCI + reference. + + Attributes + ---------- + apiVersion : str, default is Undefined, optional + APIVersion of the dependency. + configuration : str, default is Undefined, optional + Configuration is the name of a Configuration package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + function : str, default is Undefined, optional + Function is the name of a Function package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + kind : str, default is Undefined, optional + Kind of the dependency. + package : str, default is Undefined, optional + Package OCI reference of the dependency. Only used when apiVersion and + kind are set. + Must be a fully qualified image name, including the registry, + repository, and tag. For example, "registry.example.com/repo/package:tag". + provider : str, default is Undefined, optional + Provider is the name of a Provider package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion and kind instead. + version : str, default is Undefined, required + Version is the semantic version constraints of the dependency image. + """ + + + apiVersion?: str + + configuration?: str + + function?: str + + kind?: str + + package?: str + + provider?: str + + version: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0: + r""" + ImageConfig defines a set of rules for matching and rewriting images. + + Attributes + ---------- + matchImages : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required + MatchImages is a list of image matching rules that should be satisfied. + rewriteImage : MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required + rewrite image + """ + + + matchImages: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0] + + rewriteImage: MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage + + +schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0: + r""" + ImageMatch defines a rule for matching image. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix that should be matched. + $type : str, default is "Prefix", optional + Type is the type of match. + """ + + + prefix: str + + $type?: "Prefix" = "Prefix" + + +schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage: + r""" + RewriteImage defines how a matched image should be rewritten. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix to use when rewriting the image. + """ + + + prefix: str + + +schema MetaDevUpboundIoV1alpha1ProjectSpecPaths: + r""" + ProjectPaths configures the locations of various parts of the project, for + use at build time. + + Attributes + ---------- + apis : str, default is Undefined, optional + APIs is the directory holding the project's apis. If not + specified, it defaults to `apis/`. + examples : str, default is Undefined, optional + Examples is the directory holding the project's examples. If not + specified, it defaults to `examples/`. + functions : str, default is Undefined, optional + Functions is the directory holding the project's functions. If not + specified, it defaults to `functions/`. + tests : str, default is Undefined, optional + Tests is the directory holding the project's tests. If not + specified, it defaults to `tests/`. + """ + + + apis?: str + + examples?: str + + functions?: str + + tests?: str + + diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k b/tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k new file mode 100644 index 0000000..5eb6272 --- /dev/null +++ b/tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k @@ -0,0 +1,325 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema Project: + r""" + Project defines an Upbound Project, which can be built into a Crossplane + Configuration. + + Attributes + ---------- + apiVersion : str, default is "meta.dev.upbound.io/v2alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "Project", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : MetaDevUpboundIoV2alpha1ProjectSpec, default is Undefined, optional + spec + """ + + + apiVersion: "meta.dev.upbound.io/v2alpha1" = "meta.dev.upbound.io/v2alpha1" + + kind: "Project" = "Project" + + metadata?: v1.ObjectMeta + + spec?: MetaDevUpboundIoV2alpha1ProjectSpec + + +schema MetaDevUpboundIoV2alpha1ProjectSpec: + r""" + ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes + resource there is no Status, only Spec. + + Attributes + ---------- + apiDependencies : [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional + APIDependencies are the API dependencies for this project. + NOTE: This is an experimental feature and is subject to change. + architectures : [str], default is Undefined, optional + architectures + crossplane : MetaDevUpboundIoV2alpha1ProjectSpecCrossplane, default is Undefined, optional + crossplane + dependsOn : [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0], default is Undefined, optional + depends on + description : str, default is Undefined, optional + description + imageConfig : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0], default is Undefined, optional + image config + license : str, default is Undefined, optional + license + maintainer : str, default is Undefined, optional + maintainer + paths : MetaDevUpboundIoV2alpha1ProjectSpecPaths, default is Undefined, optional + paths + readme : str, default is Undefined, optional + readme + repository : str, default is Undefined, required + repository + source : str, default is Undefined, optional + source + """ + + + apiDependencies?: [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0] + + architectures?: [str] + + crossplane?: MetaDevUpboundIoV2alpha1ProjectSpecCrossplane + + dependsOn?: [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0] + + description?: str + + imageConfig?: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0] + + license?: str + + maintainer?: str + + paths?: MetaDevUpboundIoV2alpha1ProjectSpecPaths + + readme?: str + + repository: str + + source?: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0: + r""" + APIDependencies defines a reference to an external API dependency. + NOTE: This is an experimental feature and is subject to change. + + Attributes + ---------- + git : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional + git + http : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional + http + k8s : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional + k8s + $type : str, default is Undefined, required + Type defines the type of API dependency. + """ + + + git?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git + + http?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP + + k8s?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s + + $type: "k8s" | "crd" + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git: + r""" + Git defines the git repository source for the API dependency. + + Attributes + ---------- + path : str, default is Undefined, optional + Path is the path within the repository to the API definition. + ref : str, default is Undefined, optional + Ref is the git reference (branch, tag, or commit SHA). + repository : str, default is Undefined, required + Repository is the git repository URL. + """ + + + path?: str + + ref?: str + + repository: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP: + r""" + HTTP defines the HTTP source for the API dependency. + + Attributes + ---------- + url : str, default is Undefined, required + URL is the HTTP/HTTPS URL to fetch the API dependency from. + """ + + + url: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s: + r""" + K8s defines the Kubernetes API version for the dependency. + + Attributes + ---------- + version : str, default is Undefined, required + Version is the Kubernetes API version (e.g., "v1.33.0"). + """ + + + version: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecCrossplane: + r""" + CrossplaneConstraints specifies a packages compatibility with Crossplane versions. + + Attributes + ---------- + version : str, default is Undefined, required + Semantic version constraints of Crossplane that package is compatible with. + """ + + + version: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0: + r""" + Dependency is a dependency on another package. A dependency can be of an + arbitrary API version and kind, but Crossplane expects package dependencies + to behave like a Crossplane package. Specifically it expects to be able to + create the dependency and set its spec.package field to a package OCI + reference. + + Attributes + ---------- + apiVersion : str, default is Undefined, optional + APIVersion of the dependency. + configuration : str, default is Undefined, optional + Configuration is the name of a Configuration package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + function : str, default is Undefined, optional + Function is the name of a Function package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion, kind, and package instead. + kind : str, default is Undefined, optional + Kind of the dependency. + package : str, default is Undefined, optional + Package OCI reference of the dependency. Only used when apiVersion and + kind are set. + Must be a fully qualified image name, including the registry, + repository, and tag. For example, "registry.example.com/repo/package:tag". + provider : str, default is Undefined, optional + Provider is the name of a Provider package image. + Must be a fully qualified image name, including the registry, + + Deprecated: Specify an apiVersion and kind instead. + version : str, default is Undefined, required + Version is the semantic version constraints of the dependency image. + """ + + + apiVersion?: str + + configuration?: str + + function?: str + + kind?: str + + package?: str + + provider?: str + + version: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0: + r""" + ImageConfig defines a set of rules for matching and rewriting images. + + Attributes + ---------- + matchImages : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required + MatchImages is a list of image matching rules that should be satisfied. + rewriteImage : MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required + rewrite image + """ + + + matchImages: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0] + + rewriteImage: MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage + + +schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0: + r""" + ImageMatch defines a rule for matching image. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix that should be matched. + $type : str, default is "Prefix", optional + Type is the type of match. + """ + + + prefix: str + + $type?: "Prefix" = "Prefix" + + +schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage: + r""" + RewriteImage defines how a matched image should be rewritten. + + Attributes + ---------- + prefix : str, default is Undefined, required + Prefix is the prefix to use when rewriting the image. + """ + + + prefix: str + + +schema MetaDevUpboundIoV2alpha1ProjectSpecPaths: + r""" + ProjectPaths configures the locations of various parts of the project, for + use at build time. + + Attributes + ---------- + apis : str, default is Undefined, optional + APIs is the directory holding the project's apis. If not + specified, it defaults to `apis/`. + examples : str, default is Undefined, optional + Examples is the directory holding the project's examples. If not + specified, it defaults to `examples/`. + functions : str, default is Undefined, optional + Functions is the directory holding the project's functions. If not + specified, it defaults to `functions/`. + operations : str, default is Undefined, optional + Operations is the directory holding the project's operations. If not + specified, it defaults to `operations/`. + tests : str, default is Undefined, optional + Tests is the directory holding the project's tests. If not + specified, it defaults to `tests/`. + """ + + + apis?: str + + examples?: str + + functions?: str + + operations?: str + + tests?: str + + diff --git a/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k b/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k new file mode 100644 index 0000000..0705904 --- /dev/null +++ b/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k @@ -0,0 +1,97 @@ +""" +This is the object_meta module in k8s.apimachinery.pkg.apis.meta.v1 package. +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" + + +schema ObjectMeta: + r""" + ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create. + + Attributes + ---------- + annotations : {str:str}, default is Undefined, optional + Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations + clusterName : str, default is Undefined, optional + The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request. + creationTimestamp : str, default is Undefined, optional + CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. + + Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata + deletionGracePeriodSeconds : int, default is Undefined, optional + Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. + deletionTimestamp : str, default is Undefined, optional + DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested. + + Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata + finalizers : [str], default is Undefined, optional + Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. + generateName : str, default is Undefined, optional + GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server. + + If this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). + + Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency + generation : int, default is Undefined, optional + A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. + labels : {str:str}, default is Undefined, optional + Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels + managedFields : [ManagedFieldsEntry], default is Undefined, optional + ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. + name : str, default is Undefined, optional + Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names + namespace : str, default is Undefined, optional + Namespace defines the space within each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty. + + Must be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces + ownerReferences : [OwnerReference], default is Undefined, optional + List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. + resourceVersion : str, default is Undefined, optional + An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources. + + Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + selfLink : str, default is Undefined, optional + SelfLink is a URL representing this object. Populated by the system. Read-only. + + DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release. + uid : str, default is Undefined, optional + UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations. + + Populated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids + """ + + + annotations?: {str:str} + + clusterName?: str + + creationTimestamp?: str + + deletionGracePeriodSeconds?: int + + deletionTimestamp?: str + + finalizers?: [str] + + generateName?: str + + generation?: int + + labels?: {str:str} + + managedFields?: any + + name?: str + + namespace?: str + + ownerReferences?: [OwnerReference] + + resourceVersion?: str + + selfLink?: str + + uid?: str + + diff --git a/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k b/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k new file mode 100644 index 0000000..3854a5c --- /dev/null +++ b/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k @@ -0,0 +1,41 @@ +""" +This is the owner_reference module in k8s.apimachinery.pkg.apis.meta.v1 package. +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" + + +schema OwnerReference: + r""" + OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field. + + Attributes + ---------- + apiVersion : str, default is Undefined, required + API version of the referent. + blockOwnerDeletion : bool, default is Undefined, optional + If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. + controller : bool, default is Undefined, optional + If true, this reference points to the managing controller. + kind : str, default is Undefined, required + Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + name : str, default is Undefined, required + Name of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#names + uid : str, default is Undefined, required + UID of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#uids + """ + + + apiVersion: str + + blockOwnerDeletion?: bool + + controller?: bool + + kind: str + + name: str + + uid: str + + diff --git a/tests/test-cluster/model/kcl.mod b/tests/test-cluster/model/kcl.mod new file mode 100644 index 0000000..9df41d7 --- /dev/null +++ b/tests/test-cluster/model/kcl.mod @@ -0,0 +1,4 @@ +[package] +name = "models" +edition = "v0.11.2" +version = "0.0.1" diff --git a/tests/test-stack/kcl.mod b/tests/test-stack/kcl.mod new file mode 100644 index 0000000..e2b40d0 --- /dev/null +++ b/tests/test-stack/kcl.mod @@ -0,0 +1,6 @@ +[package] +name = "test-render" +version = "0.0.1" + +[dependencies] +models = { path = "./model" } diff --git a/tests/test-render/main.k b/tests/test-stack/main.k similarity index 100% rename from tests/test-render/main.k rename to tests/test-stack/main.k diff --git a/tests/test-render/model b/tests/test-stack/model similarity index 100% rename from tests/test-render/model rename to tests/test-stack/model From cdb8d5d2789b9328043afa33c698bcd40d9fee7d Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Tue, 28 Apr 2026 20:14:31 -0500 Subject: [PATCH 23/44] test(e2e): upgrade unified psql e2e to full Ready integration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit volume-snapshot-stack v0.1.0 is now published, so we can install it as a dependency Configuration package and bring snapshot-controller into the test cluster. With that, the whole chain reconciles to Ready in-cluster: 1. VolumeSnapshotStack XR (in extraResources) → snapshot-controller live 2. PSQLStack manifest → Helm-installs CNPG + atlas-operator + the psql VolumeSnapshotClass 3. PSQLCluster manifest → CNPG bootstraps a real Postgres with a real PVC 4. PSQLBranch manifest → snapshots the source PVC, restores into a new CNPG cluster Pattern mirrors the aws-observe-stack e2e (initResources for dependency Configuration packages, extraResources for the dependent XRs). defaultConditions: Synced → Ready timeoutSeconds: 1800 → 5400 (90 min for the full chain) cleanupTimeoutSeconds: 900 → 1800 --- tests/e2etest-psql/main.k | 96 ++++++++++++++++++++++++++++----------- 1 file changed, 70 insertions(+), 26 deletions(-) diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index ec04191..c7451ec 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -4,24 +4,26 @@ import models.ai.com.ops.hops.v1alpha1 as hopsv1alpha1 import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 # ============================================================================== -# Unified E2E Test for psql-stack package +# Unified E2E Test for psql-stack package — full Ready integration # -# Exercises all three XRs in the package together: +# Exercises all three XRs in the package end-to-end: # - PSQLStack (installs CNPG operator + atlas-operator via Helm) # - PSQLCluster (per-app Postgres XR; composes a CNPG Cluster CR + ExternalSecret) # - PSQLBranch (ephemeral fork; composes VolumeSnapshot + bootstrapped CNPG Cluster) # -# Bar: defaultConditions = ["Synced"]. We verify each XR composes its children -# correctly. We do NOT wait for the wrapped CNPG Cluster / VolumeSnapshot to -# reconcile — that requires CNPG + snapshot-controller to be live in-cluster. +# Bar: defaultConditions = ["Ready"]. The chain is: +# 1. snapshot-controller installed via VolumeSnapshotStack (initResources + +# extraResources VolumeSnapshotStack XR) +# 2. PSQLStack reconciles → Helm-installs CNPG + atlas-operator + a +# VolumeSnapshotClass that PSQLBranch uses as its forking target +# 3. PSQLCluster reconciles → CNPG provisions a real Postgres cluster (the +# operator is now live) with a real PVC +# 4. PSQLBranch reconciles → snapshots the source cluster's PVC, restores +# into a new CNPG cluster (snapshot-controller is now live) # -# TODO (after volume-snapshot-stack v0.1.0 releases): upgrade this test to -# defaultConditions = ["Ready"] using the observe e2e pattern: -# - initResources: install volume-snapshot-stack Configuration package -# - extraResources: VolumeSnapshotStack XR (deploys snapshot-controller) -# - bump timeoutSeconds to ~5400 (PSQLStack Helm install + CNPG bootstrap + -# real PVC + snapshot + bootstrap-from-snapshot is a long chain) -# Tracked in tasks/merge-psql-client-apis-into-stack progress log. +# Pattern mirrors aws-observe-stack's e2e: install dependent Configuration +# packages via initResources, materialize their XRs in extraResources, then +# the manifests under test depend on them. # # Run: make e2e # Run without cleanup: up test run tests/e2etest-psql --e2e --skip-control-plane-cleanup @@ -39,18 +41,42 @@ _items = [ crossplane = { autoUpgrade.channel = "Rapid" } - defaultConditions = ["Synced"] - timeoutSeconds = 1800 # 30 min — Helm install + composition apply - cleanupTimeoutSeconds = 900 # 15 min + defaultConditions = ["Ready"] + # 90 minutes — long chain: snapshot-controller install → PSQLStack + # Helm install (CNPG + atlas + VolumeSnapshotClass) → PSQLCluster + # bootstraps a real Postgres → PSQLBranch snapshots + restores. + timeoutSeconds = 5400 + cleanupTimeoutSeconds = 1800 skipDelete = False # ============================================================== - # extraResources: RBAC + Provider configs - # NOT deleted during cleanup + # initResources: dependency Configuration packages + # Installed BEFORE manifests, NOT deleted during cleanup. + # ============================================================== + initResources = [ + # volume-snapshot-stack — provides the cluster-wide + # snapshot-controller that PSQLBranch's VolumeSnapshot + # children rely on. EKS Auto Mode + colima ship the snapshot + # CRDs but no controller; this stack closes that gap. + { + apiVersion = "pkg.crossplane.io/v1" + kind = "Configuration" + metadata = { + name = "hops-ops-volume-snapshot-stack" + } + spec = { + package = "ghcr.io/hops-ops/volume-snapshot-stack:v0.1.0" + } + } + ] + + # ============================================================== + # extraResources: RBAC + Provider configs + dependent XRs + # NOT deleted during cleanup. # ============================================================== extraResources = [ - # cluster-admin to all SAs in crossplane-system so Helm + Kubernetes - # providers can apply wrapped manifests freely. + # cluster-admin to all SAs in crossplane-system so Helm + + # Kubernetes providers can apply wrapped manifests freely. { apiVersion = "rbac.authorization.k8s.io/v1" kind = "ClusterRoleBinding" @@ -68,7 +94,7 @@ _items = [ } ] } - # Helm ProviderConfig — for PSQLStack (installs CNPG + atlas-operator) + # Helm ProviderConfig — for PSQLStack + VolumeSnapshotStack { apiVersion = "helm.m.crossplane.io/v1beta1" kind = "ProviderConfig" @@ -96,11 +122,26 @@ _items = [ } } } + # VolumeSnapshotStack XR — installs the cluster-wide + # snapshot-controller via Helm. PSQLBranch's VolumeSnapshot + # objects only get reconciled to readyToUse=true when this + # controller is running. + { + apiVersion = "hops.ops.com.ai/v1alpha1" + kind = "VolumeSnapshotStack" + metadata = { + name = "snapshot" + namespace = "default" + } + spec = { + clusterName = "default" + } + } ] # ============================================================== - # manifests: all three XRs applied together - # Deleted during cleanup (unless --skip-delete) + # manifests: all three XRs under test, applied together. + # Deleted during cleanup (unless --skip-delete). # ============================================================== manifests = [ # 1. PSQLStack — Helm-installs CNPG operator + atlas-operator + VolumeSnapshotClass @@ -132,8 +173,9 @@ _items = [ } } } - # 2. PSQLCluster — per-app Postgres XR (composition only; CNPG CRDs - # aren't reconciled in kind without the operator being live) + # 2. PSQLCluster — per-app Postgres XR. CNPG (installed by + # PSQLStack above) provisions a real Postgres cluster with + # a real PVC backing the data. hopsv1alpha1.PSQLCluster { metadata = { name = _cluster_name @@ -154,8 +196,10 @@ _items = [ credentials.superuser.managedBy = "" } } - # 3. PSQLBranch — ephemeral fork (composition only; VolumeSnapshot - # + recovery bootstrap not exercised without snapshot-controller) + # 3. PSQLBranch — ephemeral fork. Snapshots the source + # PSQLCluster's PVC (snapshot-controller installed by the + # VolumeSnapshotStack extraResource handles this), restores + # into a new CNPG cluster bootstrapped from the snapshot. hopsv1alpha1.PSQLBranch { metadata = { name = _branch_name From ee3373eea71bb209aaae0ebd28de8b9842868c04 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Tue, 28 Apr 2026 20:22:33 -0500 Subject: [PATCH 24/44] fix(e2e): drop StackGres-era spec fields from PSQLStack manifest The unified e2e was carrying over `stackgresOperator.values` and `atlasOperator.values` from before the CNPG pivot. The current PSQLStack XRD uses `cnpg` (not `stackgresOperator`), `namespace` defaults to `cnpg-system` (not `stackgres`), and the kind-cluster defaults work without any operator-values overrides. Match `local/psqlstack.yaml`'s minimal shape: clusterName + labels + ProviderConfig refs only. Add `kubernetesProviderConfigRef` since PSQLStack now also applies the VolumeSnapshotClass via the kubernetes provider. --- tests/e2etest-psql/main.k | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index c7451ec..a280c04 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -144,7 +144,8 @@ _items = [ # Deleted during cleanup (unless --skip-delete). # ============================================================== manifests = [ - # 1. PSQLStack — Helm-installs CNPG operator + atlas-operator + VolumeSnapshotClass + # 1. PSQLStack — Helm-installs CNPG operator + atlas-operator + VolumeSnapshotClass. + # Defaults: namespace=cnpg-system, ProviderConfigs default to clusterName. hopsv1alpha1.PSQLStack { metadata = { name = _test_name @@ -152,7 +153,6 @@ _items = [ } spec = { clusterName = "default" - namespace = "stackgres" labels = { "hops.ops.com.ai/e2etest" = "true" "hops.ops.com.ai/test-run" = _now @@ -161,15 +161,9 @@ _items = [ name = "default" kind = "ProviderConfig" } - # Minimal resource requests for kind cluster - stackgresOperator.values = { - deploy = { - operator = True - restapi = True - } - } - atlasOperator.values = { - prewarmDevDB = False + kubernetesProviderConfigRef = { + name = "default" + kind = "ProviderConfig" } } } From 2e08b7ed9fc4c17d30be846caa18d5fed9566443 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 29 Apr 2026 12:20:08 -0500 Subject: [PATCH 25/44] docs(psqlstacks): correct stale CSI driver and VSC defaults MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three locations claimed the default driver was `ebs.csi.aws.com` and the VolumeSnapshotClass was "named after the XR" — both wrong relative to the actual schema (driver `ebs.csi.eks.amazonaws.com`, name defaults to `psql`). This text shows up in `kubectl explain`, generated docs, and template comments, so it needed to match. Also clarified `status.ready` description: components are toggleable, so readiness is "every enabled component is Ready" rather than implying all four are mandatory. From CodeRabbit review on PR #10. --- apis/psqlstacks/definition.yaml | 11 ++++++----- functions/stack/000-state-init.yaml.gotmpl | 5 +++-- functions/stack/170-volumesnapshotclass.yaml.gotmpl | 6 ++++-- 3 files changed, 13 insertions(+), 9 deletions(-) diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index 5b60ad7..b52f80c 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -147,10 +147,11 @@ spec: snapshotClass: description: | VolumeSnapshotClass used by PSQLBranch as its forking target. The - stack composes exactly one VolumeSnapshotClass named after the XR - (default: "psql"). On EKS Auto Mode the default driver - (ebs.csi.aws.com) is correct out of the box; override `driver` for - other CSI providers (e.g. driver.longhorn.io, hostpath.csi.k8s.io). + stack composes exactly one VolumeSnapshotClass with `name` defaulting + to "psql" (PSQLBranch references it by name). The default driver is + `ebs.csi.eks.amazonaws.com` (EKS Auto Mode's managed EBS CSI driver); + override `driver` for other providers (e.g. `ebs.csi.aws.com` for + self-managed EBS, `driver.longhorn.io`, `hostpath.csi.k8s.io`). PSQLClusters target whatever StorageClass the cluster already provides (Auto Mode ships gp3 by default) — the stack does not compose a StorageClass. @@ -188,7 +189,7 @@ spec: type: object properties: ready: - description: Overall readiness — true once CNPG, Atlas, the scale-to-zero plugin, and the VolumeSnapshotClass are all Ready. + description: Overall readiness — true once every enabled component is Ready (CNPG, Atlas, the scale-to-zero plugin if `scaleToZeroPlugin.enabled`, and the VolumeSnapshotClass if `snapshotClass.enabled`). Disabled components are treated as Ready. type: boolean required: - spec diff --git a/functions/stack/000-state-init.yaml.gotmpl b/functions/stack/000-state-init.yaml.gotmpl index 83b09a0..96bbfe0 100644 --- a/functions/stack/000-state-init.yaml.gotmpl +++ b/functions/stack/000-state-init.yaml.gotmpl @@ -70,8 +70,9 @@ # ============================================================================== # VolumeSnapshotClass — the stack's only storage-layer composition. # PSQLBranch targets this VSC by name when forking primaries. Default driver -# is ebs.csi.aws.com (correct for EKS Auto Mode); override for other CSI -# providers via $spec.snapshotClass.driver. +# is ebs.csi.eks.amazonaws.com (EKS Auto Mode's managed EBS CSI driver); +# override for other CSI providers via $spec.snapshotClass.driver +# (e.g. ebs.csi.aws.com for self-managed EBS, driver.longhorn.io). # ============================================================================== {{- $snapshotSpec := $spec.snapshotClass | default dict }} {{- $snapshotEnabled := true }} diff --git a/functions/stack/170-volumesnapshotclass.yaml.gotmpl b/functions/stack/170-volumesnapshotclass.yaml.gotmpl index fab378c..92fa940 100644 --- a/functions/stack/170-volumesnapshotclass.yaml.gotmpl +++ b/functions/stack/170-volumesnapshotclass.yaml.gotmpl @@ -7,9 +7,11 @@ # whatever the target cluster already provides (gp3 on EKS Auto Mode, # standard on kind/k3d, etc.). # -# Default driver: ebs.csi.aws.com (correct for EKS Auto Mode out of the box). +# Default driver: ebs.csi.eks.amazonaws.com (EKS Auto Mode's managed EBS CSI +# driver; sourced from $state.snapshotClass.driver in 000-state-init.yaml.gotmpl). # Override $spec.snapshotClass.driver for non-AWS / non-EBS providers -# (e.g. driver.longhorn.io, hostpath.csi.k8s.io). +# (e.g. ebs.csi.aws.com for self-managed EBS, driver.longhorn.io, +# hostpath.csi.k8s.io). # {{- $snap := $state.snapshotClass }} From 525fba093f86f65cabe3473d0e63491516821ff5 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 29 Apr 2026 12:21:04 -0500 Subject: [PATCH 26/44] test(psqlstacks): assert Atlas Release in scale-to-zero-disabled test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The s2z-disabled test description claimed cnpg + atlas + VSC are still composed, but only cnpg and the VSC were asserted — a regression that broke the Atlas Release would silently pass. Added the Atlas assertion. From CodeRabbit review on PR #10. --- tests/test-stack/main.k | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/test-stack/main.k b/tests/test-stack/main.k index 18adf80..72e56e6 100644 --- a/tests/test-stack/main.k +++ b/tests/test-stack/main.k @@ -303,6 +303,11 @@ _items = [ kind = "Release" metadata.name = "cloudnative-pg" } + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "atlas-operator" + } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" From e68d59275c0369864080060e6663561880041e01 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 29 Apr 2026 12:32:25 -0500 Subject: [PATCH 27/44] fix(psqlstacks): create CNPG/Atlas teardown-order Usage on existence, not readiness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Usage that protects cnpg-operator from premature deletion (so Atlas is torn down first and CNPG isn't yanked while Atlas's migration state is still live) was rendered only after both Releases reported Ready. That left a window: if Atlas was mid-progressing or in error and the user deleted the stack, no Usage existed yet, and CNPG could be deleted first — exactly the ordering this guard exists to prevent. Switched the gate from `$state.observed.{cnpg,atlasOperator}.ready` to `hasKey $.observed.resources "{cnpg,atlas}-operator"`. The Usage now appears as soon as both Releases are observed, regardless of their readiness state. Verified locally: render tests still pass; cluster reinstall via `hops config install --path` brings all three XRs back to Synced/Ready. From CodeRabbit review on PR #10. --- functions/stack/200-cnpg-operator.yaml.gotmpl | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/functions/stack/200-cnpg-operator.yaml.gotmpl b/functions/stack/200-cnpg-operator.yaml.gotmpl index 8ccecdc..e1a442c 100644 --- a/functions/stack/200-cnpg-operator.yaml.gotmpl +++ b/functions/stack/200-cnpg-operator.yaml.gotmpl @@ -60,8 +60,14 @@ spec: # Usage: Protect cnpg-operator from deletion until atlas-operator is deleted. # Atlas manages schema migrations against CNPG-managed databases; deleting CNPG # first would orphan Atlas's migration state. +# +# Gate on observed *existence* (both Releases present in $.observed.resources), +# NOT readiness. If Atlas is still progressing or in error and the user deletes +# the stack at that moment, the Usage must already exist — otherwise CNPG could +# be torn down first and orphan Atlas. # ============================================================================== -{{- if and $state.observed.cnpg.ready $state.observed.atlasOperator.ready }} +{{- $observed := $.observed.resources | default dict }} +{{- if and (hasKey $observed "cnpg-operator") (hasKey $observed "atlas-operator") }} --- apiVersion: protection.crossplane.io/v1beta1 kind: Usage From 03d233cb9ee3fd24ee69670ecc6f6673fb6f66bb Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 29 Apr 2026 13:28:11 -0500 Subject: [PATCH 28/44] feat(psqlcluster)!: rework credentials around app/superuser, ESO-shaped externalSecret MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous `credentials.superuser` shape was misleading: the secret named "superuser" was actually wired into CNPG's `bootstrap.initdb.secret`, which takes the *application user's* credentials. The actual postgres superuser secret was either auto-generated by CNPG or absent from the spec entirely. And it collided with CNPG's own `-superuser` secret naming. New shape (no backwards compat — still alpha): spec.app: # always present; wires bootstrap.initdb role: app # Postgres role name database: app # Application database name secretName: "" # K8s Secret; default -app externalSecret: # OPTIONAL — when set, ESO renders the Secret secretStore: kind: ClusterSecretStore | SecretStore name: hops-aws-secrets-manager namespace: "" # for SecretStore (defaults to XR ns) secretRef: path: my-cluster/app # remote location; JSON value with username+password spec.superuser: # OPTIONAL — omit to let CNPG auto-generate secretName: "" # default -superuser externalSecret: { ... } # same shape as app.externalSecret When `superuser` is set, the composition renders `spec.superuserSecret` on the wrapped CNPG Cluster CR; otherwise CNPG auto-generates and stores the secret at `-superuser` (its own convention). Field names mirror External Secrets Operator's CRD shape so anyone familiar with ESO can read it at a glance. Status now exposes connection details so dependent XRs can wire without hardcoding: status.app: { secretName, database, host, port } status.superuser: { secretName } Render template factored to a single `psqlcluster.externalSecret` definition so the app/superuser ExternalSecrets share one source of truth. Tests rewritten: - Test 1 ("minimal-renders-cluster-only") asserts default mode renders only the Cluster — no ExternalSecret, since externalSecret is now opt-in. - Test 7 ("external-secret-renders-when-opted-in") explicitly asserts the ESO ExternalSecret with the new shape. Migration for existing manifests: `credentials.superuser.managedBy: ""` → remove the block (BYO is now default). `credentials.superuser` with ESO fields → move under `spec.app.externalSecret` matching the new shape. From CodeRabbit review on PR #10. --- .github/workflows/on-pr.yaml | 8 + .github/workflows/on-push-main.yaml | 4 + apis/psqlclusters/definition.yaml | 136 +++- functions/cluster/000-state-init.yaml.gotmpl | 56 +- .../cluster/100-external-secret.yaml.gotmpl | 78 +- .../cluster/200-cnpg-cluster.yaml.gotmpl | 9 +- functions/cluster/999-status.yaml.gotmpl | 14 + tests/e2etest-psql/main.k | 2 +- .../ai/com/ops/hops/v1alpha1/psqlcluster.k | 664 ++++++++++++++++++ tests/test-cluster/main.k | 70 +- .../ai/com/ops/hops/v1alpha1/psqlcluster.k | 240 ++++++- 11 files changed, 1156 insertions(+), 125 deletions(-) create mode 100644 tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k diff --git a/.github/workflows/on-pr.yaml b/.github/workflows/on-pr.yaml index 44b939c..83568b2 100644 --- a/.github/workflows/on-pr.yaml +++ b/.github/workflows/on-pr.yaml @@ -44,6 +44,14 @@ jobs: e2e: uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@feat/multi-api-support + with: + # Long chain: snapshot-controller install → PSQLStack Helm installs + # (CNPG + atlas + VolumeSnapshotClass) → PSQLCluster bootstraps a real + # Postgres → PSQLBranch snapshots + restores. With cold image pulls and + # the occasional rate-limit slowness on the kind node, 90 min is what + # this chain needs to converge reliably. + timeout-minutes: 90 + cleanup-timeout-minutes: 30 publish: needs: diff --git a/.github/workflows/on-push-main.yaml b/.github/workflows/on-push-main.yaml index c70333d..1880825 100644 --- a/.github/workflows/on-push-main.yaml +++ b/.github/workflows/on-push-main.yaml @@ -40,6 +40,10 @@ jobs: e2e: uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@feat/multi-api-support + with: + # See on-pr.yaml — 90 min for the full Ready chain. + timeout-minutes: 90 + cleanup-timeout-minutes: 30 version-and-tag: name: Version and Tag diff --git a/apis/psqlclusters/definition.yaml b/apis/psqlclusters/definition.yaml index 94c3e8e..cae3c7c 100644 --- a/apis/psqlclusters/definition.yaml +++ b/apis/psqlclusters/definition.yaml @@ -102,27 +102,108 @@ spec: x-kubernetes-preserve-unknown-fields: true # ============================================================ - # Credentials — typed intent + # Application user — the role your app connects as. + # + # Common case: drop in `app.externalSecret` pointing at the + # platform's ClusterSecretStore and key. Or omit `externalSecret` + # entirely and pre-create the K8s Secret named `` + # with `username`+`password` keys (BYO mode). # ============================================================ - credentials: - description: Where Postgres credentials are sourced from. + app: + description: Application user — the Postgres role your app connects as. CNPG uses these credentials for `bootstrap.initdb`. type: object properties: - superuser: + role: + description: Postgres role name owning the application database. Defaults to "app". + type: string + default: app + database: + description: Application database name. Defaults to "app". + type: string + default: app + secretName: + description: K8s Secret holding the app role's credentials. Defaults to "-app". With `externalSecret` set, the composition writes this Secret; otherwise pre-create it yourself. + type: string + default: "" + externalSecret: + description: Source the credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. + type: object + properties: + secretStore: + description: Reference to the SecretStore or ClusterSecretStore providing the value. + type: object + properties: + kind: + description: Store kind. Use ClusterSecretStore for cluster-scoped stores (the platform default), SecretStore for namespaced stores in this XR's namespace. + type: string + enum: + - ClusterSecretStore + - SecretStore + default: ClusterSecretStore + name: + description: Name of the SecretStore or ClusterSecretStore. + type: string + namespace: + description: Namespace of the SecretStore (ignored for ClusterSecretStore). Defaults to the PSQLCluster's namespace. + type: string + required: + - name + secretRef: + description: Where in the secrets backend the credentials live. The remote value must be a JSON blob with `username` and `password` properties — both are extracted into the resulting K8s Secret. + type: object + properties: + path: + description: Path/key in the secrets backend (e.g. AWS Secrets Manager `my-cluster/app`). + type: string + required: + - path + required: + - secretStore + - secretRef + + # ============================================================ + # Postgres superuser — optional. Omit the block to let CNPG + # auto-generate the superuser secret at `-superuser`. + # Set when tooling needs the superuser password sourced from a + # specific secrets backend (e.g. schema migration platforms). + # ============================================================ + superuser: + description: Optional postgres superuser credentials. Omit the block to let CNPG auto-generate the superuser secret. + type: object + properties: + secretName: + description: K8s Secret holding the superuser credentials. Defaults to "-superuser". + type: string + default: "" + externalSecret: + description: Source the superuser credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. type: object properties: - managedBy: - description: ClusterSecretStore name for sourcing the superuser password from a secrets backend (default ESO ClusterSecretStore convention). Set "" to skip ExternalSecret composition (consumer must provide a K8s Secret named `` with `username` + `password` keys). - type: string - default: hops-aws-secrets-manager - secretName: - description: K8s Secret name receiving the credentials. Defaults to "-superuser". - type: string - default: "" - remoteKey: - description: Key in the remote secrets backend (AWS Secrets Manager) holding the superuser password. Defaults to "/superuser". - type: string - default: "" + secretStore: + type: object + properties: + kind: + type: string + enum: + - ClusterSecretStore + - SecretStore + default: ClusterSecretStore + name: + type: string + namespace: + type: string + required: + - name + secretRef: + type: object + properties: + path: + type: string + required: + - path + required: + - secretStore + - secretRef # ============================================================ # Cross-stack integration toggles @@ -204,5 +285,28 @@ spec: clusterPhase: description: Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). type: string + app: + description: Connection details for the application user. Other XRs can reference this to wire connection strings without hardcoding. + type: object + properties: + secretName: + description: K8s Secret holding the app role's credentials. + type: string + database: + description: Application database name. + type: string + host: + description: Service hostname for read-write connections (CNPG `-rw` service). + type: string + port: + description: Postgres port. + type: integer + superuser: + description: Connection details for the postgres superuser. + type: object + properties: + secretName: + description: K8s Secret holding the superuser credentials (CNPG-managed if `spec.superuser` was not set). + type: string required: - spec diff --git a/functions/cluster/000-state-init.yaml.gotmpl b/functions/cluster/000-state-init.yaml.gotmpl index a9e03a7..b8a78f7 100644 --- a/functions/cluster/000-state-init.yaml.gotmpl +++ b/functions/cluster/000-state-init.yaml.gotmpl @@ -76,27 +76,44 @@ }} # ============================================================================== -# Credentials -# ============================================================================== -{{- $credSpec := ($spec.credentials | default dict).superuser | default dict }} -{{- $credManagedBy := "hops-aws-secrets-manager" }} -{{- if hasKey $credSpec "managedBy" }} - {{- $credManagedBy = $credSpec.managedBy }} +# Application user — always present. Wires CNPG's bootstrap.initdb shape. +# `externalSecret` is optional: when set, the composition renders an ESO +# ExternalSecret pulling from the named (Cluster)SecretStore. When omitted, +# the consumer is expected to pre-create the K8s Secret at `secretName` +# (BYO mode). +# ============================================================================== +{{- $appSpec := $spec.app | default dict }} +{{- $appRole := $appSpec.role | default "app" }} +{{- $appDatabase := $appSpec.database | default "app" }} +{{- $appSecretName := $appSpec.secretName }} +{{- if not $appSecretName }} + {{- $appSecretName = printf "%s-app" $name }} {{- end }} -{{- $credSecretName := $credSpec.secretName }} -{{- if not $credSecretName }} - {{- $credSecretName = printf "%s-superuser" $name }} +{{- $app := dict + "role" $appRole + "database" $appDatabase + "secretName" $appSecretName + "externalSecret" ($appSpec.externalSecret | default dict) +}} + +# ============================================================================== +# Superuser — optional. Render only when the user explicitly opts in by +# setting `spec.superuser.secretName` or `spec.superuser.externalSecret`. +# Otherwise CNPG auto-generates the superuser secret at `-superuser`. +# ============================================================================== +{{- $suSpec := $spec.superuser | default dict }} +{{- $suEnabled := false }} +{{- if or $suSpec.secretName $suSpec.externalSecret }} + {{- $suEnabled = true }} {{- end }} -{{- $credRemoteKey := $credSpec.remoteKey }} -{{- if not $credRemoteKey }} - {{- $credRemoteKey = printf "%s/superuser" $name }} +{{- $suSecretName := $suSpec.secretName }} +{{- if not $suSecretName }} + {{- $suSecretName = printf "%s-superuser" $name }} {{- end }} -{{- $credentials := dict - "superuser" (dict - "managedBy" $credManagedBy - "secretName" $credSecretName - "remoteKey" $credRemoteKey - ) +{{- $superuser := dict + "enabled" $suEnabled + "secretName" $suSecretName + "externalSecret" ($suSpec.externalSecret | default dict) }} # ============================================================================== @@ -151,7 +168,8 @@ "instances" $instances "storage" $storage "postgresql" $postgresql - "credentials" $credentials + "app" $app + "superuser" $superuser "scaleToZero" $scaleToZero "branching" $branching "monitoring" $monitoring diff --git a/functions/cluster/100-external-secret.yaml.gotmpl b/functions/cluster/100-external-secret.yaml.gotmpl index a09ab08..d65dc01 100644 --- a/functions/cluster/100-external-secret.yaml.gotmpl +++ b/functions/cluster/100-external-secret.yaml.gotmpl @@ -1,40 +1,44 @@ # code: language=yaml # -# ExternalSecret — sources the Postgres superuser password from the platform's -# ClusterSecretStore (default: hops-aws-secrets-manager) and writes it to a -# K8s Secret that the CNPG Cluster will reference for bootstrap. +# ExternalSecrets — source Postgres role passwords from a SecretStore / +# ClusterSecretStore via External Secrets Operator and write K8s Secrets +# that the CNPG Cluster references. # -# Skipped when credentials.superuser.managedBy is empty — consumer must -# pre-create the K8s Secret themselves in that case. +# Rendered when the consumer sets `app.externalSecret` (and optionally +# `superuser.externalSecret`). When omitted, the consumer is in BYO mode +# and must pre-create the K8s Secret named `` with `username` + +# `password` keys. +# +# The remote value at `secretRef.path` must be a JSON blob with `username` +# and `password` fields — both are extracted into the resulting K8s Secret. # -{{- $cred := $state.credentials.superuser }} -{{- if $cred.managedBy }} +{{- define "psqlcluster.externalSecret" -}} --- apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object metadata: - name: {{ $state.name }}-external-secret + name: {{ .ResourceName }} annotations: - {{ setResourceNameAnnotation "external-secret" }} - labels: {{ $state.labels | toJson }} + {{ .ResourceNameAnnotation }} + labels: {{ .Labels | toJson }} spec: - managementPolicies: {{ $state.managementPolicies | toJson }} + managementPolicies: {{ .ManagementPolicies | toJson }} forProvider: manifest: apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: - name: {{ $cred.secretName }} - namespace: {{ $state.namespace }} - labels: {{ $state.labels | toJson }} + name: {{ .SecretName }} + namespace: {{ .Namespace }} + labels: {{ .Labels | toJson }} spec: refreshInterval: 1h secretStoreRef: - name: {{ $cred.managedBy }} - kind: ClusterSecretStore + name: {{ .SecretStore.name }} + kind: {{ .SecretStore.kind | default "ClusterSecretStore" }} target: - name: {{ $cred.secretName }} + name: {{ .SecretName }} creationPolicy: Owner template: type: kubernetes.io/basic-auth @@ -44,13 +48,45 @@ spec: data: - secretKey: username remoteRef: - key: {{ $cred.remoteKey | quote }} + key: {{ .SecretRef.path | quote }} property: username - secretKey: password remoteRef: - key: {{ $cred.remoteKey | quote }} + key: {{ .SecretRef.path | quote }} property: password providerConfigRef: - name: {{ $state.kubernetesProviderConfigRef.name }} - kind: {{ $state.kubernetesProviderConfigRef.kind }} + name: {{ .ProviderConfigName }} + kind: {{ .ProviderConfigKind }} +{{- end }} + +{{- $appES := $state.app.externalSecret }} +{{- if and $appES.secretStore $appES.secretRef }} +{{- template "psqlcluster.externalSecret" (dict + "ResourceName" (printf "%s-external-secret-app" $state.name) + "ResourceNameAnnotation" (setResourceNameAnnotation "external-secret-app") + "Labels" $state.labels + "ManagementPolicies" $state.managementPolicies + "Namespace" $state.namespace + "SecretName" $state.app.secretName + "SecretStore" $appES.secretStore + "SecretRef" $appES.secretRef + "ProviderConfigName" $state.kubernetesProviderConfigRef.name + "ProviderConfigKind" $state.kubernetesProviderConfigRef.kind +) }} +{{- end }} + +{{- $suES := $state.superuser.externalSecret }} +{{- if and $state.superuser.enabled $suES.secretStore $suES.secretRef }} +{{- template "psqlcluster.externalSecret" (dict + "ResourceName" (printf "%s-external-secret-superuser" $state.name) + "ResourceNameAnnotation" (setResourceNameAnnotation "external-secret-superuser") + "Labels" $state.labels + "ManagementPolicies" $state.managementPolicies + "Namespace" $state.namespace + "SecretName" $state.superuser.secretName + "SecretStore" $suES.secretStore + "SecretRef" $suES.secretRef + "ProviderConfigName" $state.kubernetesProviderConfigRef.name + "ProviderConfigKind" $state.kubernetesProviderConfigRef.kind +) }} {{- end }} diff --git a/functions/cluster/200-cnpg-cluster.yaml.gotmpl b/functions/cluster/200-cnpg-cluster.yaml.gotmpl index 0e5d284..abf8c45 100644 --- a/functions/cluster/200-cnpg-cluster.yaml.gotmpl +++ b/functions/cluster/200-cnpg-cluster.yaml.gotmpl @@ -66,9 +66,9 @@ spec: "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) "bootstrap" (dict "initdb" (dict - "database" "app" - "owner" "app" - "secret" (dict "name" $state.credentials.superuser.secretName) + "database" $state.app.database + "owner" $state.app.role + "secret" (dict "name" $state.app.secretName) ) ) "storage" (dict @@ -78,6 +78,9 @@ spec: "enablePodMonitor" $state.monitoring.enabled ) }} + {{- if $state.superuser.enabled }} + {{- $_ := set $clusterSpec "superuserSecret" (dict "name" $state.superuser.secretName) }} + {{- end }} {{- if $state.storage.class }} {{- $_ := set $clusterSpec.storage "storageClass" $state.storage.class }} {{- end }} diff --git a/functions/cluster/999-status.yaml.gotmpl b/functions/cluster/999-status.yaml.gotmpl index 08d7187..44517a4 100644 --- a/functions/cluster/999-status.yaml.gotmpl +++ b/functions/cluster/999-status.yaml.gotmpl @@ -4,12 +4,26 @@ # {{- $xr := getCompositeResource . }} +{{- /* CNPG always exposes the rw service at -rw..svc.cluster.local */}} +{{- $appHost := printf "%s-rw.%s.svc.cluster.local" $state.name $state.namespace }} +{{- /* Superuser secret location: user-provided if opted in, else CNPG's auto-generated `-superuser` */}} +{{- $superuserSecretName := printf "%s-superuser" $state.name }} +{{- if $state.superuser.enabled }} + {{- $superuserSecretName = $state.superuser.secretName }} +{{- end }} --- apiVersion: {{ $xr.apiVersion }} kind: {{ $xr.kind }} status: ready: {{ $state.status.ready }} clusterPhase: {{ $state.status.clusterPhase | quote }} + app: + secretName: {{ $state.app.secretName | quote }} + database: {{ $state.app.database | quote }} + host: {{ $appHost | quote }} + port: 5432 + superuser: + secretName: {{ $superuserSecretName | quote }} {{- if $state.warnings }} conditions: {{- range $i, $msg := $state.warnings }} diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index a280c04..e9d8f86 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -187,7 +187,7 @@ _items = [ } storage = {size = "1Gi"} # Skip ExternalSecret in e2e — no ESO ClusterSecretStore in kind harness. - credentials.superuser.managedBy = "" + app.managedBy = "" } } # 3. PSQLBranch — ephemeral fork. Snapshots the source diff --git a/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k b/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k new file mode 100644 index 0000000..123fca7 --- /dev/null +++ b/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k @@ -0,0 +1,664 @@ +""" +This file was generated by the KCL auto-gen tool. DO NOT EDIT. +Editing this file might prove futile when you re-run the KCL auto-gen generate command. +""" +import k8s.apimachinery.pkg.apis.meta.v1 + + +schema PSQLCluster: + r""" + PSQLCluster is the per-app Postgres primitive. Composes a CloudNativePG + `Cluster` plus the wiring (annotations, plugin entries, labels) required + for the platform integrations (scale-to-zero, branching, monitoring, HA). + + Companion to `psql-stack` (which provides the CNPG operator + S2Z plugin + substrate; required as a target-cluster prerequisite) and to `psql-branch` + (which composes ephemeral CoW forks of a PSQLCluster via VolumeSnapshot). + + Intent-first surface: integration toggles (`scaleToZero.enabled`, + `branching.enabled`, `monitoring.enabled`, `ha.enabled`) translate to + CNPG mechanics inside the render template — consumer manifests don't + reach for `cnpg.overrideAllValues` unless their use case is genuinely + novel. + + + Attributes + ---------- + apiVersion : str, default is "hops.ops.com.ai/v1alpha1", required + APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + kind : str, default is "PSQLCluster", required + Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + metadata : v1.ObjectMeta, default is Undefined, optional + metadata + spec : HopsOpsComAiV1alpha1PSQLClusterSpec, default is Undefined, required + spec + status : HopsOpsComAiV1alpha1PSQLClusterStatus, default is Undefined, optional + status + """ + + + apiVersion: "hops.ops.com.ai/v1alpha1" = "hops.ops.com.ai/v1alpha1" + + kind: "PSQLCluster" = "PSQLCluster" + + metadata?: v1.ObjectMeta + + spec: HopsOpsComAiV1alpha1PSQLClusterSpec + + status?: HopsOpsComAiV1alpha1PSQLClusterStatus + + +schema HopsOpsComAiV1alpha1PSQLClusterSpec: + r""" + PSQLClusterSpec defines the desired state. + + Attributes + ---------- + app : HopsOpsComAiV1alpha1PSQLClusterSpecApp, default is Undefined, optional + app + branching : HopsOpsComAiV1alpha1PSQLClusterSpecBranching, default is Undefined, optional + branching + clusterName : str, default is Undefined, required + Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. + cnpg : HopsOpsComAiV1alpha1PSQLClusterSpecCnpg, default is Undefined, optional + cnpg + crossplane : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane, default is Undefined, optional + crossplane + ha : HopsOpsComAiV1alpha1PSQLClusterSpecHa, default is Undefined, optional + ha + instances : int, default is 1, optional + Number of Postgres instances. Default 1 (single-node). HA mode bumps to ha.replicas. + kubernetesProviderConfigRef : HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef, default is Undefined, optional + kubernetes provider config ref + labels : {str:str}, default is Undefined, optional + Custom labels merged with stack defaults; applied to every composed resource. + managementPolicies : [str], default is ["*"], optional + Crossplane managementPolicies applied to every composed resource. Defaults to ["*"]. + monitoring : HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring, default is Undefined, optional + monitoring + postgresql : HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql, default is Undefined, optional + postgresql + scaleToZero : HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero, default is Undefined, optional + scale to zero + storage : HopsOpsComAiV1alpha1PSQLClusterSpecStorage, default is Undefined, required + storage + superuser : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser, default is Undefined, optional + superuser + """ + + + app?: HopsOpsComAiV1alpha1PSQLClusterSpecApp + + branching?: HopsOpsComAiV1alpha1PSQLClusterSpecBranching + + clusterName: str + + cnpg?: HopsOpsComAiV1alpha1PSQLClusterSpecCnpg + + crossplane?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane + + ha?: HopsOpsComAiV1alpha1PSQLClusterSpecHa + + instances?: int = 1 + + kubernetesProviderConfigRef?: HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef + + labels?: {str:str} + + managementPolicies?: [str] = ["*"] + + monitoring?: HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring + + postgresql?: HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql + + scaleToZero?: HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero + + storage: HopsOpsComAiV1alpha1PSQLClusterSpecStorage + + superuser?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecApp: + r""" + Application user — the Postgres role your app connects as. CNPG uses these credentials for `bootstrap.initdb`. + + Attributes + ---------- + database : str, default is "app", optional + Application database name. Defaults to "app". + externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret, default is Undefined, optional + external secret + role : str, default is "app", optional + Postgres role name owning the application database. Defaults to "app". + secretName : str, default is Undefined, optional + K8s Secret holding the app role's credentials. Defaults to "-app". With `externalSecret` set, the composition writes this Secret; otherwise pre-create it yourself. + """ + + + database?: str = "app" + + externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret + + role?: str = "app" + + secretName?: str = "" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret: + r""" + Source the credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. + + Attributes + ---------- + secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef, default is Undefined, required + secret ref + secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore, default is Undefined, required + secret store + """ + + + secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef + + secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef: + r""" + Where in the secrets backend the credentials live. The remote value must be a JSON blob with `username` and `password` properties — both are extracted into the resulting K8s Secret. + + Attributes + ---------- + path : str, default is Undefined, required + Path/key in the secrets backend (e.g. AWS Secrets Manager `my-cluster/app`). + """ + + + path: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore: + r""" + Reference to the SecretStore or ClusterSecretStore providing the value. + + Attributes + ---------- + kind : str, default is "ClusterSecretStore", optional + Store kind. Use ClusterSecretStore for cluster-scoped stores (the platform default), SecretStore for namespaced stores in this XR's namespace. + name : str, default is Undefined, required + Name of the SecretStore or ClusterSecretStore. + namespace : str, default is Undefined, optional + Namespace of the SecretStore (ignored for ClusterSecretStore). Defaults to the PSQLCluster's namespace. + """ + + + kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" + + name: str + + namespace?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecBranching: + r""" + Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). + + Attributes + ---------- + enabled : bool, default is True, optional + enabled + snapshotClassName : str, default is "psql", optional + VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. + """ + + + enabled?: bool = True + + snapshotClassName?: str = "psql" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCnpg: + r""" + Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. + + Attributes + ---------- + overrideAllValues : any, default is Undefined, optional + Replaces the entire Cluster.spec — total escape hatch. + values : any, default is Undefined, optional + Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. + """ + + + overrideAllValues?: any + + values?: any + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane: + r""" + Configures how Crossplane will reconcile this composite resource + + Attributes + ---------- + compositionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef, default is Undefined, optional + composition ref + compositionRevisionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef, default is Undefined, optional + composition revision ref + compositionRevisionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector, default is Undefined, optional + composition revision selector + compositionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector, default is Undefined, optional + composition selector + compositionUpdatePolicy : str, default is Undefined, optional + composition update policy + resourceRefs : [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0], default is Undefined, optional + resource refs + """ + + + compositionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef + + compositionRevisionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef + + compositionRevisionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector + + compositionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector + + compositionUpdatePolicy?: "Automatic" | "Manual" + + resourceRefs?: [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0] + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition ref + + Attributes + ---------- + name : str, default is Undefined, required + name + """ + + + name: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision ref + + Attributes + ---------- + name : str, default is Undefined, required + name + """ + + + name: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision selector + + Attributes + ---------- + matchLabels : {str:str}, default is Undefined, required + match labels + """ + + + matchLabels: {str:str} + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane composition selector + + Attributes + ---------- + matchLabels : {str:str}, default is Undefined, required + match labels + """ + + + matchLabels: {str:str} + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0: + r""" + hops ops com ai v1alpha1 p SQL cluster spec crossplane resource refs items0 + + Attributes + ---------- + apiVersion : str, default is Undefined, required + api version + kind : str, default is Undefined, required + kind + name : str, default is Undefined, optional + name + """ + + + apiVersion: str + + kind: str + + name?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecHa: + r""" + Stack-wide HA. Bumps instances to `replicas` (default 3) and adds zonal topology spread. + + Attributes + ---------- + enabled : bool, default is Undefined, optional + enabled + replicas : int, default is 3, optional + replicas + topologySpreadByZone : bool, default is True, optional + topology spread by zone + """ + + + enabled?: bool = False + + replicas?: int = 3 + + topologySpreadByZone?: bool = True + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef: + r""" + Reference to the Kubernetes ProviderConfig used to apply the Cluster CR + ExternalSecret. Defaults to clusterName. + + Attributes + ---------- + kind : str, default is Undefined, optional + kind + name : str, default is Undefined, optional + name + """ + + + kind?: "ProviderConfig" | "ClusterProviderConfig" + + name?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring: + r""" + Add Prometheus scrape configuration. CNPG operator handles the actual PodMonitor creation when `monitoring.enablePodMonitor` is set on the Cluster CR. + + Attributes + ---------- + enabled : bool, default is True, optional + enabled + """ + + + enabled?: bool = True + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql: + r""" + Postgres version + tuning parameters. + + Attributes + ---------- + parameters : {str:str}, default is Undefined, optional + postgres.conf parameters (e.g. `shared_buffers`, `max_connections`). + version : str, default is "17", optional + Postgres major version. Defaults to "17". + """ + + + parameters?: {str:str} + + version?: str = "17" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero: + r""" + Auto-hibernate the cluster after `idleTimeout` of no client activity. Requires the cnpg-i-scale-to-zero plugin on the target cluster (provided by psql-stack). When `ha.enabled` is also true, the cluster will not actually scale to zero (HA implies always-available); a Warning condition is emitted on the XR but the annotation is rendered for tooling that reads it. + + Attributes + ---------- + enabled : bool, default is Undefined, optional + enabled + idleTimeout : str, default is "30m", optional + How long the cluster must be idle before hibernating. Defaults to "30m". + """ + + + enabled?: bool = False + + idleTimeout?: str = "30m" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecStorage: + r""" + Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (gp3 default on EKS Auto Mode) — bumping `size` upward grows the volume in place with no downtime. + + Attributes + ---------- + class : str, default is Undefined, optional + StorageClass name. Empty = cluster default (gp3 on Auto Mode). + size : str, default is Undefined, required + PVC size, e.g. "10Gi". Required. + """ + + + class?: str = "" + + size: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser: + r""" + Optional postgres superuser credentials. Omit the block to let CNPG auto-generate the superuser secret. + + Attributes + ---------- + externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret, default is Undefined, optional + external secret + secretName : str, default is Undefined, optional + K8s Secret holding the superuser credentials. Defaults to "-superuser". + """ + + + externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret + + secretName?: str = "" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret: + r""" + Source the superuser credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. + + Attributes + ---------- + secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef, default is Undefined, required + secret ref + secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore, default is Undefined, required + secret store + """ + + + secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef + + secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef: + r""" + hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret ref + + Attributes + ---------- + path : str, default is Undefined, required + path + """ + + + path: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore: + r""" + hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret store + + Attributes + ---------- + kind : str, default is "ClusterSecretStore", optional + kind + name : str, default is Undefined, required + name + namespace : str, default is Undefined, optional + namespace + """ + + + kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" + + name: str + + namespace?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterStatus: + r""" + PSQLClusterStatus defines the observed state. + + Attributes + ---------- + app : HopsOpsComAiV1alpha1PSQLClusterStatusApp, default is Undefined, optional + app + clusterPhase : str, default is Undefined, optional + Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). + conditions : [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0], default is Undefined, optional + Conditions of the resource. + crossplane : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane, default is Undefined, optional + crossplane + ready : bool, default is Undefined, optional + Overall readiness — true once the CNPG Cluster reaches `Cluster in healthy state` AND the ExternalSecret (if composed) is SecretSynced. + superuser : HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser, default is Undefined, optional + superuser + """ + + + app?: HopsOpsComAiV1alpha1PSQLClusterStatusApp + + clusterPhase?: str + + conditions?: [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0] + + crossplane?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane + + ready?: bool + + superuser?: HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusApp: + r""" + Connection details for the application user. Other XRs can reference this to wire connection strings without hardcoding. + + Attributes + ---------- + database : str, default is Undefined, optional + Application database name. + host : str, default is Undefined, optional + Service hostname for read-write connections (CNPG `-rw` service). + port : int, default is Undefined, optional + Postgres port. + secretName : str, default is Undefined, optional + K8s Secret holding the app role's credentials. + """ + + + database?: str + + host?: str + + port?: int + + secretName?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0: + r""" + hops ops com ai v1alpha1 p SQL cluster status conditions items0 + + Attributes + ---------- + lastTransitionTime : str, default is Undefined, required + last transition time + message : str, default is Undefined, optional + message + observedGeneration : int, default is Undefined, optional + observed generation + reason : str, default is Undefined, required + reason + status : str, default is Undefined, required + status + $type : str, default is Undefined, required + type + """ + + + lastTransitionTime: str + + message?: str + + observedGeneration?: int + + reason: str + + status: str + + $type: str + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane: + r""" + Indicates how Crossplane is reconciling this composite resource + + Attributes + ---------- + connectionDetails : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails, default is Undefined, optional + connection details + """ + + + connectionDetails?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails: + r""" + hops ops com ai v1alpha1 p SQL cluster status crossplane connection details + + Attributes + ---------- + lastPublishedTime : str, default is Undefined, optional + last published time + """ + + + lastPublishedTime?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser: + r""" + Connection details for the postgres superuser. + + Attributes + ---------- + secretName : str, default is Undefined, optional + K8s Secret holding the superuser credentials (CNPG-managed if `spec.superuser` was not set). + """ + + + secretName?: str + + diff --git a/tests/test-cluster/main.k b/tests/test-cluster/main.k index 0e3c8b6..4be7c63 100644 --- a/tests/test-cluster/main.k +++ b/tests/test-cluster/main.k @@ -10,12 +10,13 @@ import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 _items = [ # ========================================================================== - # Test 1: minimal claim renders Cluster + ExternalSecret with sensible - # defaults. Branching labels on by default, monitoring on, no plugins, no - # affinity, no custom storage class. + # Test 1: minimal claim renders only the Cluster — no ExternalSecret by + # default. Bootstrap wires the app role/database/secret with default names. + # Branching labels on by default, monitoring on, no plugins, no affinity, + # no custom storage class. (Consumer is in BYO mode for the app Secret.) # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "minimal-renders-cluster-and-secret" + metadata.name = "minimal-renders-cluster-only" spec = { compositionPath = "apis/psqlclusters/composition.yaml" xrdPath = "apis/psqlclusters/definition.yaml" @@ -52,31 +53,13 @@ _items = [ bootstrap.initdb = { database = "app" owner = "app" - secret.name = "test-app-superuser" + secret.name = "test-app-app" } storage.size = "10Gi" monitoring.enablePodMonitor = True } } } - { - apiVersion = "kubernetes.m.crossplane.io/v1alpha1" - kind = "Object" - metadata.name = "test-app-external-secret" - spec.forProvider.manifest = { - apiVersion = "external-secrets.io/v1beta1" - kind = "ExternalSecret" - metadata.name = "test-app-superuser" - spec = { - refreshInterval = "1h" - secretStoreRef = { - name = "hops-aws-secrets-manager" - kind = "ClusterSecretStore" - } - target.name = "test-app-superuser" - } - } - } ] } } @@ -257,29 +240,58 @@ _items = [ } # ========================================================================== - # Test 7: credentials.superuser.managedBy="" suppresses ExternalSecret. + # Test 7: app.externalSecret renders an ESO ExternalSecret pulling from + # the named ClusterSecretStore. Default behavior (Test 1) doesn't render + # one — this asserts the opt-in path. # ========================================================================== metav1alpha1.CompositionTest { - metadata.name = "no-managedby-skips-external-secret" + metadata.name = "external-secret-renders-when-opted-in" spec = { compositionPath = "apis/psqlclusters/composition.yaml" xrdPath = "apis/psqlclusters/definition.yaml" timeoutSeconds = 60 validate = False xr = stacksv1alpha1.PSQLCluster { - metadata = {name = "byo-secret", namespace = "default"} + metadata = {name = "es-app", namespace = "default"} spec = { clusterName = "my-cluster" storage = {size = "5Gi"} - credentials.superuser.managedBy = "" + app = { + externalSecret = { + secretStore = { + kind = "ClusterSecretStore" + name = "hops-aws-secrets-manager" + } + secretRef = { + path = "my-cluster/es-app" + } + } + } } } - # Cluster still composed; ExternalSecret absent. assertResources = [ { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" - metadata.name = "byo-secret-cnpg-cluster" + metadata.name = "es-app-cnpg-cluster" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "es-app-external-secret-app" + spec.forProvider.manifest = { + apiVersion = "external-secrets.io/v1beta1" + kind = "ExternalSecret" + metadata.name = "es-app-app" + spec = { + refreshInterval = "1h" + secretStoreRef = { + name = "hops-aws-secrets-manager" + kind = "ClusterSecretStore" + } + target.name = "es-app-app" + } + } } ] } diff --git a/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k b/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k index 9d113d6..123fca7 100644 --- a/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k +++ b/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k @@ -54,14 +54,14 @@ schema HopsOpsComAiV1alpha1PSQLClusterSpec: Attributes ---------- + app : HopsOpsComAiV1alpha1PSQLClusterSpecApp, default is Undefined, optional + app branching : HopsOpsComAiV1alpha1PSQLClusterSpecBranching, default is Undefined, optional branching clusterName : str, default is Undefined, required Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. cnpg : HopsOpsComAiV1alpha1PSQLClusterSpecCnpg, default is Undefined, optional cnpg - credentials : HopsOpsComAiV1alpha1PSQLClusterSpecCredentials, default is Undefined, optional - credentials crossplane : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane, default is Undefined, optional crossplane ha : HopsOpsComAiV1alpha1PSQLClusterSpecHa, default is Undefined, optional @@ -82,17 +82,19 @@ schema HopsOpsComAiV1alpha1PSQLClusterSpec: scale to zero storage : HopsOpsComAiV1alpha1PSQLClusterSpecStorage, default is Undefined, required storage + superuser : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser, default is Undefined, optional + superuser """ + app?: HopsOpsComAiV1alpha1PSQLClusterSpecApp + branching?: HopsOpsComAiV1alpha1PSQLClusterSpecBranching clusterName: str cnpg?: HopsOpsComAiV1alpha1PSQLClusterSpecCnpg - credentials?: HopsOpsComAiV1alpha1PSQLClusterSpecCredentials - crossplane?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane ha?: HopsOpsComAiV1alpha1PSQLClusterSpecHa @@ -113,77 +115,123 @@ schema HopsOpsComAiV1alpha1PSQLClusterSpec: storage: HopsOpsComAiV1alpha1PSQLClusterSpecStorage + superuser?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser -schema HopsOpsComAiV1alpha1PSQLClusterSpecBranching: + +schema HopsOpsComAiV1alpha1PSQLClusterSpecApp: r""" - Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). + Application user — the Postgres role your app connects as. CNPG uses these credentials for `bootstrap.initdb`. Attributes ---------- - enabled : bool, default is True, optional - enabled - snapshotClassName : str, default is "psql", optional - VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. + database : str, default is "app", optional + Application database name. Defaults to "app". + externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret, default is Undefined, optional + external secret + role : str, default is "app", optional + Postgres role name owning the application database. Defaults to "app". + secretName : str, default is Undefined, optional + K8s Secret holding the app role's credentials. Defaults to "-app". With `externalSecret` set, the composition writes this Secret; otherwise pre-create it yourself. """ - enabled?: bool = True + database?: str = "app" - snapshotClassName?: str = "psql" + externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret + role?: str = "app" -schema HopsOpsComAiV1alpha1PSQLClusterSpecCnpg: + secretName?: str = "" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret: r""" - Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. + Source the credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. Attributes ---------- - overrideAllValues : any, default is Undefined, optional - Replaces the entire Cluster.spec — total escape hatch. - values : any, default is Undefined, optional - Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. + secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef, default is Undefined, required + secret ref + secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore, default is Undefined, required + secret store """ - overrideAllValues?: any + secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef - values?: any + secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore -schema HopsOpsComAiV1alpha1PSQLClusterSpecCredentials: +schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef: r""" - Where Postgres credentials are sourced from. + Where in the secrets backend the credentials live. The remote value must be a JSON blob with `username` and `password` properties — both are extracted into the resulting K8s Secret. Attributes ---------- - superuser : HopsOpsComAiV1alpha1PSQLClusterSpecCredentialsSuperuser, default is Undefined, optional - superuser + path : str, default is Undefined, required + Path/key in the secrets backend (e.g. AWS Secrets Manager `my-cluster/app`). """ - superuser?: HopsOpsComAiV1alpha1PSQLClusterSpecCredentialsSuperuser + path: str -schema HopsOpsComAiV1alpha1PSQLClusterSpecCredentialsSuperuser: +schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore: r""" - hops ops com ai v1alpha1 p SQL cluster spec credentials superuser + Reference to the SecretStore or ClusterSecretStore providing the value. Attributes ---------- - managedBy : str, default is "hops-aws-secrets-manager", optional - ClusterSecretStore name for sourcing the superuser password from a secrets backend (default ESO ClusterSecretStore convention). Set "" to skip ExternalSecret composition (consumer must provide a K8s Secret named `` with `username` + `password` keys). - remoteKey : str, default is Undefined, optional - Key in the remote secrets backend (AWS Secrets Manager) holding the superuser password. Defaults to "/superuser". - secretName : str, default is Undefined, optional - K8s Secret name receiving the credentials. Defaults to "-superuser". + kind : str, default is "ClusterSecretStore", optional + Store kind. Use ClusterSecretStore for cluster-scoped stores (the platform default), SecretStore for namespaced stores in this XR's namespace. + name : str, default is Undefined, required + Name of the SecretStore or ClusterSecretStore. + namespace : str, default is Undefined, optional + Namespace of the SecretStore (ignored for ClusterSecretStore). Defaults to the PSQLCluster's namespace. """ - managedBy?: str = "hops-aws-secrets-manager" + kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" - remoteKey?: str = "" + name: str - secretName?: str = "" + namespace?: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecBranching: + r""" + Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). + + Attributes + ---------- + enabled : bool, default is True, optional + enabled + snapshotClassName : str, default is "psql", optional + VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. + """ + + + enabled?: bool = True + + snapshotClassName?: str = "psql" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecCnpg: + r""" + Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. + + Attributes + ---------- + overrideAllValues : any, default is Undefined, optional + Replaces the entire Cluster.spec — total escape hatch. + values : any, default is Undefined, optional + Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. + """ + + + overrideAllValues?: any + + values?: any schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane: @@ -406,12 +454,86 @@ schema HopsOpsComAiV1alpha1PSQLClusterSpecStorage: size: str +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser: + r""" + Optional postgres superuser credentials. Omit the block to let CNPG auto-generate the superuser secret. + + Attributes + ---------- + externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret, default is Undefined, optional + external secret + secretName : str, default is Undefined, optional + K8s Secret holding the superuser credentials. Defaults to "-superuser". + """ + + + externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret + + secretName?: str = "" + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret: + r""" + Source the superuser credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. + + Attributes + ---------- + secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef, default is Undefined, required + secret ref + secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore, default is Undefined, required + secret store + """ + + + secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef + + secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef: + r""" + hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret ref + + Attributes + ---------- + path : str, default is Undefined, required + path + """ + + + path: str + + +schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore: + r""" + hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret store + + Attributes + ---------- + kind : str, default is "ClusterSecretStore", optional + kind + name : str, default is Undefined, required + name + namespace : str, default is Undefined, optional + namespace + """ + + + kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" + + name: str + + namespace?: str + + schema HopsOpsComAiV1alpha1PSQLClusterStatus: r""" PSQLClusterStatus defines the observed state. Attributes ---------- + app : HopsOpsComAiV1alpha1PSQLClusterStatusApp, default is Undefined, optional + app clusterPhase : str, default is Undefined, optional Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). conditions : [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0], default is Undefined, optional @@ -420,9 +542,13 @@ schema HopsOpsComAiV1alpha1PSQLClusterStatus: crossplane ready : bool, default is Undefined, optional Overall readiness — true once the CNPG Cluster reaches `Cluster in healthy state` AND the ExternalSecret (if composed) is SecretSynced. + superuser : HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser, default is Undefined, optional + superuser """ + app?: HopsOpsComAiV1alpha1PSQLClusterStatusApp + clusterPhase?: str conditions?: [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0] @@ -431,6 +557,34 @@ schema HopsOpsComAiV1alpha1PSQLClusterStatus: ready?: bool + superuser?: HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser + + +schema HopsOpsComAiV1alpha1PSQLClusterStatusApp: + r""" + Connection details for the application user. Other XRs can reference this to wire connection strings without hardcoding. + + Attributes + ---------- + database : str, default is Undefined, optional + Application database name. + host : str, default is Undefined, optional + Service hostname for read-write connections (CNPG `-rw` service). + port : int, default is Undefined, optional + Postgres port. + secretName : str, default is Undefined, optional + K8s Secret holding the app role's credentials. + """ + + + database?: str + + host?: str + + port?: int + + secretName?: str + schema HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0: r""" @@ -494,3 +648,17 @@ schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails: lastPublishedTime?: str +schema HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser: + r""" + Connection details for the postgres superuser. + + Attributes + ---------- + secretName : str, default is Undefined, optional + K8s Secret holding the superuser credentials (CNPG-managed if `spec.superuser` was not set). + """ + + + secretName?: str + + From 5d8a0924b8ef38622550787a68c05d44090b67d2 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Wed, 29 Apr 2026 13:31:08 -0500 Subject: [PATCH 29/44] build(make): derive xrd/composition/api_dir per example for multi-API repos MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The render:all and validate:all targets were hardcoded to apis/psqlstacks via $(DEFINITION)/$(COMPOSITION)/$(XRD_DIR), so multi-API examples (psqlclusters/*, psqlbranches/*) were rendered against the wrong schema. Each example now resolves its own apis// dir from the example path (`examples//.yaml` → `apis/`) and uses that for both `up composition render --xrd=...` and `crossplane beta validate `. Verified locally: `make render` and `make validate` both clean across all five examples. From CodeRabbit review on PR #10. --- Makefile | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/Makefile b/Makefile index 12e9926..62fc7a9 100644 --- a/Makefile +++ b/Makefile @@ -37,14 +37,17 @@ render\:all: for entry in $(EXAMPLES); do \ example=$${entry%%::*}; \ observed=$${entry#*::}; \ + api_dir=$$(echo "$$example" | awk -F/ '{print "apis/" $$2}'); \ + composition="$$api_dir/composition.yaml"; \ + definition="$$api_dir/definition.yaml"; \ outfile="$$tmpdir/$$(echo $$entry | tr '/:' '__')"; \ ( \ if [ -n "$$observed" ]; then \ echo "=== Rendering $$example with observed-resources $$observed ==="; \ - up composition render --xrd=$(DEFINITION) $(COMPOSITION) $$example --observed-resources=$$observed; \ + up composition render --xrd=$$definition $$composition $$example --observed-resources=$$observed; \ else \ - echo "=== Rendering $$example ==="; \ - up composition render --xrd=$(DEFINITION) $(COMPOSITION) $$example; \ + echo "=== Rendering $$example (api=$$api_dir) ==="; \ + up composition render --xrd=$$definition $$composition $$example; \ fi; \ echo "" \ ) > "$$outfile" 2>&1 & \ @@ -67,18 +70,21 @@ validate\:all: for entry in $(EXAMPLES); do \ example=$${entry%%::*}; \ observed=$${entry#*::}; \ + api_dir=$$(echo "$$example" | awk -F/ '{print "apis/" $$2}'); \ + composition="$$api_dir/composition.yaml"; \ + definition="$$api_dir/definition.yaml"; \ outfile="$$tmpdir/$$(echo $$entry | tr '/:' '__')"; \ ( \ if [ -n "$$observed" ]; then \ echo "=== Validating $$example with observed-resources $$observed ==="; \ - up composition render --xrd=$(DEFINITION) $(COMPOSITION) $$example \ + up composition render --xrd=$$definition $$composition $$example \ --observed-resources=$$observed --include-full-xr --quiet | \ - crossplane beta validate $(XRD_DIR) --error-on-missing-schemas -; \ + crossplane beta validate $$api_dir --error-on-missing-schemas -; \ else \ - echo "=== Validating $$example ==="; \ - up composition render --xrd=$(DEFINITION) $(COMPOSITION) $$example \ + echo "=== Validating $$example (api=$$api_dir) ==="; \ + up composition render --xrd=$$definition $$composition $$example \ --include-full-xr --quiet | \ - crossplane beta validate $(XRD_DIR) --error-on-missing-schemas -; \ + crossplane beta validate $$api_dir --error-on-missing-schemas -; \ fi; \ echo "" \ ) > "$$outfile" 2>&1 & \ @@ -106,9 +112,9 @@ render\:%: validate\:%: @example="examples/psqlstacks/$*.yaml"; \ - up composition render --xrd=$(DEFINITION) $(COMPOSITION) $$example \ + up composition render --xrd=$$definition $$composition $$example \ --include-full-xr --quiet | \ - crossplane beta validate $(XRD_DIR) --error-on-missing-schemas - + crossplane beta validate $$api_dir --error-on-missing-schemas - test: up test run $(RENDER_TESTS) From b789ac7af06dc2322cea9b2e0e9c1a3cc280f4ca Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Thu, 30 Apr 2026 12:41:56 -0500 Subject: [PATCH 30/44] feat(psqlcluster): auto-gen app secret when neither field set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a third credential mode: omit `app.externalSecret` AND `app.secretName` and the composition omits `bootstrap.initdb.secret` from the CNPG Cluster CR so CNPG auto-generates and owns the basic-auth Secret at `-app`. The previous shape always set `bootstrap.initdb.secret.name = -app`, which forced CNPG to read a Secret that — in the no-ESO/no-BYO case — never existed, blocking bootstrap. The unified e2e hit this exact case (kind harness, no ESO ClusterSecretStore) and a stale `app.managedBy = ""` field was masking the real failure mode. Three modes now documented on the XRD: 1. Omit `app` → CNPG auto-generates `-app`. 2. Set `app.externalSecret` → ESO writes the Secret CNPG reads. 3. Set `app.secretName` (no externalSecret) → BYO; pre-create the Secret. Backwards compatible: existing manifests with externalSecret or explicit secretName render identically. CNPG will adopt a pre-existing `-app` if one is present. Tests: - test-cluster: minimal asserts the secret line is omitted; external-secret asserts secret.name is wired; new BYO test covers explicit secretName. - e2etest-psql: drops the bogus `app.managedBy` field that broke KCL parse; relies on the new auto-gen path. Implements [[tasks/merge-psql-client-apis-into-stack]] Co-Authored-By: Claude Opus 4.7 (1M context) --- apis/psqlclusters/definition.yaml | 20 +++++--- functions/cluster/000-state-init.yaml.gotmpl | 22 +++++--- .../cluster/200-cnpg-cluster.yaml.gotmpl | 19 ++++--- tests/e2etest-psql/main.k | 4 +- tests/test-cluster/main.k | 51 +++++++++++++++++-- 5 files changed, 90 insertions(+), 26 deletions(-) diff --git a/apis/psqlclusters/definition.yaml b/apis/psqlclusters/definition.yaml index cae3c7c..d4b7ddc 100644 --- a/apis/psqlclusters/definition.yaml +++ b/apis/psqlclusters/definition.yaml @@ -104,10 +104,18 @@ spec: # ============================================================ # Application user — the role your app connects as. # - # Common case: drop in `app.externalSecret` pointing at the - # platform's ClusterSecretStore and key. Or omit `externalSecret` - # entirely and pre-create the K8s Secret named `` - # with `username`+`password` keys (BYO mode). + # Three modes, in order of typical use: + # 1. Omit `app` (or set only role/database) — CNPG generates a + # basic-auth Secret named `-app` and owns + # its lifecycle. Zero-config; use for ephemeral or kind-only + # setups. + # 2. Set `app.externalSecret` — composition renders an ESO + # ExternalSecret pulling from the named (Cluster)SecretStore + # and writes the K8s Secret CNPG reads. Common production + # shape on platforms with External Secrets Operator wired up. + # 3. Set `app.secretName` (without `externalSecret`) — BYO; you + # pre-create the named K8s Secret with `username`+`password` + # keys before the cluster bootstraps. # ============================================================ app: description: Application user — the Postgres role your app connects as. CNPG uses these credentials for `bootstrap.initdb`. @@ -122,11 +130,11 @@ spec: type: string default: app secretName: - description: K8s Secret holding the app role's credentials. Defaults to "-app". With `externalSecret` set, the composition writes this Secret; otherwise pre-create it yourself. + description: K8s Secret holding the app role's credentials. Setting this opts into BYO mode — pre-create the Secret with `username`+`password` keys. Leave unset (and omit `externalSecret`) to let CNPG auto-generate `-app`. type: string default: "" externalSecret: - description: Source the credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. + description: Source the credentials via External Secrets Operator. Omit to either BYO the K8s Secret named `secretName`, or — when `secretName` is also unset — let CNPG auto-generate. type: object properties: secretStore: diff --git a/functions/cluster/000-state-init.yaml.gotmpl b/functions/cluster/000-state-init.yaml.gotmpl index b8a78f7..b8ba233 100644 --- a/functions/cluster/000-state-init.yaml.gotmpl +++ b/functions/cluster/000-state-init.yaml.gotmpl @@ -77,23 +77,33 @@ # ============================================================================== # Application user — always present. Wires CNPG's bootstrap.initdb shape. -# `externalSecret` is optional: when set, the composition renders an ESO -# ExternalSecret pulling from the named (Cluster)SecretStore. When omitted, -# the consumer is expected to pre-create the K8s Secret at `secretName` -# (BYO mode). +# Three modes: +# - Neither `externalSecret` nor `secretName` set → CNPG auto-generates +# a Secret named `-app` and owns its lifecycle. Composition +# omits `bootstrap.initdb.secret` so CNPG fills it in. `cnpgManagedSecret` +# flag drives the omission downstream. +# - `externalSecret` set → composition renders an ESO ExternalSecret that +# writes the K8s Secret CNPG reads. +# - `secretName` set (without `externalSecret`) → BYO; consumer pre-creates +# the named Secret with `username`+`password`. # ============================================================================== {{- $appSpec := $spec.app | default dict }} {{- $appRole := $appSpec.role | default "app" }} {{- $appDatabase := $appSpec.database | default "app" }} -{{- $appSecretName := $appSpec.secretName }} +{{- $appExternalSecret := $appSpec.externalSecret | default dict }} +{{- $appExternalSecretSet := and $appExternalSecret.secretStore $appExternalSecret.secretRef }} +{{- $appSecretNameRaw := $appSpec.secretName }} +{{- $appSecretName := $appSecretNameRaw }} {{- if not $appSecretName }} {{- $appSecretName = printf "%s-app" $name }} {{- end }} +{{- $appCNPGManagedSecret := and (not $appSecretNameRaw) (not $appExternalSecretSet) }} {{- $app := dict "role" $appRole "database" $appDatabase "secretName" $appSecretName - "externalSecret" ($appSpec.externalSecret | default dict) + "cnpgManagedSecret" $appCNPGManagedSecret + "externalSecret" $appExternalSecret }} # ============================================================================== diff --git a/functions/cluster/200-cnpg-cluster.yaml.gotmpl b/functions/cluster/200-cnpg-cluster.yaml.gotmpl index abf8c45..1769e03 100644 --- a/functions/cluster/200-cnpg-cluster.yaml.gotmpl +++ b/functions/cluster/200-cnpg-cluster.yaml.gotmpl @@ -61,16 +61,21 @@ spec: spec: {{- /* Build the Cluster.spec from intent-first state, then mergeOverwrite with $cnpg.values for fine-grained passthrough overrides. */}} + {{- /* When neither app.externalSecret nor an explicit app.secretName + is set, omit bootstrap.initdb.secret so CNPG auto-generates and + owns a basic-auth Secret at -app. Otherwise point + CNPG at the secret the consumer or ESO is providing. */}} + {{- $initdb := dict + "database" $state.app.database + "owner" $state.app.role + }} + {{- if not $state.app.cnpgManagedSecret }} + {{- $_ := set $initdb "secret" (dict "name" $state.app.secretName) }} + {{- end }} {{- $clusterSpec := dict "instances" $state.instances "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) - "bootstrap" (dict - "initdb" (dict - "database" $state.app.database - "owner" $state.app.role - "secret" (dict "name" $state.app.secretName) - ) - ) + "bootstrap" (dict "initdb" $initdb) "storage" (dict "size" $state.storage.size ) diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index e9d8f86..6757aec 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -186,8 +186,8 @@ _items = [ kind = "ProviderConfig" } storage = {size = "1Gi"} - # Skip ExternalSecret in e2e — no ESO ClusterSecretStore in kind harness. - app.managedBy = "" + # Omit `app` entirely — no ESO in kind, no pre-created + # secret. CNPG auto-generates `-app`. } } # 3. PSQLBranch — ephemeral fork. Snapshots the source diff --git a/tests/test-cluster/main.k b/tests/test-cluster/main.k index 4be7c63..8369e1c 100644 --- a/tests/test-cluster/main.k +++ b/tests/test-cluster/main.k @@ -11,9 +11,10 @@ import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 _items = [ # ========================================================================== # Test 1: minimal claim renders only the Cluster — no ExternalSecret by - # default. Bootstrap wires the app role/database/secret with default names. - # Branching labels on by default, monitoring on, no plugins, no affinity, - # no custom storage class. (Consumer is in BYO mode for the app Secret.) + # default. Bootstrap wires the app role/database; `secret` is omitted so + # CNPG auto-generates `-app` and owns it. Branching labels + # on by default, monitoring on, no plugins, no affinity, no custom storage + # class. # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "minimal-renders-cluster-only" @@ -53,7 +54,6 @@ _items = [ bootstrap.initdb = { database = "app" owner = "app" - secret.name = "test-app-app" } storage.size = "10Gi" monitoring.enablePodMonitor = True @@ -274,6 +274,11 @@ _items = [ apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" metadata.name = "es-app-cnpg-cluster" + spec.forProvider.manifest.spec.bootstrap.initdb = { + database = "app" + owner = "app" + secret.name = "es-app-app" + } } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" @@ -335,7 +340,43 @@ _items = [ } # ========================================================================== - # Test 9: kubernetesProviderConfigRef defaults to clusterName. + # Test 9: BYO mode — explicit `app.secretName` (no externalSecret) wires + # CNPG at the named Secret. Consumer is responsible for pre-creating it. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "byo-secret-name-wires-cnpg-at-named-secret" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "byo-app", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "5Gi"} + app = { + secretName = "my-precreated-secret" + } + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "byo-app-cnpg-cluster" + spec.forProvider.manifest.spec.bootstrap.initdb = { + database = "app" + owner = "app" + secret.name = "my-precreated-secret" + } + } + ] + } + } + + # ========================================================================== + # Test 10: kubernetesProviderConfigRef defaults to clusterName. # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "k8s-provider-config-defaults-to-cluster-name" From 990ddfd3a1ef159c352bd8098a11fab0e8847869 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Thu, 7 May 2026 17:28:26 -0500 Subject: [PATCH 31/44] feat(psqlstack)!: compose paired psql StorageClass + pivot e2e to AWS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PSQLStack now composes a `psql` StorageClass alongside its existing `psql` VolumeSnapshotClass — they share the same CSI driver (`ebs.csi.eks.amazonaws.com` by default) since snapshots only work when the snapshotter driver matches the source PVC's provisioner. PSQLCluster + PSQLBranch default `spec.storage.class` to "psql", so consumer manifests stop leaking driver-specific knowledge. Default StorageClass shape: gp3 + WaitForFirstConsumer (correct for zonal CSI drivers — late-binds the PVC to a node so EBS volumes land in the same AZ as the consuming pod) + allowVolumeExpansion=true (CNPG resizes via the same field on its Cluster CR). E2E pivots from kind to an ephemeral EKS Auto Mode cluster (mirror of aws-observe-stack): provisions an AutoEKSCluster per run, installs volume-snapshot-stack on it, then runs PSQLStack/Cluster/Branch against the real `ebs.csi.eks.amazonaws.com` driver — same code path that runs on pat-local. Kind has no snapshot-capable CSI driver natively, so the prior kind-only e2e couldn't exercise PSQLBranch's snapshot/fork chain. Verified end-to-end on pat-local: - PSQLStack composed `psql` SC + `psql` VSC (both with ebs.csi.eks.amazonaws.com) - PSQLCluster PVC bound on `psql` SC, CNPG primary running - PSQLBranch VolumeSnapshot reached readyToUse=true, restored PVC bound on `psql` SC, CNPG fork primary running Breaking: PSQLStack adds a new composed StorageClass by default. Sites that already have a `psql` SC will conflict — set `spec.storageClass.enabled: false` to opt out. Requires three GitHub Actions vars on the repo (synced via the new `hops vars sync github`): ADMIN_ROLE_ARN, PRIVATE_SUBNET_ID_A, PRIVATE_SUBNET_ID_B. --- .github/workflows/on-pr.yaml | 42 ++- .gitignore | 3 +- README.md | 51 ++- apis/psqlbranches/definition.yaml | 6 +- apis/psqlclusters/definition.yaml | 6 +- apis/psqlstacks/definition.yaml | 89 +++++- functions/branch/000-state-init.yaml.gotmpl | 2 +- functions/cluster/000-state-init.yaml.gotmpl | 2 +- functions/stack/000-state-init.yaml.gotmpl | 35 +- functions/stack/010-state-status.yaml.gotmpl | 3 +- functions/stack/180-storageclass.yaml.gotmpl | 49 +++ tests/e2etest-psql/main.k | 319 ++++++++++++------- tests/test-stack/main.k | 113 ++++++- 13 files changed, 560 insertions(+), 160 deletions(-) create mode 100644 functions/stack/180-storageclass.yaml.gotmpl diff --git a/.github/workflows/on-pr.yaml b/.github/workflows/on-pr.yaml index 83568b2..5a20ecf 100644 --- a/.github/workflows/on-pr.yaml +++ b/.github/workflows/on-pr.yaml @@ -45,13 +45,45 @@ jobs: e2e: uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@feat/multi-api-support with: - # Long chain: snapshot-controller install → PSQLStack Helm installs - # (CNPG + atlas + VolumeSnapshotClass) → PSQLCluster bootstraps a real - # Postgres → PSQLBranch snapshots + restores. With cold image pulls and - # the occasional rate-limit slowness on the kind node, 90 min is what - # this chain needs to converge reliably. + # E2E provisions a real EKS Auto Mode cluster (mirror of aws-observe-stack) + # so the `psql` StorageClass + VolumeSnapshotClass land on a real + # ebs.csi.eks.amazonaws.com driver — same code path that runs on + # pat-local. Kind has no snapshot-capable CSI, so the prior kind-only + # e2e couldn't exercise PSQLBranch's snapshot/fork chain. + aws: true + aws-use-oidc: true + aws-account-id: "034489662075" + aws-region: us-east-2 + aws-role-duration-seconds: 7200 + write-env-files: true + env-vars: | + { + "ADMIN_ROLE_ARN": "${{ vars.ADMIN_ROLE_ARN }}", + "PRIVATE_SUBNET_ID_A": "${{ vars.PRIVATE_SUBNET_ID_A }}", + "PRIVATE_SUBNET_ID_B": "${{ vars.PRIVATE_SUBNET_ID_B }}" + } + debug-resource-types: | + [ + "psqlstacks.hops.ops.com.ai", + "psqlclusters.hops.ops.com.ai", + "psqlbranches.hops.ops.com.ai", + "volumesnapshotstacks.hops.ops.com.ai", + "autoeksclusters.aws.hops.ops.com.ai" + ] + # AutoEKSCluster outlives the manifests' cascade delete; the workflow + # tears it down separately. VolumeSnapshotStack is colocated with the + # cluster — drop it together so the Helm release on the EKS cluster + # is uninstalled before the cluster goes. + delete-extra-resources: | + [ + "volumesnapshotstacks.hops.ops.com.ai", + "autoeksclusters.aws.hops.ops.com.ai" + ] + # 90 min: ~10–15 min EKS Auto Mode cold start, then platform install + + # CNPG bootstrap + snapshot fork. timeout-minutes: 90 cleanup-timeout-minutes: 30 + delete-extra-resources-timeout-minutes: 30 publish: needs: diff --git a/.gitignore b/.gitignore index 1ef5a9e..923dadd 100644 --- a/.gitignore +++ b/.gitignore @@ -8,7 +8,8 @@ _output/ # Temporary files .tmp/ -# E2E test credentials (never commit secrets) +# E2E test credentials + per-run env (never commit secrets) **/aws-creds tests/**/secrets/ +tests/**/env/ apis/**/configuration.yaml diff --git a/README.md b/README.md index 5706272..9e3a631 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,13 @@ # psql-stack -PostgreSQL platform layer on top of [CloudNativePG](https://cloudnative-pg.io/). Composes the CNPG operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), the [Atlas Operator](https://atlasgo.io/integrations/kubernetes/operator) (declarative schema migrations), and a `VolumeSnapshotClass` that PSQLBranch uses as a stable forking target. +PostgreSQL platform layer on top of [CloudNativePG](https://cloudnative-pg.io/). Composes the CNPG operator, the [`cnpg-i-scale-to-zero` plugin](https://github.com/xataio/cnpg-i-scale-to-zero), the [Atlas Operator](https://atlasgo.io/integrations/kubernetes/operator) (declarative schema migrations), a paired `psql` `StorageClass` + `VolumeSnapshotClass` that PSQLClusters and PSQLBranches default to. This is the **platform layer** — it does not create any serving Postgres clusters. Per-app DBs live in [`PSQLCluster`](../../psql-cluster/), ephemeral forks in [`PSQLBranch`](../../psql-branch/). ## Design -The stack is intentionally **OS-agnostic and storage-agnostic**: - -- **No StorageClass.** PSQLClusters target whatever StorageClass the cluster already provides (`gp3` on EKS Auto Mode, `standard` on kind/k3d, etc.). +- **Paired StorageClass + VolumeSnapshotClass, both named `psql`.** Snapshots only work when the snapshotter driver matches the underlying StorageClass provisioner, so the stack composes both with the same CSI driver value. PSQLClusters and PSQLBranches default `spec.storage.class: psql` (XRD default), so consumer manifests don't have to know the driver. Default driver/provisioner is `ebs.csi.eks.amazonaws.com` (EKS Auto Mode); override `storageClass.provisioner` + `snapshotClass.driver` together for non-EKS targets (kind, self-managed Longhorn, etc.). - **No NodePool / node-prep.** Components run wherever the cluster's scheduler puts them. Auto Mode handles node provisioning end-to-end. -- **Single composed snapshot target.** The stack ships a `VolumeSnapshotClass` named `psql` so PSQLBranch can request snapshots without leaking driver-specific knowledge into PSQLBranch's spec. Default driver is `ebs.csi.eks.amazonaws.com` (EKS Auto Mode's managed EBS CSI driver); override for non-AWS clusters or self-managed EBS. If you need replicated CoW storage (true block-level branches with delta-only economics), that's a separate concern — see `aws-storage-stack` for self-managed nodes that can host Longhorn or similar. The default psql-stack stays on the AWS-blessed Auto Mode path. @@ -21,12 +18,13 @@ If you need replicated CoW storage (true block-level branches with delta-only ec | **CNPG operator** | always-on | The CNCF Postgres operator. CRDs include `Cluster`, `Backup`, `Pooler`, `ScheduledBackup`. | | **cnpg-i-scale-to-zero plugin** | on (`spec.scaleToZeroPlugin.enabled: true`) | Auto-hibernates idle CNPG `Cluster`s. **Requires cert-manager** (provided by [`aws-cert-stack`](../../aws/cert/)). | | **Atlas operator** | always-on | Declarative schema migrations via `AtlasMigration` / `AtlasSchema` CRDs. | -| **VolumeSnapshotClass** | on (`spec.snapshotClass.enabled: true`) | Named `psql` by default. Driver: `ebs.csi.eks.amazonaws.com`. PSQLBranch references this name. | +| **StorageClass** | on (`spec.storageClass.enabled: true`) | Named `psql` by default. Provisioner: `ebs.csi.eks.amazonaws.com` with `type: gp3`. PSQLCluster + PSQLBranch reference it as their default `spec.storage.class`. | +| **VolumeSnapshotClass** | on (`spec.snapshotClass.enabled: true`) | Named `psql` by default. Driver matches `storageClass.provisioner`. PSQLBranch references it for snapshot/fork. | | **HA mode** | off (`spec.ha.enabled: false`) | When enabled: 3 replicas + `topologySpreadConstraints` by zone on CNPG, Atlas, S2Z plugin. | ## Prerequisites -- **A working CSI driver + StorageClass** on the cluster. EKS Auto Mode provides `ebs.csi.eks.amazonaws.com` automatically. For kind/k3d, the bundled `standard` SC works. +- **A working CSI driver** on the cluster matching `storageClass.provisioner`. EKS Auto Mode provides `ebs.csi.eks.amazonaws.com` automatically. For other targets, override `storageClass.provisioner` (and the matching `snapshotClass.driver`) to whatever the target cluster has — e.g. `hostpath.csi.k8s.io` for kind, `driver.longhorn.io` for self-managed Longhorn. - **VolumeSnapshot CRDs + snapshot-controller** (`snapshot.storage.k8s.io`). EKS Auto Mode ships the snapshot CRDs but **not** the cluster-wide snapshot-controller. Without one, the composed VolumeSnapshotClass is inert and PSQLBranch snapshots will never reach `ReadyToUse`. Install [`volume-snapshot-stack`](../volume-snapshot/) (also in this org) — it composes the upstream snapshot-controller via the piraeus-charts Helm chart and is the canonical CRD installer for the cluster. - **cert-manager** (only when `scaleToZeroPlugin.enabled` — the plugin uses cert-manager Issuer+Certificate for its gRPC TLS). Provided by [`aws-cert-stack`](../../aws/cert/). @@ -70,9 +68,9 @@ spec: prewarmDevDB: true ``` -### Stage 3: Non-AWS / non-EBS cluster +### Stage 3: Non-EKS cluster -Override the snapshot driver (e.g. for a self-managed cluster running Longhorn, or a local cluster using hostpath CSI). +Override the SC + VSC driver together (they must match for snapshots to work). Example: self-managed cluster running Longhorn. ```yaml apiVersion: hops.ops.com.ai/v1alpha1 @@ -84,11 +82,33 @@ spec: clusterName: edge helmProviderConfigRef: name: default + storageClass: + provisioner: driver.longhorn.io + parameters: + numberOfReplicas: "3" snapshotClass: driver: driver.longhorn.io ``` -### Stage 4: Local / no-snapshot cluster +### Stage 4: Opt out of the composed StorageClass + +If the cluster already ships a suitable default StorageClass, disable composition and have PSQLCluster/PSQLBranch consumers set `spec.storage.class` explicitly. + +```yaml +apiVersion: hops.ops.com.ai/v1alpha1 +kind: PSQLStack +metadata: + name: psql + namespace: default +spec: + clusterName: shared-cluster + helmProviderConfigRef: + name: default + storageClass: + enabled: false +``` + +### Stage 5: Local / no-snapshot cluster For dev clusters without a snapshot-controller, disable the VSC composition. PSQLBranch won't function (it needs the VSC), but PSQLCluster still works. @@ -138,10 +158,18 @@ spec: | `atlasOperator.namespace` | string | shared `namespace` | Override | | `atlasOperator.values` | object | — | Helm values merged with chart defaults | | `atlasOperator.overrideAllValues` | object | — | Helm values that replace all defaults | +| **Storage class** | | | | +| `storageClass.enabled` | bool | `true` | Compose the StorageClass | +| `storageClass.name` | string | `psql` | StorageClass name (PSQLCluster + PSQLBranch reference this) | +| `storageClass.provisioner` | string | `ebs.csi.eks.amazonaws.com` | CSI driver name. Must match `snapshotClass.driver` | +| `storageClass.reclaimPolicy` | enum | `Delete` | `Delete` or `Retain` | +| `storageClass.volumeBindingMode` | enum | `WaitForFirstConsumer` | `Immediate` or `WaitForFirstConsumer` | +| `storageClass.allowVolumeExpansion` | bool | `true` | Online PVC expansion (CNPG resizes via the same field on its `Cluster` CR) | +| `storageClass.parameters` | object | `{type: gp3}` | Provisioner-specific parameters | | **Snapshot class** | | | | | `snapshotClass.enabled` | bool | `true` | Compose the VolumeSnapshotClass | | `snapshotClass.name` | string | `psql` | VolumeSnapshotClass name (PSQLBranch references this) | -| `snapshotClass.driver` | string | `ebs.csi.eks.amazonaws.com` | CSI driver (Auto Mode default; override for self-managed EBS or non-AWS) | +| `snapshotClass.driver` | string | `ebs.csi.eks.amazonaws.com` | CSI driver. Must match `storageClass.provisioner` | | `snapshotClass.deletionPolicy` | enum | `Delete` | `Delete` or `Retain` | | `snapshotClass.parameters` | object | — | Driver-specific parameters | @@ -152,6 +180,7 @@ spec: | `cloudnative-pg` | `helm.m.crossplane.io/Release` | always | | `atlas-operator` | `helm.m.crossplane.io/Release` | always | | 9× `-s2z-*` | `kubernetes.m.crossplane.io/Object` | `scaleToZeroPlugin.enabled: true` | +| `-storageclass` | `kubernetes.m.crossplane.io/Object` | `storageClass.enabled: true` | | `-volumesnapshotclass` | `kubernetes.m.crossplane.io/Object` | `snapshotClass.enabled: true` | | Various `Usage` | `protection.crossplane.io/Usage` | when both ends Ready (deletion ordering) | diff --git a/apis/psqlbranches/definition.yaml b/apis/psqlbranches/definition.yaml index a62e024..2989a79 100644 --- a/apis/psqlbranches/definition.yaml +++ b/apis/psqlbranches/definition.yaml @@ -102,7 +102,7 @@ spec: type: integer default: 1 storage: - description: Branch PVC sizing. Empty fields inherit from the source's storage on recovery. + description: Branch PVC sizing. `size` must be ≥ the source PVC size for snapshot recovery to succeed. type: object properties: size: @@ -110,9 +110,9 @@ spec: type: string default: "" class: - description: Branch StorageClass. Empty = inherit from source. + description: Branch StorageClass. Defaults to "psql" — the StorageClass composed by PSQLStack. Override only when forking onto a different driver than the source. type: string - default: "" + default: psql postgresql: description: Postgres version on the branch (must match source for snapshot recovery to succeed). diff --git a/apis/psqlclusters/definition.yaml b/apis/psqlclusters/definition.yaml index d4b7ddc..a82cc85 100644 --- a/apis/psqlclusters/definition.yaml +++ b/apis/psqlclusters/definition.yaml @@ -70,16 +70,16 @@ spec: type: integer default: 1 storage: - description: Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (gp3 default on EKS Auto Mode) — bumping `size` upward grows the volume in place with no downtime. + description: Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (the stack's `psql` SC sets this) — bumping `size` upward grows the volume in place with no downtime. type: object properties: size: description: PVC size, e.g. "10Gi". Required. type: string class: - description: StorageClass name. Empty = cluster default (gp3 on Auto Mode). + description: StorageClass name. Defaults to "psql" — the StorageClass composed by PSQLStack. Override only to opt out of the stack's paired SC + VSC (e.g. for a different driver on the same cluster). type: string - default: "" + default: psql required: - size diff --git a/apis/psqlstacks/definition.yaml b/apis/psqlstacks/definition.yaml index b52f80c..5799467 100644 --- a/apis/psqlstacks/definition.yaml +++ b/apis/psqlstacks/definition.yaml @@ -17,13 +17,18 @@ spec: description: | PSQLStack is the platform layer for Postgres-on-Kubernetes. It composes CloudNativePG (the operator), the cnpg-i-scale-to-zero plugin, the Atlas - Operator (declarative schema migrations), and a VolumeSnapshotClass that - PSQLBranch uses as a stable target for forking primaries. + Operator (declarative schema migrations), a `psql` StorageClass that + PSQLClusters and PSQLBranches default to, and a `psql` VolumeSnapshotClass + that PSQLBranch uses as a stable target for forking primaries. - The stack is intentionally OS-agnostic and storage-agnostic — it relies - on whatever CSI driver and StorageClass the target cluster already - provides (e.g., gp3 on EKS Auto Mode, standard on kind/k3d). Per-app - serving clusters live in PSQLCluster; ephemeral forks in PSQLBranch. + The StorageClass + VolumeSnapshotClass are paired by design: snapshots + can only be taken of PVCs whose CSI driver matches the snapshot driver, + so the stack composes both with the same driver and exposes them under + the shared name `psql`. Default driver is `ebs.csi.eks.amazonaws.com` + (EKS Auto Mode); override `storageClass.provisioner` + `snapshotClass.driver` + together for non-EKS targets (e.g. `hostpath.csi.k8s.io` for kind tests, + `driver.longhorn.io` for self-managed). Per-app serving clusters live in + PSQLCluster; ephemeral forks in PSQLBranch. type: object properties: spec: @@ -144,17 +149,67 @@ spec: description: Helm values that replace all defaults for this component. type: object x-kubernetes-preserve-unknown-fields: true + storageClass: + description: | + StorageClass that PSQLCluster and PSQLBranch default their PVCs to. + Pairs with `snapshotClass` — VolumeSnapshot only works when the + snapshotter driver matches the StorageClass provisioner, so the + stack composes both with the same CSI driver under the shared + name "psql". Default provisioner is `ebs.csi.eks.amazonaws.com` + with `type: gp3` parameters (EKS Auto Mode's managed driver). + For non-EKS targets, override `provisioner` + `parameters` to + match (e.g. `hostpath.csi.k8s.io` with empty parameters for kind, + `driver.longhorn.io` for self-managed Longhorn). + type: object + properties: + enabled: + description: Compose the StorageClass. Defaults to true. + type: boolean + default: true + name: + description: StorageClass name. Defaults to "psql" — PSQLCluster + PSQLBranch reference this name. + type: string + default: psql + provisioner: + description: CSI driver name. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode). Must match snapshotClass.driver for VolumeSnapshot to work. + type: string + default: ebs.csi.eks.amazonaws.com + reclaimPolicy: + description: Reclaim policy. Defaults to Delete. + type: string + enum: + - Delete + - Retain + default: Delete + volumeBindingMode: + description: Volume binding mode. Defaults to WaitForFirstConsumer (matches EKS Auto Mode's gp3 default and is correct for zonal CSI drivers). + type: string + enum: + - Immediate + - WaitForFirstConsumer + default: WaitForFirstConsumer + allowVolumeExpansion: + description: Allow online PVC expansion. Defaults to true (CNPG resizes via the same field on the Cluster CR). + type: boolean + default: true + parameters: + description: | + Provisioner-specific parameters. Defaults to `{type: gp3}` for + the EBS provisioner. When overriding `provisioner`, set this + to whatever the new provisioner expects (e.g. `{}` for + hostpath.csi.k8s.io, driver-specific keys for Longhorn). + type: object + additionalProperties: + type: string + x-kubernetes-preserve-unknown-fields: true snapshotClass: description: | - VolumeSnapshotClass used by PSQLBranch as its forking target. The - stack composes exactly one VolumeSnapshotClass with `name` defaulting - to "psql" (PSQLBranch references it by name). The default driver is - `ebs.csi.eks.amazonaws.com` (EKS Auto Mode's managed EBS CSI driver); - override `driver` for other providers (e.g. `ebs.csi.aws.com` for - self-managed EBS, `driver.longhorn.io`, `hostpath.csi.k8s.io`). - PSQLClusters target whatever StorageClass the cluster already - provides (Auto Mode ships gp3 by default) — the stack does not - compose a StorageClass. + VolumeSnapshotClass that PSQLBranch references when forking a + PSQLCluster's PVC. Pairs with `storageClass`: the driver must + match the StorageClass provisioner. Default driver is + `ebs.csi.eks.amazonaws.com` (EKS Auto Mode); override for + other CSI providers (`hostpath.csi.k8s.io` on kind, + `driver.longhorn.io` for self-managed Longhorn, etc.). type: object properties: enabled: @@ -166,7 +221,7 @@ spec: type: string default: psql driver: - description: CSI driver. Defaults to ebs.csi.eks.amazonaws.com (EKS Auto Mode's managed EBS CSI driver). For self-managed EBS use ebs.csi.aws.com; for non-AWS providers, use the appropriate driver name. + description: CSI driver. Defaults to ebs.csi.eks.amazonaws.com. Must match storageClass.provisioner. type: string default: ebs.csi.eks.amazonaws.com deletionPolicy: @@ -189,7 +244,7 @@ spec: type: object properties: ready: - description: Overall readiness — true once every enabled component is Ready (CNPG, Atlas, the scale-to-zero plugin if `scaleToZeroPlugin.enabled`, and the VolumeSnapshotClass if `snapshotClass.enabled`). Disabled components are treated as Ready. + description: Overall readiness — true once every enabled component is Ready (CNPG, Atlas, the scale-to-zero plugin if `scaleToZeroPlugin.enabled`, the StorageClass if `storageClass.enabled`, and the VolumeSnapshotClass if `snapshotClass.enabled`). Disabled components are treated as Ready. type: boolean required: - spec diff --git a/functions/branch/000-state-init.yaml.gotmpl b/functions/branch/000-state-init.yaml.gotmpl index dc7d545..5d7d2d7 100644 --- a/functions/branch/000-state-init.yaml.gotmpl +++ b/functions/branch/000-state-init.yaml.gotmpl @@ -69,7 +69,7 @@ "instances" ($branchSpec.instances | default 1) "storage" (dict "size" $branchSize - "class" ($branchStorageSpec.class | default "") + "class" ($branchStorageSpec.class | default "psql") ) }} diff --git a/functions/cluster/000-state-init.yaml.gotmpl b/functions/cluster/000-state-init.yaml.gotmpl index b8ba233..d2dc67b 100644 --- a/functions/cluster/000-state-init.yaml.gotmpl +++ b/functions/cluster/000-state-init.yaml.gotmpl @@ -63,7 +63,7 @@ {{- $storageSpec := $spec.storage }} {{- $storage := dict "size" $storageSpec.size - "class" ($storageSpec.class | default "") + "class" ($storageSpec.class | default "psql") }} # ============================================================================== diff --git a/functions/stack/000-state-init.yaml.gotmpl b/functions/stack/000-state-init.yaml.gotmpl index 96bbfe0..fa61e1f 100644 --- a/functions/stack/000-state-init.yaml.gotmpl +++ b/functions/stack/000-state-init.yaml.gotmpl @@ -68,12 +68,36 @@ {{- $atlas := $spec.atlasOperator | default dict }} # ============================================================================== -# VolumeSnapshotClass — the stack's only storage-layer composition. -# PSQLBranch targets this VSC by name when forking primaries. Default driver -# is ebs.csi.eks.amazonaws.com (EKS Auto Mode's managed EBS CSI driver); -# override for other CSI providers via $spec.snapshotClass.driver -# (e.g. ebs.csi.aws.com for self-managed EBS, driver.longhorn.io). +# StorageClass + VolumeSnapshotClass — paired storage-layer composition. +# +# PSQLCluster + PSQLBranch default their PVCs to the composed StorageClass; +# PSQLBranch references the VolumeSnapshotClass when forking primaries. The +# two CSI drivers MUST match (snapshotter only operates on PVCs whose +# provisioner matches its driver), so they default to the same value +# (ebs.csi.eks.amazonaws.com — EKS Auto Mode's managed EBS CSI driver). +# Override both together for non-EKS targets (hostpath.csi.k8s.io for kind, +# driver.longhorn.io for self-managed Longhorn, etc.). # ============================================================================== +{{- $storageSpec := $spec.storageClass | default dict }} +{{- $storageEnabled := true }} +{{- if hasKey $storageSpec "enabled" }} + {{- $storageEnabled = $storageSpec.enabled }} +{{- end }} +{{- $storageAllowExpansion := true }} +{{- if hasKey $storageSpec "allowVolumeExpansion" }} + {{- $storageAllowExpansion = $storageSpec.allowVolumeExpansion }} +{{- end }} +{{- $defaultStorageParams := dict "type" "gp3" }} +{{- $storageClass := dict + "enabled" $storageEnabled + "name" ($storageSpec.name | default "psql") + "provisioner" ($storageSpec.provisioner | default "ebs.csi.eks.amazonaws.com") + "reclaimPolicy" ($storageSpec.reclaimPolicy | default "Delete") + "volumeBindingMode" ($storageSpec.volumeBindingMode | default "WaitForFirstConsumer") + "allowVolumeExpansion" $storageAllowExpansion + "parameters" ($storageSpec.parameters | default $defaultStorageParams) +}} + {{- $snapshotSpec := $spec.snapshotClass | default dict }} {{- $snapshotEnabled := true }} {{- if hasKey $snapshotSpec "enabled" }} @@ -117,6 +141,7 @@ "values" ($atlas.values | default dict) "overrideAllValues" ($atlas.overrideAllValues | default dict) ) + "storageClass" $storageClass "snapshotClass" $snapshotClass "observed" (dict) "status" (dict) diff --git a/functions/stack/010-state-status.yaml.gotmpl b/functions/stack/010-state-status.yaml.gotmpl index a1b2a9b..1d4b1e1 100644 --- a/functions/stack/010-state-status.yaml.gotmpl +++ b/functions/stack/010-state-status.yaml.gotmpl @@ -10,7 +10,7 @@ # ============================================================================== {{- $checkReady := dict }} -{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "volumesnapshotclass" }} +{{- range $key := list "cnpg-operator" "cnpg-scale-to-zero" "atlas-operator" "storageclass" "volumesnapshotclass" }} {{- $entry := get $observed $key | default dict }} {{- $resource := $entry.resource | default dict }} {{- $status := $resource.status | default dict }} @@ -30,6 +30,7 @@ "cnpg" (dict "ready" (get $checkReady "cnpg-operator")) "scaleToZeroPlugin" (dict "ready" (get $checkReady "cnpg-scale-to-zero")) "atlasOperator" (dict "ready" (get $checkReady "atlas-operator")) + "storageClass" (dict "ready" (get $checkReady "storageclass")) "snapshotClass" (dict "ready" (get $checkReady "volumesnapshotclass")) ) }} diff --git a/functions/stack/180-storageclass.yaml.gotmpl b/functions/stack/180-storageclass.yaml.gotmpl new file mode 100644 index 0000000..c377e37 --- /dev/null +++ b/functions/stack/180-storageclass.yaml.gotmpl @@ -0,0 +1,49 @@ +# code: language=yaml +# +# StorageClass — paired with the VolumeSnapshotClass composed in 170-. +# +# PSQLCluster's PVC and PSQLBranch's restored PVC both default to this class +# (XRD default: "psql"). Snapshots only work when the snapshotter driver +# matches the StorageClass provisioner, so the two are composed together +# with the same CSI driver value. +# +# Default provisioner: ebs.csi.eks.amazonaws.com (EKS Auto Mode's managed EBS +# CSI driver), parameters {type: gp3}, volumeBindingMode WaitForFirstConsumer +# (correct for zonal CSI drivers — late-binds the PVC to a node so EBS volumes +# land in the same AZ as the consuming pod). Override +# $spec.storageClass.{provisioner,parameters} for non-EBS providers. +# + +{{- $sc := $state.storageClass }} +{{- if $sc.enabled }} +--- +apiVersion: kubernetes.m.crossplane.io/v1alpha1 +kind: Object +metadata: + name: {{ $state.name }}-storageclass + annotations: + {{ setResourceNameAnnotation "storageclass" }} + labels: {{ $state.labels | toJson }} +spec: + managementPolicies: {{ $state.managementPolicies | toJson }} + forProvider: + manifest: + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: {{ $sc.name }} + labels: {{ $state.labels | toJson }} + provisioner: {{ $sc.provisioner | quote }} + reclaimPolicy: {{ $sc.reclaimPolicy }} + volumeBindingMode: {{ $sc.volumeBindingMode }} + allowVolumeExpansion: {{ $sc.allowVolumeExpansion }} + {{- if $sc.parameters }} + parameters: + {{- range $k, $v := $sc.parameters }} + {{ $k }}: {{ $v | quote }} + {{- end }} + {{- end }} + providerConfigRef: + name: {{ $state.kubernetesProviderConfigRef.name }} + kind: {{ $state.kubernetesProviderConfigRef.kind }} +{{- end }} diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index 6757aec..f0894dc 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -1,38 +1,55 @@ +import base64 +import file import datetime import math import models.ai.com.ops.hops.v1alpha1 as hopsv1alpha1 import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 # ============================================================================== -# Unified E2E Test for psql-stack package — full Ready integration +# Unified E2E Test for psql-stack — full Ready integration on a real EKS cluster. # -# Exercises all three XRs in the package end-to-end: -# - PSQLStack (installs CNPG operator + atlas-operator via Helm) -# - PSQLCluster (per-app Postgres XR; composes a CNPG Cluster CR + ExternalSecret) -# - PSQLBranch (ephemeral fork; composes VolumeSnapshot + bootstrapped CNPG Cluster) +# Local-fidelity pattern (mirrors aws-observe-stack): the kind cluster the +# runner spins up is the *management* cluster. The actual XRs target an +# ephemeral EKS Auto Mode cluster provisioned during the test, where the +# `psql` StorageClass + VolumeSnapshotClass composed by PSQLStack land on a +# real `ebs.csi.eks.amazonaws.com` driver — same code path that runs on +# pat-local. No fake CSI on kind; snapshots and forks actually work. # -# Bar: defaultConditions = ["Ready"]. The chain is: -# 1. snapshot-controller installed via VolumeSnapshotStack (initResources + -# extraResources VolumeSnapshotStack XR) -# 2. PSQLStack reconciles → Helm-installs CNPG + atlas-operator + a -# VolumeSnapshotClass that PSQLBranch uses as its forking target -# 3. PSQLCluster reconciles → CNPG provisions a real Postgres cluster (the -# operator is now live) with a real PVC -# 4. PSQLBranch reconciles → snapshots the source cluster's PVC, restores -# into a new CNPG cluster (snapshot-controller is now live) -# -# Pattern mirrors aws-observe-stack's e2e: install dependent Configuration -# packages via initResources, materialize their XRs in extraResources, then -# the manifests under test depend on them. +# Chain: +# 1. extraResources: AutoEKSCluster XR provisions the EKS cluster +# (subnets reused from the persistent hops-test network created by the +# aws-network e2e test). aws-auto-eks-cluster auto-creates Helm + +# Kubernetes ProviderConfigs named after the cluster. +# 2. extraResources: VolumeSnapshotStack XR installs the cluster-wide +# snapshot-controller via Helm into the new cluster. +# 3. manifests: PSQLStack reconciles → Helm-installs CNPG + Atlas + S2Z + +# `psql` StorageClass + `psql` VolumeSnapshotClass. +# 4. manifests: PSQLCluster reconciles → CNPG provisions a real Postgres +# cluster on a real EBS-backed PVC. +# 5. manifests: PSQLBranch reconciles → snapshots the source PVC via the +# composed `psql` VSC, restores into a new CNPG cluster bootstrapped +# from the snapshot. # # Run: make e2e # Run without cleanup: up test run tests/e2etest-psql --e2e --skip-control-plane-cleanup # ============================================================================== +# Workflow writes credentials + env vars to these files (write-env-files: true). +_creds = file.read("secrets/aws-creds") +_base64_creds = base64.encode(_creds) +_region = file.read("env/AWS_REGION").strip() +_account_id = file.read("env/AWS_ACCOUNT_ID").strip() +_admin_role_arn = file.read("env/ADMIN_ROLE_ARN").strip() +_private_subnet_ids = [ + file.read("env/PRIVATE_SUBNET_ID_A").strip() + file.read("env/PRIVATE_SUBNET_ID_B").strip() +] + _now = str(int(math.floor(datetime.ticks()))) _test_name = "e2e-psql-" + _now _cluster_name = "e2e-psql-cluster-" + _now _branch_name = "e2e-psql-branch-" + _now +_namespace = "default" _items = [ metav1alpha1.E2ETest { @@ -42,173 +59,259 @@ _items = [ autoUpgrade.channel = "Rapid" } defaultConditions = ["Ready"] - # 90 minutes — long chain: snapshot-controller install → PSQLStack - # Helm install (CNPG + atlas + VolumeSnapshotClass) → PSQLCluster - # bootstraps a real Postgres → PSQLBranch snapshots + restores. + # 90 minutes — EKS Auto Mode cold start is ~10–15min; CNPG bootstrap + # + snapshot fork add another ~10min on top of the platform install. timeoutSeconds = 5400 + # 30 minutes for cleanup (delete-extra-resources tears the cluster + # down; that takes ~10–15min on its own). cleanupTimeoutSeconds = 1800 skipDelete = False - # ============================================================== + # ================================================================== # initResources: dependency Configuration packages # Installed BEFORE manifests, NOT deleted during cleanup. - # ============================================================== + # ================================================================== initResources = [ - # volume-snapshot-stack — provides the cluster-wide - # snapshot-controller that PSQLBranch's VolumeSnapshot - # children rely on. EKS Auto Mode + colima ship the snapshot - # CRDs but no controller; this stack closes that gap. + # aws-auto-eks-cluster — provides the AutoEKSCluster XRD. { apiVersion = "pkg.crossplane.io/v1" kind = "Configuration" - metadata = { - name = "hops-ops-volume-snapshot-stack" - } - spec = { - package = "ghcr.io/hops-ops/volume-snapshot-stack:v0.1.0" - } + metadata.name = "aws-auto-eks-cluster" + spec.package = "ghcr.io/hops-ops/aws-auto-eks-cluster:v0.11.0" + } + # volume-snapshot-stack — provides the VolumeSnapshotStack XRD + # (installs the cluster-wide snapshot-controller). PSQLBranch's + # VolumeSnapshot only reaches readyToUse=true with this running. + { + apiVersion = "pkg.crossplane.io/v1" + kind = "Configuration" + metadata.name = "hops-ops-volume-snapshot-stack" + spec.package = "ghcr.io/hops-ops/volume-snapshot-stack:v0.1.0" } ] - # ============================================================== - # extraResources: RBAC + Provider configs + dependent XRs - # NOT deleted during cleanup. - # ============================================================== + # ================================================================== + # extraResources: AWS creds + ProviderConfigs + the EKS cluster XR + # itself. NOT deleted during cleanup (the cluster XR is torn down + # separately via delete-extra-resources in the workflow so it + # outlives the manifests' cascade delete). + # ================================================================== extraResources = [ - # cluster-admin to all SAs in crossplane-system so Helm + - # Kubernetes providers can apply wrapped manifests freely. + # AWS credentials secret (sourced from the OIDC-assumed role). { - apiVersion = "rbac.authorization.k8s.io/v1" - kind = "ClusterRoleBinding" - metadata.name = "crossplane-system-cluster-admin" - roleRef = { - apiGroup = "rbac.authorization.k8s.io" - kind = "ClusterRole" - name = "cluster-admin" - } - subjects = [ - { - kind = "Group" - name = "system:serviceaccounts:crossplane-system" - apiGroup = "rbac.authorization.k8s.io" - } - ] + apiVersion = "v1" + kind = "Secret" + metadata = { + name = "aws-creds" + namespace = _namespace + } + data = { + creds = _base64_creds + } } - # Helm ProviderConfig — for PSQLStack + VolumeSnapshotStack + # AWS ProviderConfig — for AutoEKSCluster's IAM/EKS resources. { - apiVersion = "helm.m.crossplane.io/v1beta1" + apiVersion = "aws.m.upbound.io/v1beta1" kind = "ProviderConfig" metadata = { name = "default" - namespace = "default" + namespace = _namespace } spec = { credentials = { - source = "InjectedIdentity" + source = "Secret" + secretRef = { + namespace = _namespace + name = "aws-creds" + key = "creds" + } } } } - # Kubernetes ProviderConfig — for PSQLCluster + PSQLBranch + # Kubernetes ProviderConfig "default" — used by aws-auto-eks-cluster + # to apply the per-cluster Helm + Kubernetes ProviderConfigs into + # the management cluster (kind). Cluster-targeting XRs reference + # the per-cluster ProviderConfigs (named after _test_name). { - apiVersion = "kubernetes.m.crossplane.io/v1beta1" - kind = "ProviderConfig" + apiVersion: "kubernetes.m.crossplane.io/v1alpha1" + kind: "ProviderConfig" + metadata: { + name: "default" + namespace: "default" + } + spec: { + credentials: { + source: "InjectedIdentity" + } + } + } + # DeploymentRuntimeConfig for provider-kubernetes — gives the + # provider a known SA name we can grant cluster-admin on the + # management cluster. + { + apiVersion: "pkg.crossplane.io/v1beta1" + kind: "DeploymentRuntimeConfig" + metadata: { + name: "crossplane-contrib-provider-kubernetes" + } + spec: { + serviceAccountTemplate: { + metadata: { + name: "crossplane-contrib-provider-kubernetes" + } + } + } + } + { + apiVersion: "v1" + kind: "ServiceAccount" + metadata: { + name: "crossplane-contrib-provider-kubernetes" + namespace: "crossplane-system" + } + } + { + apiVersion: "rbac.authorization.k8s.io/v1" + kind: "ClusterRoleBinding" + metadata: { + name: "crossplane-contrib-provider-kubernetes-admin" + } + roleRef: { + apiGroup: "rbac.authorization.k8s.io" + kind: "ClusterRole" + name: "cluster-admin" + } + subjects: [ + { + kind: "ServiceAccount" + name: "crossplane-contrib-provider-kubernetes" + namespace: "crossplane-system" + } + ] + } + # Pin the K8s provider so the runtime config takes effect. + { + apiVersion: "pkg.crossplane.io/v1" + kind: "Provider" + metadata: { + name: "crossplane-contrib-provider-kubernetes" + } + spec: { + package: "xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.2.0" + runtimeConfigRef: { + name: "crossplane-contrib-provider-kubernetes" + } + } + } + # The ephemeral EKS cluster. Reuses the persistent hops-test + # network's private subnets so we don't pay VPC create cost + # per PR. AutoEKSCluster auto-creates Helm + Kubernetes + # ProviderConfigs named _test_name once Ready. + { + apiVersion = "aws.hops.ops.com.ai/v1alpha1" + kind = "AutoEKSCluster" metadata = { - name = "default" - namespace = "default" + name = _test_name + namespace = _namespace } spec = { - credentials = { - source = "InjectedIdentity" + adminRoleArn = _admin_role_arn + clusterName = _test_name + region = _region + accountId = _account_id + version = "1.35" + subnetIds = _private_subnet_ids + providerConfigRef = { + name = "default" + kind = "ProviderConfig" + } + kubernetesProviderConfigRef = { + name = "default" + kind = "ProviderConfig" + } + tags = { + "e2etest" = "true" + "test-run" = _now + "repo": "https://github.com/hops-ops/psql-stack" + } + publicAccess = True + nodeConfig = { + enabled = True + } + oidc = { + enabled = False } } } - # VolumeSnapshotStack XR — installs the cluster-wide - # snapshot-controller via Helm. PSQLBranch's VolumeSnapshot - # objects only get reconciled to readyToUse=true when this - # controller is running. + # snapshot-controller on the new cluster — gates VolumeSnapshot + # readiness for PSQLBranch's fork. clusterName = _test_name + # routes Helm installs through the per-cluster ProviderConfig + # auto-created by AutoEKSCluster. { apiVersion = "hops.ops.com.ai/v1alpha1" kind = "VolumeSnapshotStack" metadata = { name = "snapshot" - namespace = "default" + namespace = _namespace } spec = { - clusterName = "default" + clusterName = _test_name + labels = { + "hops.ops.com.ai/e2etest" = "true" + "hops.ops.com.ai/test-run" = _now + } } } ] - # ============================================================== - # manifests: all three XRs under test, applied together. - # Deleted during cleanup (unless --skip-delete). - # ============================================================== + # ================================================================== + # manifests: the three psql-stack XRs. Each targets the EKS cluster + # via clusterName=_test_name, which the compositions translate into + # providerConfigRef.name on the per-cluster ProviderConfigs created + # by AutoEKSCluster. Cleaned up cascade-deletes the CNPG primaries + + # branch + their PVCs/snapshots; the cluster itself stays up until + # the workflow's delete-extra-resources step tears it down. + # ================================================================== manifests = [ - # 1. PSQLStack — Helm-installs CNPG operator + atlas-operator + VolumeSnapshotClass. - # Defaults: namespace=cnpg-system, ProviderConfigs default to clusterName. hopsv1alpha1.PSQLStack { metadata = { name = _test_name - namespace = "default" + namespace = _namespace } spec = { - clusterName = "default" + clusterName = _test_name labels = { "hops.ops.com.ai/e2etest" = "true" "hops.ops.com.ai/test-run" = _now } - helmProviderConfigRef = { - name = "default" - kind = "ProviderConfig" - } - kubernetesProviderConfigRef = { - name = "default" - kind = "ProviderConfig" - } } } - # 2. PSQLCluster — per-app Postgres XR. CNPG (installed by - # PSQLStack above) provisions a real Postgres cluster with - # a real PVC backing the data. hopsv1alpha1.PSQLCluster { metadata = { name = _cluster_name - namespace = "default" + namespace = _namespace } spec = { - clusterName = "default" + clusterName = _test_name labels = { "hops.ops.com.ai/e2etest" = "true" "hops.ops.com.ai/test-run" = _now } - kubernetesProviderConfigRef = { - name = "default" - kind = "ProviderConfig" - } storage = {size = "1Gi"} - # Omit `app` entirely — no ESO in kind, no pre-created - # secret. CNPG auto-generates `-app`. + # No app block — CNPG auto-generates `-app`. } } - # 3. PSQLBranch — ephemeral fork. Snapshots the source - # PSQLCluster's PVC (snapshot-controller installed by the - # VolumeSnapshotStack extraResource handles this), restores - # into a new CNPG cluster bootstrapped from the snapshot. hopsv1alpha1.PSQLBranch { metadata = { name = _branch_name - namespace = "default" + namespace = _namespace } spec = { - clusterName = "default" + clusterName = _test_name labels = { "hops.ops.com.ai/e2etest" = "true" "hops.ops.com.ai/test-run" = _now } - kubernetesProviderConfigRef = { - name = "default" - kind = "ProviderConfig" - } source = {name = _cluster_name} } } diff --git a/tests/test-stack/main.k b/tests/test-stack/main.k index 72e56e6..c7e8815 100644 --- a/tests/test-stack/main.k +++ b/tests/test-stack/main.k @@ -12,8 +12,8 @@ import models.io.upbound.dev.meta.v1alpha1 as metav1alpha1 _items = [ # ========================================================================== # Test 1: minimal claim renders the platform operators (CNPG, Atlas), the - # scale-to-zero plugin (default-on), and the VolumeSnapshotClass - # (default driver: ebs.csi.aws.com, default name: "psql"). + # scale-to-zero plugin (default-on), the StorageClass and VolumeSnapshotClass + # (default name: "psql", paired driver/provisioner: ebs.csi.eks.amazonaws.com). # ========================================================================== metav1alpha1.CompositionTest { metadata.name = "minimal-renders-platform-stack" @@ -37,6 +37,11 @@ _items = [ kind = "Release" metadata.name = "atlas-operator" } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "test-psql-storageclass" + } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" @@ -295,7 +300,7 @@ _items = [ scaleToZeroPlugin = {enabled = False} } } - # Positive presence: cnpg + atlas + VSC still composed. + # Positive presence: cnpg + atlas + SC + VSC still composed. # Absence of s2z-* resources is implicit (composition gates them). assertResources = [ { @@ -308,6 +313,11 @@ _items = [ kind = "Release" metadata.name = "atlas-operator" } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "no-s2z-storageclass" + } { apiVersion = "kubernetes.m.crossplane.io/v1alpha1" kind = "Object" @@ -412,7 +422,7 @@ _items = [ snapshotClass = {enabled = False} } } - # Cnpg + atlas still composed. No vsc-* Object expected (implicit). + # Cnpg + atlas + SC still composed. No vsc-* Object expected (implicit). assertResources = [ { apiVersion = "helm.m.crossplane.io/v1beta1" @@ -424,6 +434,101 @@ _items = [ kind = "Release" metadata.name = "atlas-operator" } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "no-vsc-storageclass" + } + ] + } + } + + # ========================================================================== + # Test 12: storageClass settings flow through to the composed StorageClass. + # Override provisioner + parameters to verify they propagate (e.g., for + # non-EKS targets that use a different CSI driver). + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "storageclass-provisioner-flows-through" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "sc-tweaks" + spec = { + clusterName = "my-cluster" + storageClass = { + provisioner = "driver.longhorn.io" + parameters = {numberOfReplicas = "3"} + reclaimPolicy = "Retain" + } + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "sc-tweaks-storageclass" + spec.forProvider.manifest = { + apiVersion = "storage.k8s.io/v1" + kind = "StorageClass" + metadata = { + name = "psql" + labels = { + "hops.ops.com.ai/managed" = "true" + "hops.ops.com.ai/psql" = "sc-tweaks" + } + } + provisioner = "driver.longhorn.io" + reclaimPolicy = "Retain" + volumeBindingMode = "WaitForFirstConsumer" + allowVolumeExpansion = True + parameters = {numberOfReplicas = "3"} + } + } + ] + } + } + + # ========================================================================== + # Test 13: storageClass.enabled=false suppresses the SC composition. + # Use case: cluster already ships a suitable default SC; opt out of the + # stack composing one. PSQLCluster/PSQLBranch consumers must then set + # `spec.storage.class` explicitly. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "storageclass-can-be-disabled" + spec = { + compositionPath = "apis/psqlstacks/composition.yaml" + xrdPath = "apis/psqlstacks/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLStack { + metadata.name = "no-sc" + spec = { + clusterName = "my-cluster" + storageClass = {enabled = False} + } + } + # Cnpg + atlas + VSC still composed. No sc-* Object expected (implicit). + assertResources = [ + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "cloudnative-pg" + } + { + apiVersion = "helm.m.crossplane.io/v1beta1" + kind = "Release" + metadata.name = "atlas-operator" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "no-sc-volumesnapshotclass" + } ] } } From 877c9c1943d75b16b5f28eaa814823beb18c7bb5 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 10:46:00 -0500 Subject: [PATCH 32/44] ci(e2e): install cert-stack alongside volume-snapshot-stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The full chain converged on the previous run except for PSQLStack's scaleToZeroPlugin objects (s2z-cert-client, s2z-cert-server, s2z-issuer) — the S2Z plugin uses cert-manager's Issuer + Certificate for its gRPC mTLS pair, and the ephemeral EKS cluster has no cert-manager. pat-local already has cert-manager from earlier setup, which masked this in local validation. cert-stack v0.1.0 mirrors volume-snapshot-stack's shape (single clusterName field, composes a Helm release on the target). Adding it as initResource + extraResource closes the gap. --- .github/workflows/on-pr.yaml | 8 +++++--- tests/e2etest-psql/main.k | 28 ++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/.github/workflows/on-pr.yaml b/.github/workflows/on-pr.yaml index 5a20ecf..ef33723 100644 --- a/.github/workflows/on-pr.yaml +++ b/.github/workflows/on-pr.yaml @@ -68,15 +68,17 @@ jobs: "psqlclusters.hops.ops.com.ai", "psqlbranches.hops.ops.com.ai", "volumesnapshotstacks.hops.ops.com.ai", + "certstacks.hops.ops.com.ai", "autoeksclusters.aws.hops.ops.com.ai" ] # AutoEKSCluster outlives the manifests' cascade delete; the workflow - # tears it down separately. VolumeSnapshotStack is colocated with the - # cluster — drop it together so the Helm release on the EKS cluster - # is uninstalled before the cluster goes. + # tears it down separately. VolumeSnapshotStack + CertStack are + # colocated with the cluster — drop them together so their Helm + # releases on the EKS cluster are uninstalled before the cluster goes. delete-extra-resources: | [ "volumesnapshotstacks.hops.ops.com.ai", + "certstacks.hops.ops.com.ai", "autoeksclusters.aws.hops.ops.com.ai" ] # 90 min: ~10–15 min EKS Auto Mode cold start, then platform install + diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index f0894dc..669c9ae 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -88,6 +88,16 @@ _items = [ metadata.name = "hops-ops-volume-snapshot-stack" spec.package = "ghcr.io/hops-ops/volume-snapshot-stack:v0.1.0" } + # cert-stack — installs cert-manager onto the target cluster. + # PSQLStack's scaleToZeroPlugin uses cert-manager Issuer + + # Certificate for the plugin's gRPC mTLS pair, so without this + # the stack stays NotReady on `s2z-cert-{client,server,issuer}`. + { + apiVersion = "pkg.crossplane.io/v1" + kind = "Configuration" + metadata.name = "hops-ops-cert-stack" + spec.package = "ghcr.io/hops-ops/cert-stack:v0.1.0" + } ] # ================================================================== @@ -262,6 +272,24 @@ _items = [ } } } + # cert-manager on the new cluster — gates the s2z plugin's TLS + # pair (Issuer + client/server Certificates). Without this, + # PSQLStack stays NotReady on s2z-cert-* objects. + { + apiVersion = "hops.ops.com.ai/v1alpha1" + kind = "CertStack" + metadata = { + name = "cert" + namespace = _namespace + } + spec = { + clusterName = _test_name + labels = { + "hops.ops.com.ai/e2etest" = "true" + "hops.ops.com.ai/test-run" = _now + } + } + } ] # ================================================================== From e537e262139bcb6de99da7c2942ae85064254af4 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 13:16:39 -0500 Subject: [PATCH 33/44] ci: pin reusable workflows to hops-ops/workflows-crossplane@v3.0.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the four mutable `@feat/multi-api-support` branch refs with the immutable `@v3.0.0` tag now that the multi-api work is shipped. Also retargets the org from `unbounded-tech` to `hops-ops` — the canonical home for hops platform CI workflows. --- .github/workflows/on-pr.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/on-pr.yaml b/.github/workflows/on-pr.yaml index ef33723..4d40c9a 100644 --- a/.github/workflows/on-pr.yaml +++ b/.github/workflows/on-pr.yaml @@ -27,7 +27,7 @@ permissions: jobs: validate: - uses: unbounded-tech/workflows-crossplane/.github/workflows/validate.yaml@feat/multi-api-support + uses: hops-ops/workflows-crossplane/.github/workflows/validate.yaml@v3.0.0 with: examples: | [ @@ -40,10 +40,10 @@ jobs: error_on_missing_schemas: true test: - uses: unbounded-tech/workflows-crossplane/.github/workflows/test.yaml@feat/multi-api-support + uses: hops-ops/workflows-crossplane/.github/workflows/test.yaml@v3.0.0 e2e: - uses: unbounded-tech/workflows-crossplane/.github/workflows/e2e.yaml@feat/multi-api-support + uses: hops-ops/workflows-crossplane/.github/workflows/e2e.yaml@v3.0.0 with: # E2E provisions a real EKS Auto Mode cluster (mirror of aws-observe-stack) # so the `psql` StorageClass + VolumeSnapshotClass land on a real @@ -92,7 +92,7 @@ jobs: - validate - test - e2e - uses: unbounded-tech/workflows-crossplane/.github/workflows/publish.yaml@feat/multi-api-support + uses: hops-ops/workflows-crossplane/.github/workflows/publish.yaml@v3.0.0 secrets: inherit with: tag: pr-${{ github.event.pull_request.number }}-${{ github.sha }} From 663b5e73f4842c334aa6585a1ae135d1b27506c1 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 13:26:15 -0500 Subject: [PATCH 34/44] fix(psqlbranch): drop default postgres version, gate imageName on explicit set The previous schema and gotmpl both defaulted branch postgres version to "17", silently coercing branches off PG 15/16 sources to a major mismatch. Volume snapshot recovery is binary-compatible only within a major version (Postgres won't open a data dir from a different major), so a hardcoded default produced silent failures at restore time. Now: omit imageName entirely when version is unset, letting CNPG fall back to its operator-default image (close-enough when source tracks the same chart). Setting spec.postgresql.version is now an explicit pin, typically used to fix a minor (e.g. "17.4") to match the source's reported version. Adds a render test for the explicit-pin path; existing default-path test asserts imageName absence. --- apis/psqlbranches/definition.yaml | 12 +++++-- functions/branch/000-state-init.yaml.gotmpl | 6 ++-- functions/branch/200-cnpg-cluster.yaml.gotmpl | 8 ++++- tests/test-branch/main.k | 36 ++++++++++++++++++- 4 files changed, 56 insertions(+), 6 deletions(-) diff --git a/apis/psqlbranches/definition.yaml b/apis/psqlbranches/definition.yaml index 2989a79..6730fdc 100644 --- a/apis/psqlbranches/definition.yaml +++ b/apis/psqlbranches/definition.yaml @@ -115,12 +115,20 @@ spec: default: psql postgresql: - description: Postgres version on the branch (must match source for snapshot recovery to succeed). + description: | + Postgres version on the branch. **Must match the source's + major version** for snapshot recovery to succeed — Postgres + doesn't downgrade or skip-upgrade across data dirs. + Intentionally has no default: omit to let CNPG use its + operator-default image (close-enough when source tracks + the same chart), or set explicitly to mirror the source's + version. A `default: "17"` here was a footgun — branches + off PG 15/16 sources would silently mismatch. type: object properties: version: + description: Postgres major version (e.g. "17"). Empty = let CNPG pick its default; mirror the source's spec.postgresql.version for safety. type: string - default: "17" scaleToZero: description: | diff --git a/functions/branch/000-state-init.yaml.gotmpl b/functions/branch/000-state-init.yaml.gotmpl index 5d7d2d7..a283da1 100644 --- a/functions/branch/000-state-init.yaml.gotmpl +++ b/functions/branch/000-state-init.yaml.gotmpl @@ -74,11 +74,13 @@ }} # ============================================================================== -# Postgres version (must match source for snapshot recovery to succeed) +# Postgres version — must match source's major for snapshot recovery to +# succeed. No default: omit `imageName` downstream so CNPG falls back to its +# operator default (matches the chart's pinned PG when source tracks it). # ============================================================================== {{- $pgSpec := $spec.postgresql | default dict }} {{- $postgresql := dict - "version" ($pgSpec.version | default "17") + "version" ($pgSpec.version | default "") }} # ============================================================================== diff --git a/functions/branch/200-cnpg-cluster.yaml.gotmpl b/functions/branch/200-cnpg-cluster.yaml.gotmpl index 7329e5d..1a62699 100644 --- a/functions/branch/200-cnpg-cluster.yaml.gotmpl +++ b/functions/branch/200-cnpg-cluster.yaml.gotmpl @@ -62,7 +62,6 @@ spec: spec: {{- $clusterSpec := dict "instances" $state.branch.instances - "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) "bootstrap" (dict "recovery" (dict "volumeSnapshots" (dict @@ -75,6 +74,13 @@ spec: ) ) }} + {{- /* imageName only set when version is explicit. Empty = let CNPG + use its operator-default image (close-enough when source tracks + the same chart). Hardcoding e.g. "17" was a footgun: branches + off PG 15/16 sources would silently mismatch and fail recovery. */}} + {{- if $state.postgresql.version }} + {{- $_ := set $clusterSpec "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) }} + {{- end }} {{- /* Storage — size always set (state-init defaults to 10Gi; consumer must override for larger sources). class only set when explicitly provided. */}} diff --git a/tests/test-branch/main.k b/tests/test-branch/main.k index 9d2500e..e731280 100644 --- a/tests/test-branch/main.k +++ b/tests/test-branch/main.k @@ -61,9 +61,12 @@ _items = [ "hops.ops.com.ai/branch-of" = "src-app" } } + # No imageName — when spec.postgresql.version is omitted, + # CNPG falls back to its operator-default image. See + # test "explicit-postgres-version-sets-imagename" for + # the override path. spec = { instances = 1 - imageName = "ghcr.io/cloudnative-pg/postgresql:17" bootstrap.recovery.volumeSnapshots.storage = { name = "br-same-snap" kind = "VolumeSnapshot" @@ -76,6 +79,37 @@ _items = [ } } + # ========================================================================== + # Test 1b: explicit spec.postgresql.version sets imageName. Branches MUST + # match the source's major version (snapshot recovery is binary-compatible + # only within a major) — pinning the minor here is the typical use case. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "explicit-postgres-version-sets-imagename" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-pinned", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = {name = "src-app"} + postgresql.version = "17.4" + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-pinned-cnpg-cluster" + spec.forProvider.manifest.spec.imageName = "ghcr.io/cloudnative-pg/postgresql:17.4" + } + ] + } + } + # ========================================================================== # Test 2: cross-namespace branch — source-snapshot in source ns + branch- # snapshot in branch ns + Cluster in branch ns. From b4c6e62b94cc744c9d465d4ccd50a6949549b6b8 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 13:29:17 -0500 Subject: [PATCH 35/44] test(psqlcluster): lock in monitoring.enabled=false honors explicit off Existing state-init logic correctly uses hasKey to distinguish "explicitly false" from "absent" before reading $monSpec.enabled, but no test exercised the explicit-false path. Added regression test asserting that spec.monitoring.enabled=false propagates to the composed Cluster CR's monitoring.enablePodMonitor=false. Default-on path covered by Test 1. --- tests/test-cluster/main.k | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/tests/test-cluster/main.k b/tests/test-cluster/main.k index 8369e1c..dbb1905 100644 --- a/tests/test-cluster/main.k +++ b/tests/test-cluster/main.k @@ -405,6 +405,38 @@ _items = [ ] } } + + # ========================================================================== + # monitoring.enabled=false propagates to the Cluster CR. Default is true + # (covered by Test 1's monitoring.enablePodMonitor = True assertion); this + # locks in that explicit `false` is honored — the hasKey guard in + # state-init distinguishes "explicitly off" from "absent". + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "monitoring-disabled-when-explicit-false" + spec = { + compositionPath = "apis/psqlclusters/composition.yaml" + xrdPath = "apis/psqlclusters/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLCluster { + metadata = {name = "no-mon", namespace = "default"} + spec = { + clusterName = "my-cluster" + storage = {size = "10Gi"} + monitoring.enabled = False + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "no-mon-cnpg-cluster" + spec.forProvider.manifest.spec.monitoring.enablePodMonitor = False + } + ] + } + } ] items = _items From 18d31154f5ddc55c6f66194457eaf09a0038bf6b Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 13:36:34 -0500 Subject: [PATCH 36/44] fix(psqlbranch): prefer source-snapshot content over empty branch content MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In cross-namespace branching, the branch-ns VolumeSnapshot can only bind once it learns the source's VolumeSnapshotContent name — that name is propagated through $state.observed.snapshotContent into 110-branch-snapshot's render. The previous logic preferred branch-snapshot whenever it was present in observed, even when its boundVolumeSnapshotContentName was still empty — which it always is on first reconcile, before the source content has propagated. Result: empty content overwrites the populated source content, the branch-ns snapshot renders with `volumeSnapshotContentName: ""`, and the chain stalls. Now: read both branch-ns and source-ns content via the existing pipeline, prefer branch when non-empty, fall back to source otherwise. Fixes the chicken-and-egg that prevented cross-namespace branches from binding on first-pass reconcile. Same-namespace branching is unaffected: source-snapshot is gated on crossNamespace and isn't composed there, so $sourceContent is empty and the branch-ns content (always populated post-bind) wins. --- functions/branch/010-state-status.yaml.gotmpl | 37 ++++++++++++++----- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/functions/branch/010-state-status.yaml.gotmpl b/functions/branch/010-state-status.yaml.gotmpl index ea788a9..0ebecaf 100644 --- a/functions/branch/010-state-status.yaml.gotmpl +++ b/functions/branch/010-state-status.yaml.gotmpl @@ -14,17 +14,34 @@ {{- $clusterStatus := $clusterManifest.status | default dict }} {{- $clusterPhase := $clusterStatus.phase | default "" }} -# Read the bound VolumeSnapshotContent name from the branch-ns VolumeSnapshot -# (or the source-ns one when same-namespace). -{{- $snapEntry := get $observed "branch-snapshot" | default dict }} -{{- if not (get $observed "branch-snapshot") }} - {{- $snapEntry = get $observed "source-snapshot" | default dict }} +# Read the bound VolumeSnapshotContent name. Two paths: +# - same-namespace: only `branch-snapshot` exists (it references the source +# PVC directly), so we read from it. +# - cross-namespace: both `source-snapshot` (in source ns) and +# `branch-snapshot` (in branch ns) exist. The branch-ns snapshot can only +# bind once it learns the source's content name (propagated through this +# status field into 110-branch-snapshot's render), so reading branch-ns +# first creates a chicken-and-egg: empty branch content overwrites the +# populated source content. Prefer branch-ns only when its content is +# non-empty; otherwise fall back to source-ns. +{{- $snapEntryBranch := get $observed "branch-snapshot" | default dict }} +{{- $branchResource := $snapEntryBranch.resource | default dict }} +{{- $branchAtProvider := (($branchResource.status | default dict).atProvider | default dict) }} +{{- $branchManifest := $branchAtProvider.manifest | default dict }} +{{- $branchStatus := $branchManifest.status | default dict }} +{{- $branchContent := $branchStatus.boundVolumeSnapshotContentName | default "" }} + +{{- $snapEntrySource := get $observed "source-snapshot" | default dict }} +{{- $sourceResource := $snapEntrySource.resource | default dict }} +{{- $sourceAtProvider := (($sourceResource.status | default dict).atProvider | default dict) }} +{{- $sourceManifest := $sourceAtProvider.manifest | default dict }} +{{- $sourceStatus := $sourceManifest.status | default dict }} +{{- $sourceContent := $sourceStatus.boundVolumeSnapshotContentName | default "" }} + +{{- $snapContent := $branchContent }} +{{- if not $branchContent }} + {{- $snapContent = $sourceContent }} {{- end }} -{{- $snapResource := $snapEntry.resource | default dict }} -{{- $snapAtProvider := (($snapResource.status | default dict).atProvider | default dict) }} -{{- $snapManifest := $snapAtProvider.manifest | default dict }} -{{- $snapStatus := $snapManifest.status | default dict }} -{{- $snapContent := $snapStatus.boundVolumeSnapshotContentName | default "" }} {{- $state = set $state "observed" (dict "cluster" (dict "phase" $clusterPhase) From dd72818c40002a392bb6fe134b2ecb55bf1cf38c Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 13:39:12 -0500 Subject: [PATCH 37/44] test(psqlbranch): cross-namespace state-status with observed fixtures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds two CompositionTests that exercise 010-state-status's snapshot- content fallback against inline observedResources: 1. branch-snapshot bound content empty + source-snapshot bound: branch-ns VolumeSnapshot must render with the source's content name (proves the chicken-and-egg fix from 18d3115). 2. branch-snapshot bound + source-snapshot bound: steady-state — branch's content wins (proves the fallback doesn't overwrite a populated branch content with source's). Without the fix, test 1 would render volumeSnapshotContentName: "" and fail. With it, source's content propagates through state.observed.snapshotContent into 110-branch-snapshot's render. Locks in the cross-namespace bind behavior — the previous test surface only exercised render shape, not state propagation across reconciles. --- tests/test-branch/main.k | 121 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) diff --git a/tests/test-branch/main.k b/tests/test-branch/main.k index e731280..44f62f8 100644 --- a/tests/test-branch/main.k +++ b/tests/test-branch/main.k @@ -79,6 +79,127 @@ _items = [ } } + # ========================================================================== + # Cross-namespace, first reconcile: source-snapshot has bound to its + # VolumeSnapshotContent, branch-snapshot's underlying VolumeSnapshot was + # rendered earlier with an empty volumeSnapshotContentName and hasn't + # bound yet. The state-status logic must read from source-snapshot + # (since branch-snapshot's bound content is still empty), so the + # branch-ns VolumeSnapshot now renders with the source's content name — + # closing the chicken-and-egg. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "cross-ns-prefers-source-content-when-branch-empty" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-x", namespace = "preview-pr-1"} + spec = { + clusterName = "prod" + source = {name = "src-app", namespace = "team-app"} + } + } + observedResources = [ + # source-ns VolumeSnapshot (Object wrapper) — bound. + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata = { + name = "br-x-source-snapshot" + annotations = { + "crossplane.io/composition-resource-name" = "source-snapshot" + "gotemplating.fn.crossplane.io/composition-resource-name" = "source-snapshot" + } + } + status.atProvider.manifest.status.boundVolumeSnapshotContentName = "snapcontent-from-source" + } + # branch-ns VolumeSnapshot (Object wrapper) — created with + # empty content reference on the prior reconcile, not bound. + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata = { + name = "br-x-branch-snapshot" + annotations = { + "crossplane.io/composition-resource-name" = "branch-snapshot" + "gotemplating.fn.crossplane.io/composition-resource-name" = "branch-snapshot" + } + } + status.atProvider.manifest.status.boundVolumeSnapshotContentName = "" + } + ] + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-x-branch-snapshot" + spec.forProvider.manifest.spec.source.volumeSnapshotContentName = "snapcontent-from-source" + } + ] + } + } + + # ========================================================================== + # Cross-namespace, steady state: branch-snapshot has bound (its + # boundVolumeSnapshotContentName is populated). Reading branch's value + # is fine here — it will match source's. This locks in that the + # fallback doesn't override a populated branch content with the + # source's value. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "cross-ns-uses-branch-content-when-bound" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-y", namespace = "preview-pr-2"} + spec = { + clusterName = "prod" + source = {name = "src-app", namespace = "team-app"} + } + } + observedResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata = { + name = "br-y-source-snapshot" + annotations = { + "crossplane.io/composition-resource-name" = "source-snapshot" + "gotemplating.fn.crossplane.io/composition-resource-name" = "source-snapshot" + } + } + status.atProvider.manifest.status.boundVolumeSnapshotContentName = "snapcontent-from-source" + } + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata = { + name = "br-y-branch-snapshot" + annotations = { + "crossplane.io/composition-resource-name" = "branch-snapshot" + "gotemplating.fn.crossplane.io/composition-resource-name" = "branch-snapshot" + } + } + status.atProvider.manifest.status.boundVolumeSnapshotContentName = "snapcontent-from-branch" + } + ] + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-y-branch-snapshot" + spec.forProvider.manifest.spec.source.volumeSnapshotContentName = "snapcontent-from-branch" + } + ] + } + } + # ========================================================================== # Test 1b: explicit spec.postgresql.version sets imageName. Branches MUST # match the source's major version (snapshot recovery is binary-compatible From ed61c1d0be2567fcbcee45a566cb0eed2e5bb398 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 13:47:06 -0500 Subject: [PATCH 38/44] fix(psqlbranch): prefix source-ns VolumeSnapshot name with branch namespace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The source-ns VolumeSnapshot lives in a namespace shared by multiple branches (the source PSQLCluster's). Naming it just `-src` collides when two PSQLBranch XRs have the same metadata.name in different branch namespaces — both would create `preview-pr-1-src` in `team-app`, racing on the same object. Now: `--src`. Encodes the branch XR's own namespace into the name so (sourceNS, branchNS, branchName) is unique. K8s names are bound by RFC 1123 subdomain (253 chars) — concatenation is safe at any reasonable namespace/branch length. Branch-ns snapshot (`-snap`) is unchanged: the branch namespace is already implicit by where the resource lives, and branch XR names are unique within a single namespace. Adds a regression test that asserts the source-ns name varies with branch namespace. --- .../branch/100-source-snapshot.yaml.gotmpl | 12 +++++- tests/test-branch/main.k | 40 ++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/functions/branch/100-source-snapshot.yaml.gotmpl b/functions/branch/100-source-snapshot.yaml.gotmpl index 46be663..5141f67 100644 --- a/functions/branch/100-source-snapshot.yaml.gotmpl +++ b/functions/branch/100-source-snapshot.yaml.gotmpl @@ -12,9 +12,19 @@ # For same-namespace branching, this template is skipped — the branch-ns # snapshot references the source PVC directly. # +# Naming: the underlying VolumeSnapshot lives in the SOURCE namespace, but +# its identity belongs to the BRANCH XR. Two PSQLBranch XRs with the same +# `metadata.name` in different branch namespaces (e.g. `preview-pr-1` in +# `team-a-preview` and `team-b-preview`) would otherwise collide on a single +# `-src` in the shared source ns. Prefix with the branch's own +# namespace to make the name (sourceNS, branchNS, branchName)-unique. +# K8s names are bound by RFC 1123 subdomain (253 chars) — concatenation is +# safe at any reasonable namespace/branch length. +# {{- $source := $state.source }} {{- if $state.crossNamespace }} +{{- $sourceSnapName := printf "%s-%s-src" $state.namespace $state.name }} --- apiVersion: kubernetes.m.crossplane.io/v1alpha1 kind: Object @@ -30,7 +40,7 @@ spec: apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: - name: {{ $state.name }}-src + name: {{ $sourceSnapName }} namespace: {{ $source.namespace }} labels: {{ $state.labels | toJson }} spec: diff --git a/tests/test-branch/main.k b/tests/test-branch/main.k index 44f62f8..515898e 100644 --- a/tests/test-branch/main.k +++ b/tests/test-branch/main.k @@ -257,8 +257,11 @@ _items = [ spec.forProvider.manifest = { apiVersion = "snapshot.storage.k8s.io/v1" kind = "VolumeSnapshot" + # Name prefixed with branch namespace so two branches + # with the same XR name in different namespaces don't + # collide on a single `-src` in the shared source ns. metadata = { - name = "br-x-src" + name = "preview-pr-1-br-x-src" namespace = "team-app" } spec.source.persistentVolumeClaimName = "src-app-1" @@ -286,6 +289,41 @@ _items = [ } } + # ========================================================================== + # Test 2b: same branch XR name (`br-x`) in a DIFFERENT branch namespace + # (`other-team-preview` vs `preview-pr-1` above) produces a different + # source-snapshot name in the shared source ns. Locks in the + # collision-avoidance contract: source-ns name encodes the branch's own + # namespace. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "cross-ns-source-snapshot-name-distinguishes-branch-ns" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-x", namespace = "other-team-preview"} + spec = { + clusterName = "prod" + source = {name = "src-app", namespace = "team-app"} + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-x-source-snapshot" + spec.forProvider.manifest.metadata = { + name = "other-team-preview-br-x-src" + namespace = "team-app" + } + } + ] + } + } + # ========================================================================== # Test 3: scaleToZero default-on; renders annotation + plugin entry. # ========================================================================== From e1ad9ec6e1bacdfa58a5c425da8eb5e25141fe03 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 14:21:25 -0500 Subject: [PATCH 39/44] fix(psqlbranch): fall back to source.storage.size when branch size empty MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously the gotmpl coerced unset `branch.storage.size` to a hardcoded "10Gi" — silently mis-sizing branches off any source PVC larger than that. CNPG/EBS can't shrink during recovery, so a 10Gi branch off a 100Gi source fails when CNPG tries to bind the restored PVC. New shape: - PSQLBranch XRD adds optional `spec.source.storage.size` so consumers can mirror the source PSQLCluster's known capacity. The branch composition has no automatic visibility into the source's PVC, so the user declares it once on the branch spec. - Size precedence in the cnpg-cluster render: branch.storage.size (explicit override, e.g. growing the branch) → source.storage.size (inherit from the source) → omitted (CNPG's webhook rejects with a clear error) - Drops the silent 10Gi gotmpl fallback in 000-state-init. Examples + e2e branch updated to declare source.storage.size mirroring the source PSQLCluster's spec.storage.size. Local pat-local manifests already set branch.storage.size explicitly so unaffected. Symlinks the per-test KCL `model/` directory to `.up/kcl/models` so schema changes from `up project build` propagate to tests automatically. Previously the bundled models drifted from the XRD on every change and needed manual regeneration. `.up/` is gitignored — fresh clones run `make build` (or `up project build`) to populate the symlink target. Adds two regression tests: branch.size overrides source.size, and source.size used when branch.size is empty. --- apis/psqlbranches/definition.yaml | 24 +- examples/psqlbranches/same-namespace.yaml | 4 + functions/branch/000-state-init.yaml.gotmpl | 16 +- functions/branch/200-cnpg-cluster.yaml.gotmpl | 17 +- tests/e2etest-psql/main.k | 12 +- tests/test-branch/main.k | 63 ++ tests/test-branch/model | 1 + .../ai/com/ops/hops/v1alpha1/psqlbranch.k | 481 ------------- .../ai/com/ops/hops/v1alpha1/psqlcluster.k | 664 ------------------ .../dev/meta/v1alpha1/compositiontest.k | 113 --- .../io/upbound/dev/meta/v1alpha1/e2etest.k | 167 ----- .../upbound/dev/meta/v1alpha1/operationtest.k | 98 --- .../io/upbound/dev/meta/v1alpha1/project.k | 320 --------- .../io/upbound/dev/meta/v2alpha1/project.k | 325 --------- .../pkg/apis/meta/v1/object_meta.k | 97 --- .../pkg/apis/meta/v1/owner_reference.k | 41 -- tests/test-branch/model/kcl.mod | 4 - tests/test-cluster/model | 1 + .../ai/com/ops/hops/v1alpha1/psqlcluster.k | 664 ------------------ .../dev/meta/v1alpha1/compositiontest.k | 113 --- .../io/upbound/dev/meta/v1alpha1/e2etest.k | 167 ----- .../upbound/dev/meta/v1alpha1/operationtest.k | 98 --- .../io/upbound/dev/meta/v1alpha1/project.k | 320 --------- .../io/upbound/dev/meta/v2alpha1/project.k | 325 --------- .../pkg/apis/meta/v1/object_meta.k | 97 --- .../pkg/apis/meta/v1/owner_reference.k | 41 -- tests/test-cluster/model/kcl.mod | 4 - 27 files changed, 123 insertions(+), 4154 deletions(-) create mode 120000 tests/test-branch/model delete mode 100644 tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k delete mode 100644 tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k delete mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k delete mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k delete mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k delete mode 100644 tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k delete mode 100644 tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k delete mode 100644 tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k delete mode 100644 tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k delete mode 100644 tests/test-branch/model/kcl.mod create mode 120000 tests/test-cluster/model delete mode 100644 tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k delete mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k delete mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k delete mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k delete mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k delete mode 100644 tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k delete mode 100644 tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k delete mode 100644 tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k delete mode 100644 tests/test-cluster/model/kcl.mod diff --git a/apis/psqlbranches/definition.yaml b/apis/psqlbranches/definition.yaml index 6730fdc..9420ff4 100644 --- a/apis/psqlbranches/definition.yaml +++ b/apis/psqlbranches/definition.yaml @@ -90,6 +90,21 @@ spec: description: VolumeSnapshotClass to use for the snapshot. Defaults to "psql" (composed by psql-stack). type: string default: psql + storage: + description: | + Source PVC sizing. The branch composition has no + automatic visibility into the source PSQLCluster's + storage shape, so when `branch.storage.size` is empty + the composition falls back to the size declared here. + Set this to the source's `spec.storage.size` so the + branch PVC is at least as large as the source — CNPG + can't restore a snapshot into a smaller PVC. + type: object + properties: + size: + description: Source PVC size, e.g. "100Gi". Used as the branch size when `branch.storage.size` is empty. + type: string + default: "" required: - name @@ -102,11 +117,16 @@ spec: type: integer default: 1 storage: - description: Branch PVC sizing. `size` must be ≥ the source PVC size for snapshot recovery to succeed. + description: Branch PVC sizing. `size` must be ≥ the source PVC size for snapshot recovery to succeed (CNPG/EBS can't shrink during recovery). type: object properties: size: - description: Branch PVC size, e.g. "10Gi". Empty = match source. + description: | + Branch PVC size, e.g. "10Gi". Empty = fall back to + `source.storage.size`. If both are empty the + composition omits storage.size on the Cluster CR + and CNPG's webhook will reject — set one or both + to a value at least as large as the source PVC. type: string default: "" class: diff --git a/examples/psqlbranches/same-namespace.yaml b/examples/psqlbranches/same-namespace.yaml index a0361ba..e0281c8 100644 --- a/examples/psqlbranches/same-namespace.yaml +++ b/examples/psqlbranches/same-namespace.yaml @@ -11,3 +11,7 @@ spec: clusterName: my-cluster source: name: my-app # source PSQLCluster in same namespace + storage: + size: 10Gi # mirror source's spec.storage.size — branch + # PVC must be ≥ source PVC for CNPG recovery + diff --git a/functions/branch/000-state-init.yaml.gotmpl b/functions/branch/000-state-init.yaml.gotmpl index a283da1..e61c9a1 100644 --- a/functions/branch/000-state-init.yaml.gotmpl +++ b/functions/branch/000-state-init.yaml.gotmpl @@ -42,11 +42,15 @@ {{- if not $sourcePvcName }} {{- $sourcePvcName = printf "%s-1" $sourceName }} {{- end }} +{{- $sourceStorageSpec := $sourceSpec.storage | default dict }} {{- $source := dict "name" $sourceName "namespace" $sourceNamespace "pvcName" $sourcePvcName "snapshotClassName" ($sourceSpec.snapshotClassName | default "psql") + "storage" (dict + "size" ($sourceStorageSpec.size | default "") + ) }} # Cross-namespace bridging gate — render the source-ns VolumeSnapshot only when @@ -58,17 +62,13 @@ # ============================================================================== {{- $branchSpec := $spec.branch | default dict }} {{- $branchStorageSpec := $branchSpec.storage | default dict }} -# CNPG requires spec.storage.size on the Cluster CR — default to "10Gi" when -# unspecified. Users with larger sources (>10Gi) MUST set this explicitly to -# at least the source PVC size; CNPG/EBS can't shrink during recovery. -{{- $branchSize := $branchStorageSpec.size | default "" }} -{{- if not $branchSize }} - {{- $branchSize = "10Gi" }} -{{- end }} +# Branch size — empty here is intentional. The cnpg-cluster template falls +# back to source.storage.size when the branch size is unset; the previous +# `default "10Gi"` was a footgun (sources >10Gi silently mis-sized). {{- $branch := dict "instances" ($branchSpec.instances | default 1) "storage" (dict - "size" $branchSize + "size" ($branchStorageSpec.size | default "") "class" ($branchStorageSpec.class | default "psql") ) }} diff --git a/functions/branch/200-cnpg-cluster.yaml.gotmpl b/functions/branch/200-cnpg-cluster.yaml.gotmpl index 1a62699..1e65b22 100644 --- a/functions/branch/200-cnpg-cluster.yaml.gotmpl +++ b/functions/branch/200-cnpg-cluster.yaml.gotmpl @@ -81,10 +81,19 @@ spec: {{- if $state.postgresql.version }} {{- $_ := set $clusterSpec "imageName" (printf "ghcr.io/cloudnative-pg/postgresql:%s" $state.postgresql.version) }} {{- end }} - {{- /* Storage — size always set (state-init defaults to 10Gi; consumer - must override for larger sources). class only set when explicitly - provided. */}} - {{- $storage := dict "size" $state.branch.storage.size }} + {{- /* Storage — size precedence: branch.storage.size > source.storage.size. + Both empty = omit storage.size on the Cluster CR; CNPG's webhook + rejects with a clear error rather than silently sizing wrong. + Hardcoding 10Gi here was a footgun for sources >10Gi (CNPG/EBS + can't shrink during recovery). class only set when provided. */}} + {{- $size := $state.branch.storage.size }} + {{- if not $size }} + {{- $size = $state.source.storage.size }} + {{- end }} + {{- $storage := dict }} + {{- if $size }} + {{- $_ := set $storage "size" $size }} + {{- end }} {{- if $state.branch.storage.class }} {{- $_ := set $storage "storageClass" $state.branch.storage.class }} {{- end }} diff --git a/tests/e2etest-psql/main.k b/tests/e2etest-psql/main.k index 669c9ae..e95370c 100644 --- a/tests/e2etest-psql/main.k +++ b/tests/e2etest-psql/main.k @@ -340,7 +340,17 @@ _items = [ "hops.ops.com.ai/e2etest" = "true" "hops.ops.com.ai/test-run" = _now } - source = {name = _cluster_name} + # source.storage.size mirrors the PSQLCluster's + # `spec.storage.size = "1Gi"` above. The branch + # composition has no automatic visibility into the + # source cluster's PVC capacity, so this explicit + # mirror lets the rendered Cluster CR set the + # required storage.size without forcing the consumer + # to also set `branch.storage.size`. + source = { + name = _cluster_name + storage = {size = "1Gi"} + } } } ] diff --git a/tests/test-branch/main.k b/tests/test-branch/main.k index 515898e..4d3a76c 100644 --- a/tests/test-branch/main.k +++ b/tests/test-branch/main.k @@ -200,6 +200,69 @@ _items = [ } } + # ========================================================================== + # Storage size precedence: branch.storage.size wins over source.storage.size. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "branch-storage-size-overrides-source-size" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-explicit", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = { + name = "src-app" + storage = {size = "100Gi"} + } + branch.storage.size = "150Gi" + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-explicit-cnpg-cluster" + spec.forProvider.manifest.spec.storage.size = "150Gi" + } + ] + } + } + + # ========================================================================== + # Storage size precedence: source.storage.size used when branch size empty. + # ========================================================================== + metav1alpha1.CompositionTest { + metadata.name = "branch-storage-size-falls-back-to-source-size" + spec = { + compositionPath = "apis/psqlbranches/composition.yaml" + xrdPath = "apis/psqlbranches/definition.yaml" + timeoutSeconds = 60 + validate = False + xr = stacksv1alpha1.PSQLBranch { + metadata = {name = "br-inherit", namespace = "default"} + spec = { + clusterName = "my-cluster" + source = { + name = "src-app" + storage = {size = "100Gi"} + } + } + } + assertResources = [ + { + apiVersion = "kubernetes.m.crossplane.io/v1alpha1" + kind = "Object" + metadata.name = "br-inherit-cnpg-cluster" + spec.forProvider.manifest.spec.storage.size = "100Gi" + } + ] + } + } + # ========================================================================== # Test 1b: explicit spec.postgresql.version sets imageName. Branches MUST # match the source's major version (snapshot recovery is binary-compatible diff --git a/tests/test-branch/model b/tests/test-branch/model new file mode 120000 index 0000000..faff6e4 --- /dev/null +++ b/tests/test-branch/model @@ -0,0 +1 @@ +../../.up/kcl/models \ No newline at end of file diff --git a/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k b/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k deleted file mode 100644 index 2b4276e..0000000 --- a/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlbranch.k +++ /dev/null @@ -1,481 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema PSQLBranch: - r""" - PSQLBranch is an ephemeral CoW fork of a PSQLCluster, designed for - preview environments (one branch per PR) and short-lived data - branches. Composes a CSI VolumeSnapshot of the source's PVC plus a - new CNPG Cluster bootstrapped via `bootstrap.recovery.volumeSnapshots`. - - Cross-namespace forks are supported (preview-pr-N namespace forking a - source in team-app namespace) via a bridging VolumeSnapshotContent — - the render template composes a VolumeSnapshot in the source namespace - AND a static-bound VolumeSnapshot in the branch namespace, then the - new Cluster references the branch-ns VolumeSnapshot for recovery. - - Same-namespace forks collapse this into a single VolumeSnapshot. - - Required prerequisites on the target cluster: psql-stack (CNPG - operator), volume-snapshot-stack (snapshot-controller + CRDs), and - a source PSQLCluster with `branching.enabled: true`. - - - Attributes - ---------- - apiVersion : str, default is "hops.ops.com.ai/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "PSQLBranch", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : HopsOpsComAiV1alpha1PSQLBranchSpec, default is Undefined, required - spec - status : HopsOpsComAiV1alpha1PSQLBranchStatus, default is Undefined, optional - status - """ - - - apiVersion: "hops.ops.com.ai/v1alpha1" = "hops.ops.com.ai/v1alpha1" - - kind: "PSQLBranch" = "PSQLBranch" - - metadata?: v1.ObjectMeta - - spec: HopsOpsComAiV1alpha1PSQLBranchSpec - - status?: HopsOpsComAiV1alpha1PSQLBranchStatus - - -schema HopsOpsComAiV1alpha1PSQLBranchSpec: - r""" - PSQLBranchSpec defines the desired state. - - Attributes - ---------- - branch : HopsOpsComAiV1alpha1PSQLBranchSpecBranch, default is Undefined, optional - branch - clusterName : str, default is Undefined, required - Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. - cnpg : HopsOpsComAiV1alpha1PSQLBranchSpecCnpg, default is Undefined, optional - cnpg - crossplane : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplane, default is Undefined, optional - crossplane - kubernetesProviderConfigRef : HopsOpsComAiV1alpha1PSQLBranchSpecKubernetesProviderConfigRef, default is Undefined, optional - kubernetes provider config ref - labels : {str:str}, default is Undefined, optional - Custom labels merged with stack defaults; applied to every composed resource. - managementPolicies : [str], default is ["*"], optional - Crossplane managementPolicies. Defaults to ["*"]. - postgresql : HopsOpsComAiV1alpha1PSQLBranchSpecPostgresql, default is Undefined, optional - postgresql - scaleToZero : HopsOpsComAiV1alpha1PSQLBranchSpecScaleToZero, default is Undefined, optional - scale to zero - source : HopsOpsComAiV1alpha1PSQLBranchSpecSource, default is Undefined, required - source - ttl : HopsOpsComAiV1alpha1PSQLBranchSpecTTL, default is Undefined, optional - ttl - """ - - - branch?: HopsOpsComAiV1alpha1PSQLBranchSpecBranch - - clusterName: str - - cnpg?: HopsOpsComAiV1alpha1PSQLBranchSpecCnpg - - crossplane?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplane - - kubernetesProviderConfigRef?: HopsOpsComAiV1alpha1PSQLBranchSpecKubernetesProviderConfigRef - - labels?: {str:str} - - managementPolicies?: [str] = ["*"] - - postgresql?: HopsOpsComAiV1alpha1PSQLBranchSpecPostgresql - - scaleToZero?: HopsOpsComAiV1alpha1PSQLBranchSpecScaleToZero - - source: HopsOpsComAiV1alpha1PSQLBranchSpecSource - - ttl?: HopsOpsComAiV1alpha1PSQLBranchSpecTTL - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecBranch: - r""" - Branch cluster sizing. - - Attributes - ---------- - instances : int, default is 1, optional - Branch instance count. Defaults to 1 (HA on branches is unusual). - storage : HopsOpsComAiV1alpha1PSQLBranchSpecBranchStorage, default is Undefined, optional - storage - """ - - - instances?: int = 1 - - storage?: HopsOpsComAiV1alpha1PSQLBranchSpecBranchStorage - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecBranchStorage: - r""" - Branch PVC sizing. Empty fields inherit from the source's storage on recovery. - - Attributes - ---------- - class : str, default is Undefined, optional - Branch StorageClass. Empty = inherit from source. - size : str, default is Undefined, optional - Branch PVC size, e.g. "10Gi". Empty = match source. - """ - - - class?: str = "" - - size?: str = "" - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCnpg: - r""" - Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not covered by the toggles above. - - Attributes - ---------- - overrideAllValues : any, default is Undefined, optional - Replaces the entire Cluster.spec. - values : any, default is Undefined, optional - Merged into the Cluster.spec. - """ - - - overrideAllValues?: any - - values?: any - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplane: - r""" - Configures how Crossplane will reconcile this composite resource - - Attributes - ---------- - compositionRef : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRef, default is Undefined, optional - composition ref - compositionRevisionRef : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionRef, default is Undefined, optional - composition revision ref - compositionRevisionSelector : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionSelector, default is Undefined, optional - composition revision selector - compositionSelector : HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionSelector, default is Undefined, optional - composition selector - compositionUpdatePolicy : str, default is Undefined, optional - composition update policy - resourceRefs : [HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneResourceRefsItems0], default is Undefined, optional - resource refs - """ - - - compositionRef?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRef - - compositionRevisionRef?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionRef - - compositionRevisionSelector?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionSelector - - compositionSelector?: HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionSelector - - compositionUpdatePolicy?: "Automatic" | "Manual" - - resourceRefs?: [HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneResourceRefsItems0] - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRef: - r""" - hops ops com ai v1alpha1 p SQL branch spec crossplane composition ref - - Attributes - ---------- - name : str, default is Undefined, required - name - """ - - - name: str - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionRef: - r""" - hops ops com ai v1alpha1 p SQL branch spec crossplane composition revision ref - - Attributes - ---------- - name : str, default is Undefined, required - name - """ - - - name: str - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionRevisionSelector: - r""" - hops ops com ai v1alpha1 p SQL branch spec crossplane composition revision selector - - Attributes - ---------- - matchLabels : {str:str}, default is Undefined, required - match labels - """ - - - matchLabels: {str:str} - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneCompositionSelector: - r""" - hops ops com ai v1alpha1 p SQL branch spec crossplane composition selector - - Attributes - ---------- - matchLabels : {str:str}, default is Undefined, required - match labels - """ - - - matchLabels: {str:str} - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecCrossplaneResourceRefsItems0: - r""" - hops ops com ai v1alpha1 p SQL branch spec crossplane resource refs items0 - - Attributes - ---------- - apiVersion : str, default is Undefined, required - api version - kind : str, default is Undefined, required - kind - name : str, default is Undefined, optional - name - """ - - - apiVersion: str - - kind: str - - name?: str - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecKubernetesProviderConfigRef: - r""" - Reference to the Kubernetes ProviderConfig used to apply composed resources. Defaults to clusterName. - - Attributes - ---------- - kind : str, default is Undefined, optional - kind - name : str, default is Undefined, optional - name - """ - - - kind?: "ProviderConfig" | "ClusterProviderConfig" - - name?: str - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecPostgresql: - r""" - Postgres version on the branch (must match source for snapshot recovery to succeed). - - Attributes - ---------- - version : str, default is "17", optional - version - """ - - - version?: str = "17" - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecScaleToZero: - r""" - Auto-hibernate the branch after `idleTimeout` of no client - activity. Default-on with aggressive timeout — preview-PR - branches should hibernate fast to keep cost flat. - - - Attributes - ---------- - enabled : bool, default is True, optional - enabled - idleTimeout : str, default is "10m", optional - Idle duration before hibernate. Defaults to "10m" (more aggressive than PSQLCluster's 30m). - """ - - - enabled?: bool = True - - idleTimeout?: str = "10m" - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecSource: - r""" - Source PSQLCluster to fork. Cross-namespace supported — leave - `namespace` empty for same-namespace branching, or set it to - the team's namespace for preview-PR-style forks. - - - Attributes - ---------- - name : str, default is Undefined, required - Source PSQLCluster name (required). - namespace : str, default is Undefined, optional - Source PSQLCluster namespace. Empty = same as branch. - pvcName : str, default is Undefined, optional - Source PVC name to snapshot. Empty = derive from CNPG - naming convention (`-1` for the primary - instance). - - snapshotClassName : str, default is "psql", optional - VolumeSnapshotClass to use for the snapshot. Defaults to "psql" (composed by psql-stack). - """ - - - name: str - - namespace?: str = "" - - pvcName?: str = "" - - snapshotClassName?: str = "psql" - - -schema HopsOpsComAiV1alpha1PSQLBranchSpecTTL: - r""" - Recorded intent for branch expiration. v0 does NOT enforce - the TTL automatically — pair with an external cleanup - mechanism (Kyverno cleanupPolicies, ArgoCD ApplicationSet - with pullRequest generator that deletes on PR close, etc.). - Status reports `expiresAt` (creation + after) for tooling. - - - Attributes - ---------- - after : str, default is "168h", optional - Duration after which the branch should be deleted. Defaults to "168h" (7 days). - enabled : bool, default is Undefined, optional - enabled - """ - - - after?: str = "168h" - - enabled?: bool = False - - -schema HopsOpsComAiV1alpha1PSQLBranchStatus: - r""" - PSQLBranchStatus defines the observed state. - - Attributes - ---------- - bootstrapMethod : str, default is Undefined, optional - Bootstrap method actually used. v0 always "volumeSnapshot". - bootstrapPhase : str, default is Undefined, optional - Current branch CNPG Cluster `.status.phase`. - conditions : [HopsOpsComAiV1alpha1PSQLBranchStatusConditionsItems0], default is Undefined, optional - Conditions of the resource. - crossplane : HopsOpsComAiV1alpha1PSQLBranchStatusCrossplane, default is Undefined, optional - crossplane - expiresAt : str, default is Undefined, optional - Computed deletion deadline when ttl.enabled is true. - ready : bool, default is Undefined, optional - ready - sourceSnapshotContent : str, default is Undefined, optional - Name of the cluster-scoped VolumeSnapshotContent backing this branch (for cross-ns bridging visibility). - """ - - - bootstrapMethod?: str - - bootstrapPhase?: str - - conditions?: [HopsOpsComAiV1alpha1PSQLBranchStatusConditionsItems0] - - crossplane?: HopsOpsComAiV1alpha1PSQLBranchStatusCrossplane - - expiresAt?: str - - ready?: bool - - sourceSnapshotContent?: str - - -schema HopsOpsComAiV1alpha1PSQLBranchStatusConditionsItems0: - r""" - hops ops com ai v1alpha1 p SQL branch status conditions items0 - - Attributes - ---------- - lastTransitionTime : str, default is Undefined, required - last transition time - message : str, default is Undefined, optional - message - observedGeneration : int, default is Undefined, optional - observed generation - reason : str, default is Undefined, required - reason - status : str, default is Undefined, required - status - $type : str, default is Undefined, required - type - """ - - - lastTransitionTime: str - - message?: str - - observedGeneration?: int - - reason: str - - status: str - - $type: str - - -schema HopsOpsComAiV1alpha1PSQLBranchStatusCrossplane: - r""" - Indicates how Crossplane is reconciling this composite resource - - Attributes - ---------- - connectionDetails : HopsOpsComAiV1alpha1PSQLBranchStatusCrossplaneConnectionDetails, default is Undefined, optional - connection details - """ - - - connectionDetails?: HopsOpsComAiV1alpha1PSQLBranchStatusCrossplaneConnectionDetails - - -schema HopsOpsComAiV1alpha1PSQLBranchStatusCrossplaneConnectionDetails: - r""" - hops ops com ai v1alpha1 p SQL branch status crossplane connection details - - Attributes - ---------- - lastPublishedTime : str, default is Undefined, optional - last published time - """ - - - lastPublishedTime?: str - - diff --git a/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k b/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k deleted file mode 100644 index 123fca7..0000000 --- a/tests/test-branch/model/ai/com/ops/hops/v1alpha1/psqlcluster.k +++ /dev/null @@ -1,664 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema PSQLCluster: - r""" - PSQLCluster is the per-app Postgres primitive. Composes a CloudNativePG - `Cluster` plus the wiring (annotations, plugin entries, labels) required - for the platform integrations (scale-to-zero, branching, monitoring, HA). - - Companion to `psql-stack` (which provides the CNPG operator + S2Z plugin - substrate; required as a target-cluster prerequisite) and to `psql-branch` - (which composes ephemeral CoW forks of a PSQLCluster via VolumeSnapshot). - - Intent-first surface: integration toggles (`scaleToZero.enabled`, - `branching.enabled`, `monitoring.enabled`, `ha.enabled`) translate to - CNPG mechanics inside the render template — consumer manifests don't - reach for `cnpg.overrideAllValues` unless their use case is genuinely - novel. - - - Attributes - ---------- - apiVersion : str, default is "hops.ops.com.ai/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "PSQLCluster", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : HopsOpsComAiV1alpha1PSQLClusterSpec, default is Undefined, required - spec - status : HopsOpsComAiV1alpha1PSQLClusterStatus, default is Undefined, optional - status - """ - - - apiVersion: "hops.ops.com.ai/v1alpha1" = "hops.ops.com.ai/v1alpha1" - - kind: "PSQLCluster" = "PSQLCluster" - - metadata?: v1.ObjectMeta - - spec: HopsOpsComAiV1alpha1PSQLClusterSpec - - status?: HopsOpsComAiV1alpha1PSQLClusterStatus - - -schema HopsOpsComAiV1alpha1PSQLClusterSpec: - r""" - PSQLClusterSpec defines the desired state. - - Attributes - ---------- - app : HopsOpsComAiV1alpha1PSQLClusterSpecApp, default is Undefined, optional - app - branching : HopsOpsComAiV1alpha1PSQLClusterSpecBranching, default is Undefined, optional - branching - clusterName : str, default is Undefined, required - Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. - cnpg : HopsOpsComAiV1alpha1PSQLClusterSpecCnpg, default is Undefined, optional - cnpg - crossplane : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane, default is Undefined, optional - crossplane - ha : HopsOpsComAiV1alpha1PSQLClusterSpecHa, default is Undefined, optional - ha - instances : int, default is 1, optional - Number of Postgres instances. Default 1 (single-node). HA mode bumps to ha.replicas. - kubernetesProviderConfigRef : HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef, default is Undefined, optional - kubernetes provider config ref - labels : {str:str}, default is Undefined, optional - Custom labels merged with stack defaults; applied to every composed resource. - managementPolicies : [str], default is ["*"], optional - Crossplane managementPolicies applied to every composed resource. Defaults to ["*"]. - monitoring : HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring, default is Undefined, optional - monitoring - postgresql : HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql, default is Undefined, optional - postgresql - scaleToZero : HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero, default is Undefined, optional - scale to zero - storage : HopsOpsComAiV1alpha1PSQLClusterSpecStorage, default is Undefined, required - storage - superuser : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser, default is Undefined, optional - superuser - """ - - - app?: HopsOpsComAiV1alpha1PSQLClusterSpecApp - - branching?: HopsOpsComAiV1alpha1PSQLClusterSpecBranching - - clusterName: str - - cnpg?: HopsOpsComAiV1alpha1PSQLClusterSpecCnpg - - crossplane?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane - - ha?: HopsOpsComAiV1alpha1PSQLClusterSpecHa - - instances?: int = 1 - - kubernetesProviderConfigRef?: HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef - - labels?: {str:str} - - managementPolicies?: [str] = ["*"] - - monitoring?: HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring - - postgresql?: HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql - - scaleToZero?: HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero - - storage: HopsOpsComAiV1alpha1PSQLClusterSpecStorage - - superuser?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecApp: - r""" - Application user — the Postgres role your app connects as. CNPG uses these credentials for `bootstrap.initdb`. - - Attributes - ---------- - database : str, default is "app", optional - Application database name. Defaults to "app". - externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret, default is Undefined, optional - external secret - role : str, default is "app", optional - Postgres role name owning the application database. Defaults to "app". - secretName : str, default is Undefined, optional - K8s Secret holding the app role's credentials. Defaults to "-app". With `externalSecret` set, the composition writes this Secret; otherwise pre-create it yourself. - """ - - - database?: str = "app" - - externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret - - role?: str = "app" - - secretName?: str = "" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret: - r""" - Source the credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. - - Attributes - ---------- - secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef, default is Undefined, required - secret ref - secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore, default is Undefined, required - secret store - """ - - - secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef - - secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef: - r""" - Where in the secrets backend the credentials live. The remote value must be a JSON blob with `username` and `password` properties — both are extracted into the resulting K8s Secret. - - Attributes - ---------- - path : str, default is Undefined, required - Path/key in the secrets backend (e.g. AWS Secrets Manager `my-cluster/app`). - """ - - - path: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore: - r""" - Reference to the SecretStore or ClusterSecretStore providing the value. - - Attributes - ---------- - kind : str, default is "ClusterSecretStore", optional - Store kind. Use ClusterSecretStore for cluster-scoped stores (the platform default), SecretStore for namespaced stores in this XR's namespace. - name : str, default is Undefined, required - Name of the SecretStore or ClusterSecretStore. - namespace : str, default is Undefined, optional - Namespace of the SecretStore (ignored for ClusterSecretStore). Defaults to the PSQLCluster's namespace. - """ - - - kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" - - name: str - - namespace?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecBranching: - r""" - Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). - - Attributes - ---------- - enabled : bool, default is True, optional - enabled - snapshotClassName : str, default is "psql", optional - VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. - """ - - - enabled?: bool = True - - snapshotClassName?: str = "psql" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCnpg: - r""" - Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. - - Attributes - ---------- - overrideAllValues : any, default is Undefined, optional - Replaces the entire Cluster.spec — total escape hatch. - values : any, default is Undefined, optional - Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. - """ - - - overrideAllValues?: any - - values?: any - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane: - r""" - Configures how Crossplane will reconcile this composite resource - - Attributes - ---------- - compositionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef, default is Undefined, optional - composition ref - compositionRevisionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef, default is Undefined, optional - composition revision ref - compositionRevisionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector, default is Undefined, optional - composition revision selector - compositionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector, default is Undefined, optional - composition selector - compositionUpdatePolicy : str, default is Undefined, optional - composition update policy - resourceRefs : [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0], default is Undefined, optional - resource refs - """ - - - compositionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef - - compositionRevisionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef - - compositionRevisionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector - - compositionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector - - compositionUpdatePolicy?: "Automatic" | "Manual" - - resourceRefs?: [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0] - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition ref - - Attributes - ---------- - name : str, default is Undefined, required - name - """ - - - name: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision ref - - Attributes - ---------- - name : str, default is Undefined, required - name - """ - - - name: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision selector - - Attributes - ---------- - matchLabels : {str:str}, default is Undefined, required - match labels - """ - - - matchLabels: {str:str} - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition selector - - Attributes - ---------- - matchLabels : {str:str}, default is Undefined, required - match labels - """ - - - matchLabels: {str:str} - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane resource refs items0 - - Attributes - ---------- - apiVersion : str, default is Undefined, required - api version - kind : str, default is Undefined, required - kind - name : str, default is Undefined, optional - name - """ - - - apiVersion: str - - kind: str - - name?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecHa: - r""" - Stack-wide HA. Bumps instances to `replicas` (default 3) and adds zonal topology spread. - - Attributes - ---------- - enabled : bool, default is Undefined, optional - enabled - replicas : int, default is 3, optional - replicas - topologySpreadByZone : bool, default is True, optional - topology spread by zone - """ - - - enabled?: bool = False - - replicas?: int = 3 - - topologySpreadByZone?: bool = True - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef: - r""" - Reference to the Kubernetes ProviderConfig used to apply the Cluster CR + ExternalSecret. Defaults to clusterName. - - Attributes - ---------- - kind : str, default is Undefined, optional - kind - name : str, default is Undefined, optional - name - """ - - - kind?: "ProviderConfig" | "ClusterProviderConfig" - - name?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring: - r""" - Add Prometheus scrape configuration. CNPG operator handles the actual PodMonitor creation when `monitoring.enablePodMonitor` is set on the Cluster CR. - - Attributes - ---------- - enabled : bool, default is True, optional - enabled - """ - - - enabled?: bool = True - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql: - r""" - Postgres version + tuning parameters. - - Attributes - ---------- - parameters : {str:str}, default is Undefined, optional - postgres.conf parameters (e.g. `shared_buffers`, `max_connections`). - version : str, default is "17", optional - Postgres major version. Defaults to "17". - """ - - - parameters?: {str:str} - - version?: str = "17" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero: - r""" - Auto-hibernate the cluster after `idleTimeout` of no client activity. Requires the cnpg-i-scale-to-zero plugin on the target cluster (provided by psql-stack). When `ha.enabled` is also true, the cluster will not actually scale to zero (HA implies always-available); a Warning condition is emitted on the XR but the annotation is rendered for tooling that reads it. - - Attributes - ---------- - enabled : bool, default is Undefined, optional - enabled - idleTimeout : str, default is "30m", optional - How long the cluster must be idle before hibernating. Defaults to "30m". - """ - - - enabled?: bool = False - - idleTimeout?: str = "30m" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecStorage: - r""" - Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (gp3 default on EKS Auto Mode) — bumping `size` upward grows the volume in place with no downtime. - - Attributes - ---------- - class : str, default is Undefined, optional - StorageClass name. Empty = cluster default (gp3 on Auto Mode). - size : str, default is Undefined, required - PVC size, e.g. "10Gi". Required. - """ - - - class?: str = "" - - size: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser: - r""" - Optional postgres superuser credentials. Omit the block to let CNPG auto-generate the superuser secret. - - Attributes - ---------- - externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret, default is Undefined, optional - external secret - secretName : str, default is Undefined, optional - K8s Secret holding the superuser credentials. Defaults to "-superuser". - """ - - - externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret - - secretName?: str = "" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret: - r""" - Source the superuser credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. - - Attributes - ---------- - secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef, default is Undefined, required - secret ref - secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore, default is Undefined, required - secret store - """ - - - secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef - - secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef: - r""" - hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret ref - - Attributes - ---------- - path : str, default is Undefined, required - path - """ - - - path: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore: - r""" - hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret store - - Attributes - ---------- - kind : str, default is "ClusterSecretStore", optional - kind - name : str, default is Undefined, required - name - namespace : str, default is Undefined, optional - namespace - """ - - - kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" - - name: str - - namespace?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatus: - r""" - PSQLClusterStatus defines the observed state. - - Attributes - ---------- - app : HopsOpsComAiV1alpha1PSQLClusterStatusApp, default is Undefined, optional - app - clusterPhase : str, default is Undefined, optional - Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). - conditions : [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0], default is Undefined, optional - Conditions of the resource. - crossplane : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane, default is Undefined, optional - crossplane - ready : bool, default is Undefined, optional - Overall readiness — true once the CNPG Cluster reaches `Cluster in healthy state` AND the ExternalSecret (if composed) is SecretSynced. - superuser : HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser, default is Undefined, optional - superuser - """ - - - app?: HopsOpsComAiV1alpha1PSQLClusterStatusApp - - clusterPhase?: str - - conditions?: [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0] - - crossplane?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane - - ready?: bool - - superuser?: HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusApp: - r""" - Connection details for the application user. Other XRs can reference this to wire connection strings without hardcoding. - - Attributes - ---------- - database : str, default is Undefined, optional - Application database name. - host : str, default is Undefined, optional - Service hostname for read-write connections (CNPG `-rw` service). - port : int, default is Undefined, optional - Postgres port. - secretName : str, default is Undefined, optional - K8s Secret holding the app role's credentials. - """ - - - database?: str - - host?: str - - port?: int - - secretName?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0: - r""" - hops ops com ai v1alpha1 p SQL cluster status conditions items0 - - Attributes - ---------- - lastTransitionTime : str, default is Undefined, required - last transition time - message : str, default is Undefined, optional - message - observedGeneration : int, default is Undefined, optional - observed generation - reason : str, default is Undefined, required - reason - status : str, default is Undefined, required - status - $type : str, default is Undefined, required - type - """ - - - lastTransitionTime: str - - message?: str - - observedGeneration?: int - - reason: str - - status: str - - $type: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane: - r""" - Indicates how Crossplane is reconciling this composite resource - - Attributes - ---------- - connectionDetails : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails, default is Undefined, optional - connection details - """ - - - connectionDetails?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails: - r""" - hops ops com ai v1alpha1 p SQL cluster status crossplane connection details - - Attributes - ---------- - lastPublishedTime : str, default is Undefined, optional - last published time - """ - - - lastPublishedTime?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser: - r""" - Connection details for the postgres superuser. - - Attributes - ---------- - secretName : str, default is Undefined, optional - K8s Secret holding the superuser credentials (CNPG-managed if `spec.superuser` was not set). - """ - - - secretName?: str - - diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k deleted file mode 100644 index 82fd889..0000000 --- a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/compositiontest.k +++ /dev/null @@ -1,113 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema CompositionTest: - r""" - CompositionTest defines the schema for the CompositionTest custom resource. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "CompositionTest", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV1alpha1CompositionTestSpec, default is Undefined, required - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "CompositionTest" = "CompositionTest" - - metadata?: v1.ObjectMeta - - spec: MetaDevUpboundIoV1alpha1CompositionTestSpec - - -schema MetaDevUpboundIoV1alpha1CompositionTestSpec: - r""" - CompositionTestSpec defines the specification for the CompositionTest custom resource. - - Attributes - ---------- - assertResources : [any], default is Undefined, optional - AssertResources defines assertions to validate resources after test completion. - Optional. - composition : any, default is Undefined, optional - Composition specifies the composition definition inline. - Optional. - compositionPath : str, default is Undefined, optional - Composition specifies the composition definition path. - Optional. - context : {str:any}, default is Undefined, optional - Context specifies context for the Function Pipeline inline as key-value pairs. - Keys are context keys, values are JSON data. - Optional. - extraResources : [any], default is Undefined, optional - ExtraResources specifies additional resources inline. - Optional. - functionCredentialsPath : str, default is Undefined, optional - FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. - Optional. - observedResources : [any], default is Undefined, optional - ObservedResources specifies additional observed resources inline. - Optional. - timeoutSeconds : int, default is 30, required - Timeout for the test in seconds - Required. Default is 30s. - validate : bool, default is Undefined, optional - Validate indicates whether to validate managed resources against schemas. - Optional. - xr : any, default is Undefined, optional - XR specifies the composite resource (XR) inline. - Mutually exclusive with XRPath. At least one of XR or XRPath must be specified. - xrPath : str, default is Undefined, optional - XRPath specifies the composite resource (XR) path. - Mutually exclusive with XR. At least one of XR or XRPath must be specified. - xrd : any, default is Undefined, optional - XRD specifies the XRD definition inline. - Optional. - xrdPath : str, default is Undefined, optional - XRD specifies the XRD definition path. - Optional. - """ - - - assertResources?: [any] - - composition?: any - - compositionPath?: str - - context?: {str:any} - - extraResources?: [any] - - functionCredentialsPath?: str - - observedResources?: [any] - - timeoutSeconds: int = 30 - - validate?: bool - - xr?: any - - xrPath?: str - - xrd?: any - - xrdPath?: str - - - check: - timeoutSeconds >= 1 - - diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k deleted file mode 100644 index bccda70..0000000 --- a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/e2etest.k +++ /dev/null @@ -1,167 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema E2ETest: - r""" - E2ETest defines the schema for the E2ETest custom resource used for e2e - testing of Crossplane configurations in controlplanes. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "E2ETest", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV1alpha1E2ETestSpec, default is Undefined, required - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "E2ETest" = "E2ETest" - - metadata?: v1.ObjectMeta - - spec: MetaDevUpboundIoV1alpha1E2ETestSpec - - -schema MetaDevUpboundIoV1alpha1E2ETestSpec: - r""" - E2ETestSpec defines the specification for e2e testing of Crossplane - configurations. It orchestrates the complete test lifecycle including setting - up controlplane, applying test resources in the correct order (InitResources - → Configuration → ExtraResources → Manifests), validating conditions, and - handling cleanup. This spec allows you to define e2e tests that verify your - Crossplane compositions, providers, and managed resources work correctly - together in a real controlplane environment. - - Attributes - ---------- - cleanupTimeoutSeconds : int, default is 600, optional - CleanupTimeoutSeconds defines the maximum duration in seconds for cleanup - operations after the test completes. This timeout applies to the deletion - of test resources and any associated managed resources. If not specified, - defaults to 600 seconds (10 minutes). Consider increasing this value for - tests with many resources or complex deletion dependencies. - crossplane : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane, default is Undefined, required - crossplane - defaultConditions : [str], default is Undefined, optional - DefaultConditions specifies the expected conditions that should be met - after the manifests are applied. These are validation checks that verify - the resources are functioning correctly. Each condition is a string - expression that will be evaluated against the deployed resources. Common - conditions include checking resource status for readiness - extraResources : [any], default is Undefined, optional - ExtraResources specifies additional Kubernetes resources that should be - created or updated after the configuration has been successfully applied. - These resources may depend on the primary configuration being in place. - Common use cases include ConfigMaps, Secrets, providerConfigs. Each - resource must be a valid Kubernetes object. - initResources : [any], default is Undefined, optional - InitResources specifies Kubernetes resources that must be created or - updated before the configuration is applied. These are typically - prerequisite resources that the configuration depends on. Common use - cases include ImageConfigs, DeploymentRuntimeConfigs, or any foundational - resources required for the configuration to work. Each resource must be a - valid Kubernetes object. - manifests : [any], default is Undefined, required - Manifests contains the Kubernetes resources that will be applied as part - of this e2e test. These are the primary resources being tested - they - will be created in the controlplane and then validated against the - conditions specified in DefaultConditions. Each manifest must be a valid - Kubernetes object. At least one manifest is required. Examples include - Claims, Composite Resources or any Kubernetes resource you want to test. - skipDelete : bool, default is Undefined, optional - If true, skip resource deletion after test - timeoutSeconds : int, default is Undefined, optional - TimeoutSeconds defines the maximum duration in seconds that the test is - allowed to run before being marked as failed. This includes time for - resource creation, condition checks, and any reconciliation processes. If - not specified, a default timeout will be used. Consider setting higher - values for tests involving complex resources or those requiring multiple - reconciliation cycles. - """ - - - cleanupTimeoutSeconds?: int = 600 - - crossplane: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane - - defaultConditions?: [str] - - extraResources?: [any] - - initResources?: [any] - - manifests: [any] - - skipDelete?: bool - - timeoutSeconds?: int - - - check: - cleanupTimeoutSeconds >= 1 if cleanupTimeoutSeconds not in [None, Undefined] - len(defaultConditions) >= 1 if defaultConditions - len(manifests) >= 1 - timeoutSeconds >= 1 if timeoutSeconds not in [None, Undefined] - - -schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane: - r""" - Crossplane specifies the Crossplane configuration and settings required - for this test. This includes the version of Universal Crossplane to - install, and optional auto-upgrade settings. The configuration defined - here will be used to set up the controlplane before applying the test - manifests. - - Attributes - ---------- - autoUpgrade : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade, default is Undefined, optional - auto upgrade - state : str, default is "Running", optional - State defines the state for crossplane and provider workloads. We support - the following states where 'Running' is the default: - - Running: Starts/Scales up all crossplane and provider workloads in the ControlPlane - - Paused: Pauses/Scales down all crossplane and provider workloads in the ControlPlane - version : str, default is Undefined, optional - Version is the version of Universal Crossplane to install. - """ - - - autoUpgrade?: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade - - state?: "Running" | "Paused" = "Running" - - version?: str - - -schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade: - r""" - AutoUpgrades defines the auto upgrade configuration for Crossplane. - - Attributes - ---------- - channel : str, default is "Stable", optional - Channel defines the upgrade channels for Crossplane. We support the following channels where 'Stable' is the - default: - - None: disables auto-upgrades and keeps the control plane at its current version of Crossplane. - - Patch: automatically upgrades the control plane to the latest supported patch version when it - becomes available while keeping the minor version the same. - - Stable: automatically upgrades the control plane to the latest supported patch release on minor - version N-1, where N is the latest supported minor version. - - Rapid: automatically upgrades the cluster to the latest supported patch release on the latest - supported minor version. - """ - - - channel?: "None" | "Patch" | "Stable" | "Rapid" = "Stable" - - diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k deleted file mode 100644 index f9e8c95..0000000 --- a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/operationtest.k +++ /dev/null @@ -1,98 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema OperationTest: - r""" - OperationTest defines the schema for the OperationTest custom resource. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "OperationTest", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, required - metadata - spec : MetaDevUpboundIoV1alpha1OperationTestSpec, default is Undefined, required - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "OperationTest" = "OperationTest" - - metadata: v1.ObjectMeta - - spec: MetaDevUpboundIoV1alpha1OperationTestSpec - - -schema MetaDevUpboundIoV1alpha1OperationTestSpec: - r""" - OperationTestSpec defines the specification for the OperationTest custom resource. - - Attributes - ---------- - assertResources : [any], default is Undefined, optional - AssertResources defines assertions to validate resources after test completion. - Optional. - context : {str:any}, default is Undefined, optional - Context specifies context for the Function Pipeline inline as key-value pairs. - Keys are context keys, values are JSON data. - Optional. - functionCredentialsPath : str, default is Undefined, optional - FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. - Optional. - operation : any, default is Undefined, required - Operation specifies the Operation definition inline. - Optional. - operationPath : str, default is Undefined, optional - OperationPath specifies the XRD definition path. - Optional. - requiredResources : [any], default is Undefined, optional - RequiredResources specifies additional required resources inline. - Optional. - requiredResourcesPath : str, default is Undefined, optional - RequiredResourcesPath specifies a path to required resources file. - Optional. - timeoutSeconds : int, default is 30, required - Timeout for the test in seconds - Required. Default is 30s. - watchedResource : any, default is Undefined, optional - WatchedResource specifies additional watched resource inline. - Optional. - watchedResourcePath : str, default is Undefined, optional - WatchedResourcePath specifies a path to watched resource file. - Optional. - """ - - - assertResources?: [any] - - context?: {str:any} - - functionCredentialsPath?: str - - operation: any - - operationPath?: str - - requiredResources?: [any] - - requiredResourcesPath?: str - - timeoutSeconds: int = 30 - - watchedResource?: any - - watchedResourcePath?: str - - - check: - timeoutSeconds >= 1 - - diff --git a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k b/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k deleted file mode 100644 index 334e249..0000000 --- a/tests/test-branch/model/io/upbound/dev/meta/v1alpha1/project.k +++ /dev/null @@ -1,320 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema Project: - r""" - Project defines an Upbound Project, which can be built into a Crossplane - Configuration. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "Project", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV1alpha1ProjectSpec, default is Undefined, optional - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "Project" = "Project" - - metadata?: v1.ObjectMeta - - spec?: MetaDevUpboundIoV1alpha1ProjectSpec - - -schema MetaDevUpboundIoV1alpha1ProjectSpec: - r""" - ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes - resource there is no Status, only Spec. - - Attributes - ---------- - apiDependencies : [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional - APIDependencies are the API dependencies for this project. - NOTE: This is an experimental feature and is subject to change. - architectures : [str], default is Undefined, optional - architectures - crossplane : MetaDevUpboundIoV1alpha1ProjectSpecCrossplane, default is Undefined, optional - crossplane - dependsOn : [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0], default is Undefined, optional - depends on - description : str, default is Undefined, optional - description - imageConfig : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0], default is Undefined, optional - image config - license : str, default is Undefined, optional - license - maintainer : str, default is Undefined, optional - maintainer - paths : MetaDevUpboundIoV1alpha1ProjectSpecPaths, default is Undefined, optional - paths - readme : str, default is Undefined, optional - readme - repository : str, default is Undefined, required - repository - source : str, default is Undefined, optional - source - """ - - - apiDependencies?: [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0] - - architectures?: [str] - - crossplane?: MetaDevUpboundIoV1alpha1ProjectSpecCrossplane - - dependsOn?: [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0] - - description?: str - - imageConfig?: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0] - - license?: str - - maintainer?: str - - paths?: MetaDevUpboundIoV1alpha1ProjectSpecPaths - - readme?: str - - repository: str - - source?: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0: - r""" - APIDependencies defines a reference to an external API dependency. - NOTE: This is an experimental feature and is subject to change. - - Attributes - ---------- - git : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional - git - http : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional - http - k8s : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional - k8s - $type : str, default is Undefined, required - Type defines the type of API dependency. - """ - - - git?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git - - http?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP - - k8s?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s - - $type: "k8s" | "crd" - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git: - r""" - Git defines the git repository source for the API dependency. - - Attributes - ---------- - path : str, default is Undefined, optional - Path is the path within the repository to the API definition. - ref : str, default is Undefined, optional - Ref is the git reference (branch, tag, or commit SHA). - repository : str, default is Undefined, required - Repository is the git repository URL. - """ - - - path?: str - - ref?: str - - repository: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP: - r""" - HTTP defines the HTTP source for the API dependency. - - Attributes - ---------- - url : str, default is Undefined, required - URL is the HTTP/HTTPS URL to fetch the API dependency from. - """ - - - url: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s: - r""" - K8s defines the Kubernetes API version for the dependency. - - Attributes - ---------- - version : str, default is Undefined, required - Version is the Kubernetes API version (e.g., "v1.33.0"). - """ - - - version: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecCrossplane: - r""" - CrossplaneConstraints specifies a packages compatibility with Crossplane versions. - - Attributes - ---------- - version : str, default is Undefined, required - Semantic version constraints of Crossplane that package is compatible with. - """ - - - version: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0: - r""" - Dependency is a dependency on another package. A dependency can be of an - arbitrary API version and kind, but Crossplane expects package dependencies - to behave like a Crossplane package. Specifically it expects to be able to - create the dependency and set its spec.package field to a package OCI - reference. - - Attributes - ---------- - apiVersion : str, default is Undefined, optional - APIVersion of the dependency. - configuration : str, default is Undefined, optional - Configuration is the name of a Configuration package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - function : str, default is Undefined, optional - Function is the name of a Function package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - kind : str, default is Undefined, optional - Kind of the dependency. - package : str, default is Undefined, optional - Package OCI reference of the dependency. Only used when apiVersion and - kind are set. - Must be a fully qualified image name, including the registry, - repository, and tag. For example, "registry.example.com/repo/package:tag". - provider : str, default is Undefined, optional - Provider is the name of a Provider package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion and kind instead. - version : str, default is Undefined, required - Version is the semantic version constraints of the dependency image. - """ - - - apiVersion?: str - - configuration?: str - - function?: str - - kind?: str - - package?: str - - provider?: str - - version: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0: - r""" - ImageConfig defines a set of rules for matching and rewriting images. - - Attributes - ---------- - matchImages : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required - MatchImages is a list of image matching rules that should be satisfied. - rewriteImage : MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required - rewrite image - """ - - - matchImages: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0] - - rewriteImage: MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage - - -schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0: - r""" - ImageMatch defines a rule for matching image. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix that should be matched. - $type : str, default is "Prefix", optional - Type is the type of match. - """ - - - prefix: str - - $type?: "Prefix" = "Prefix" - - -schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage: - r""" - RewriteImage defines how a matched image should be rewritten. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix to use when rewriting the image. - """ - - - prefix: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecPaths: - r""" - ProjectPaths configures the locations of various parts of the project, for - use at build time. - - Attributes - ---------- - apis : str, default is Undefined, optional - APIs is the directory holding the project's apis. If not - specified, it defaults to `apis/`. - examples : str, default is Undefined, optional - Examples is the directory holding the project's examples. If not - specified, it defaults to `examples/`. - functions : str, default is Undefined, optional - Functions is the directory holding the project's functions. If not - specified, it defaults to `functions/`. - tests : str, default is Undefined, optional - Tests is the directory holding the project's tests. If not - specified, it defaults to `tests/`. - """ - - - apis?: str - - examples?: str - - functions?: str - - tests?: str - - diff --git a/tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k b/tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k deleted file mode 100644 index 5eb6272..0000000 --- a/tests/test-branch/model/io/upbound/dev/meta/v2alpha1/project.k +++ /dev/null @@ -1,325 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema Project: - r""" - Project defines an Upbound Project, which can be built into a Crossplane - Configuration. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v2alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "Project", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV2alpha1ProjectSpec, default is Undefined, optional - spec - """ - - - apiVersion: "meta.dev.upbound.io/v2alpha1" = "meta.dev.upbound.io/v2alpha1" - - kind: "Project" = "Project" - - metadata?: v1.ObjectMeta - - spec?: MetaDevUpboundIoV2alpha1ProjectSpec - - -schema MetaDevUpboundIoV2alpha1ProjectSpec: - r""" - ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes - resource there is no Status, only Spec. - - Attributes - ---------- - apiDependencies : [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional - APIDependencies are the API dependencies for this project. - NOTE: This is an experimental feature and is subject to change. - architectures : [str], default is Undefined, optional - architectures - crossplane : MetaDevUpboundIoV2alpha1ProjectSpecCrossplane, default is Undefined, optional - crossplane - dependsOn : [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0], default is Undefined, optional - depends on - description : str, default is Undefined, optional - description - imageConfig : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0], default is Undefined, optional - image config - license : str, default is Undefined, optional - license - maintainer : str, default is Undefined, optional - maintainer - paths : MetaDevUpboundIoV2alpha1ProjectSpecPaths, default is Undefined, optional - paths - readme : str, default is Undefined, optional - readme - repository : str, default is Undefined, required - repository - source : str, default is Undefined, optional - source - """ - - - apiDependencies?: [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0] - - architectures?: [str] - - crossplane?: MetaDevUpboundIoV2alpha1ProjectSpecCrossplane - - dependsOn?: [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0] - - description?: str - - imageConfig?: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0] - - license?: str - - maintainer?: str - - paths?: MetaDevUpboundIoV2alpha1ProjectSpecPaths - - readme?: str - - repository: str - - source?: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0: - r""" - APIDependencies defines a reference to an external API dependency. - NOTE: This is an experimental feature and is subject to change. - - Attributes - ---------- - git : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional - git - http : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional - http - k8s : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional - k8s - $type : str, default is Undefined, required - Type defines the type of API dependency. - """ - - - git?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git - - http?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP - - k8s?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s - - $type: "k8s" | "crd" - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git: - r""" - Git defines the git repository source for the API dependency. - - Attributes - ---------- - path : str, default is Undefined, optional - Path is the path within the repository to the API definition. - ref : str, default is Undefined, optional - Ref is the git reference (branch, tag, or commit SHA). - repository : str, default is Undefined, required - Repository is the git repository URL. - """ - - - path?: str - - ref?: str - - repository: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP: - r""" - HTTP defines the HTTP source for the API dependency. - - Attributes - ---------- - url : str, default is Undefined, required - URL is the HTTP/HTTPS URL to fetch the API dependency from. - """ - - - url: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s: - r""" - K8s defines the Kubernetes API version for the dependency. - - Attributes - ---------- - version : str, default is Undefined, required - Version is the Kubernetes API version (e.g., "v1.33.0"). - """ - - - version: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecCrossplane: - r""" - CrossplaneConstraints specifies a packages compatibility with Crossplane versions. - - Attributes - ---------- - version : str, default is Undefined, required - Semantic version constraints of Crossplane that package is compatible with. - """ - - - version: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0: - r""" - Dependency is a dependency on another package. A dependency can be of an - arbitrary API version and kind, but Crossplane expects package dependencies - to behave like a Crossplane package. Specifically it expects to be able to - create the dependency and set its spec.package field to a package OCI - reference. - - Attributes - ---------- - apiVersion : str, default is Undefined, optional - APIVersion of the dependency. - configuration : str, default is Undefined, optional - Configuration is the name of a Configuration package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - function : str, default is Undefined, optional - Function is the name of a Function package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - kind : str, default is Undefined, optional - Kind of the dependency. - package : str, default is Undefined, optional - Package OCI reference of the dependency. Only used when apiVersion and - kind are set. - Must be a fully qualified image name, including the registry, - repository, and tag. For example, "registry.example.com/repo/package:tag". - provider : str, default is Undefined, optional - Provider is the name of a Provider package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion and kind instead. - version : str, default is Undefined, required - Version is the semantic version constraints of the dependency image. - """ - - - apiVersion?: str - - configuration?: str - - function?: str - - kind?: str - - package?: str - - provider?: str - - version: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0: - r""" - ImageConfig defines a set of rules for matching and rewriting images. - - Attributes - ---------- - matchImages : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required - MatchImages is a list of image matching rules that should be satisfied. - rewriteImage : MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required - rewrite image - """ - - - matchImages: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0] - - rewriteImage: MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage - - -schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0: - r""" - ImageMatch defines a rule for matching image. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix that should be matched. - $type : str, default is "Prefix", optional - Type is the type of match. - """ - - - prefix: str - - $type?: "Prefix" = "Prefix" - - -schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage: - r""" - RewriteImage defines how a matched image should be rewritten. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix to use when rewriting the image. - """ - - - prefix: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecPaths: - r""" - ProjectPaths configures the locations of various parts of the project, for - use at build time. - - Attributes - ---------- - apis : str, default is Undefined, optional - APIs is the directory holding the project's apis. If not - specified, it defaults to `apis/`. - examples : str, default is Undefined, optional - Examples is the directory holding the project's examples. If not - specified, it defaults to `examples/`. - functions : str, default is Undefined, optional - Functions is the directory holding the project's functions. If not - specified, it defaults to `functions/`. - operations : str, default is Undefined, optional - Operations is the directory holding the project's operations. If not - specified, it defaults to `operations/`. - tests : str, default is Undefined, optional - Tests is the directory holding the project's tests. If not - specified, it defaults to `tests/`. - """ - - - apis?: str - - examples?: str - - functions?: str - - operations?: str - - tests?: str - - diff --git a/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k b/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k deleted file mode 100644 index 0705904..0000000 --- a/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k +++ /dev/null @@ -1,97 +0,0 @@ -""" -This is the object_meta module in k8s.apimachinery.pkg.apis.meta.v1 package. -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" - - -schema ObjectMeta: - r""" - ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create. - - Attributes - ---------- - annotations : {str:str}, default is Undefined, optional - Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations - clusterName : str, default is Undefined, optional - The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request. - creationTimestamp : str, default is Undefined, optional - CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. - - Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata - deletionGracePeriodSeconds : int, default is Undefined, optional - Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. - deletionTimestamp : str, default is Undefined, optional - DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested. - - Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata - finalizers : [str], default is Undefined, optional - Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. - generateName : str, default is Undefined, optional - GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server. - - If this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). - - Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency - generation : int, default is Undefined, optional - A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. - labels : {str:str}, default is Undefined, optional - Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels - managedFields : [ManagedFieldsEntry], default is Undefined, optional - ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. - name : str, default is Undefined, optional - Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names - namespace : str, default is Undefined, optional - Namespace defines the space within each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty. - - Must be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces - ownerReferences : [OwnerReference], default is Undefined, optional - List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. - resourceVersion : str, default is Undefined, optional - An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources. - - Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency - selfLink : str, default is Undefined, optional - SelfLink is a URL representing this object. Populated by the system. Read-only. - - DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release. - uid : str, default is Undefined, optional - UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations. - - Populated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids - """ - - - annotations?: {str:str} - - clusterName?: str - - creationTimestamp?: str - - deletionGracePeriodSeconds?: int - - deletionTimestamp?: str - - finalizers?: [str] - - generateName?: str - - generation?: int - - labels?: {str:str} - - managedFields?: any - - name?: str - - namespace?: str - - ownerReferences?: [OwnerReference] - - resourceVersion?: str - - selfLink?: str - - uid?: str - - diff --git a/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k b/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k deleted file mode 100644 index 3854a5c..0000000 --- a/tests/test-branch/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k +++ /dev/null @@ -1,41 +0,0 @@ -""" -This is the owner_reference module in k8s.apimachinery.pkg.apis.meta.v1 package. -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" - - -schema OwnerReference: - r""" - OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field. - - Attributes - ---------- - apiVersion : str, default is Undefined, required - API version of the referent. - blockOwnerDeletion : bool, default is Undefined, optional - If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. - controller : bool, default is Undefined, optional - If true, this reference points to the managing controller. - kind : str, default is Undefined, required - Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - name : str, default is Undefined, required - Name of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#names - uid : str, default is Undefined, required - UID of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#uids - """ - - - apiVersion: str - - blockOwnerDeletion?: bool - - controller?: bool - - kind: str - - name: str - - uid: str - - diff --git a/tests/test-branch/model/kcl.mod b/tests/test-branch/model/kcl.mod deleted file mode 100644 index 9df41d7..0000000 --- a/tests/test-branch/model/kcl.mod +++ /dev/null @@ -1,4 +0,0 @@ -[package] -name = "models" -edition = "v0.11.2" -version = "0.0.1" diff --git a/tests/test-cluster/model b/tests/test-cluster/model new file mode 120000 index 0000000..faff6e4 --- /dev/null +++ b/tests/test-cluster/model @@ -0,0 +1 @@ +../../.up/kcl/models \ No newline at end of file diff --git a/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k b/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k deleted file mode 100644 index 123fca7..0000000 --- a/tests/test-cluster/model/ai/com/ops/hops/v1alpha1/psqlcluster.k +++ /dev/null @@ -1,664 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema PSQLCluster: - r""" - PSQLCluster is the per-app Postgres primitive. Composes a CloudNativePG - `Cluster` plus the wiring (annotations, plugin entries, labels) required - for the platform integrations (scale-to-zero, branching, monitoring, HA). - - Companion to `psql-stack` (which provides the CNPG operator + S2Z plugin - substrate; required as a target-cluster prerequisite) and to `psql-branch` - (which composes ephemeral CoW forks of a PSQLCluster via VolumeSnapshot). - - Intent-first surface: integration toggles (`scaleToZero.enabled`, - `branching.enabled`, `monitoring.enabled`, `ha.enabled`) translate to - CNPG mechanics inside the render template — consumer manifests don't - reach for `cnpg.overrideAllValues` unless their use case is genuinely - novel. - - - Attributes - ---------- - apiVersion : str, default is "hops.ops.com.ai/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "PSQLCluster", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : HopsOpsComAiV1alpha1PSQLClusterSpec, default is Undefined, required - spec - status : HopsOpsComAiV1alpha1PSQLClusterStatus, default is Undefined, optional - status - """ - - - apiVersion: "hops.ops.com.ai/v1alpha1" = "hops.ops.com.ai/v1alpha1" - - kind: "PSQLCluster" = "PSQLCluster" - - metadata?: v1.ObjectMeta - - spec: HopsOpsComAiV1alpha1PSQLClusterSpec - - status?: HopsOpsComAiV1alpha1PSQLClusterStatus - - -schema HopsOpsComAiV1alpha1PSQLClusterSpec: - r""" - PSQLClusterSpec defines the desired state. - - Attributes - ---------- - app : HopsOpsComAiV1alpha1PSQLClusterSpecApp, default is Undefined, optional - app - branching : HopsOpsComAiV1alpha1PSQLClusterSpecBranching, default is Undefined, optional - branching - clusterName : str, default is Undefined, required - Name of the target cluster. Used as default for kubernetesProviderConfigRef.name and label values. - cnpg : HopsOpsComAiV1alpha1PSQLClusterSpecCnpg, default is Undefined, optional - cnpg - crossplane : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane, default is Undefined, optional - crossplane - ha : HopsOpsComAiV1alpha1PSQLClusterSpecHa, default is Undefined, optional - ha - instances : int, default is 1, optional - Number of Postgres instances. Default 1 (single-node). HA mode bumps to ha.replicas. - kubernetesProviderConfigRef : HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef, default is Undefined, optional - kubernetes provider config ref - labels : {str:str}, default is Undefined, optional - Custom labels merged with stack defaults; applied to every composed resource. - managementPolicies : [str], default is ["*"], optional - Crossplane managementPolicies applied to every composed resource. Defaults to ["*"]. - monitoring : HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring, default is Undefined, optional - monitoring - postgresql : HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql, default is Undefined, optional - postgresql - scaleToZero : HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero, default is Undefined, optional - scale to zero - storage : HopsOpsComAiV1alpha1PSQLClusterSpecStorage, default is Undefined, required - storage - superuser : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser, default is Undefined, optional - superuser - """ - - - app?: HopsOpsComAiV1alpha1PSQLClusterSpecApp - - branching?: HopsOpsComAiV1alpha1PSQLClusterSpecBranching - - clusterName: str - - cnpg?: HopsOpsComAiV1alpha1PSQLClusterSpecCnpg - - crossplane?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane - - ha?: HopsOpsComAiV1alpha1PSQLClusterSpecHa - - instances?: int = 1 - - kubernetesProviderConfigRef?: HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef - - labels?: {str:str} - - managementPolicies?: [str] = ["*"] - - monitoring?: HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring - - postgresql?: HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql - - scaleToZero?: HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero - - storage: HopsOpsComAiV1alpha1PSQLClusterSpecStorage - - superuser?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecApp: - r""" - Application user — the Postgres role your app connects as. CNPG uses these credentials for `bootstrap.initdb`. - - Attributes - ---------- - database : str, default is "app", optional - Application database name. Defaults to "app". - externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret, default is Undefined, optional - external secret - role : str, default is "app", optional - Postgres role name owning the application database. Defaults to "app". - secretName : str, default is Undefined, optional - K8s Secret holding the app role's credentials. Defaults to "-app". With `externalSecret` set, the composition writes this Secret; otherwise pre-create it yourself. - """ - - - database?: str = "app" - - externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret - - role?: str = "app" - - secretName?: str = "" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecret: - r""" - Source the credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. - - Attributes - ---------- - secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef, default is Undefined, required - secret ref - secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore, default is Undefined, required - secret store - """ - - - secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef - - secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretRef: - r""" - Where in the secrets backend the credentials live. The remote value must be a JSON blob with `username` and `password` properties — both are extracted into the resulting K8s Secret. - - Attributes - ---------- - path : str, default is Undefined, required - Path/key in the secrets backend (e.g. AWS Secrets Manager `my-cluster/app`). - """ - - - path: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecAppExternalSecretSecretStore: - r""" - Reference to the SecretStore or ClusterSecretStore providing the value. - - Attributes - ---------- - kind : str, default is "ClusterSecretStore", optional - Store kind. Use ClusterSecretStore for cluster-scoped stores (the platform default), SecretStore for namespaced stores in this XR's namespace. - name : str, default is Undefined, required - Name of the SecretStore or ClusterSecretStore. - namespace : str, default is Undefined, optional - Namespace of the SecretStore (ignored for ClusterSecretStore). Defaults to the PSQLCluster's namespace. - """ - - - kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" - - name: str - - namespace?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecBranching: - r""" - Mark this cluster as a fork-source. PSQLBranch finds branchable clusters by reading the labels emitted here. Default-on (cheap when unused — just metadata). - - Attributes - ---------- - enabled : bool, default is True, optional - enabled - snapshotClassName : str, default is "psql", optional - VolumeSnapshotClass name composed by psql-stack (default "psql"). PSQLBranch references this name to fork the cluster's PVC. - """ - - - enabled?: bool = True - - snapshotClassName?: str = "psql" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCnpg: - r""" - Direct passthrough into the composed Cluster CR's spec. Use for novel CNPG features not yet covered by the intent-first surface above. - - Attributes - ---------- - overrideAllValues : any, default is Undefined, optional - Replaces the entire Cluster.spec — total escape hatch. - values : any, default is Undefined, optional - Merged into the Cluster.spec — adds to / overrides individual fields rendered by the template. - """ - - - overrideAllValues?: any - - values?: any - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplane: - r""" - Configures how Crossplane will reconcile this composite resource - - Attributes - ---------- - compositionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef, default is Undefined, optional - composition ref - compositionRevisionRef : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef, default is Undefined, optional - composition revision ref - compositionRevisionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector, default is Undefined, optional - composition revision selector - compositionSelector : HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector, default is Undefined, optional - composition selector - compositionUpdatePolicy : str, default is Undefined, optional - composition update policy - resourceRefs : [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0], default is Undefined, optional - resource refs - """ - - - compositionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef - - compositionRevisionRef?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef - - compositionRevisionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector - - compositionSelector?: HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector - - compositionUpdatePolicy?: "Automatic" | "Manual" - - resourceRefs?: [HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0] - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRef: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition ref - - Attributes - ---------- - name : str, default is Undefined, required - name - """ - - - name: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionRef: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision ref - - Attributes - ---------- - name : str, default is Undefined, required - name - """ - - - name: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionRevisionSelector: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition revision selector - - Attributes - ---------- - matchLabels : {str:str}, default is Undefined, required - match labels - """ - - - matchLabels: {str:str} - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneCompositionSelector: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane composition selector - - Attributes - ---------- - matchLabels : {str:str}, default is Undefined, required - match labels - """ - - - matchLabels: {str:str} - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecCrossplaneResourceRefsItems0: - r""" - hops ops com ai v1alpha1 p SQL cluster spec crossplane resource refs items0 - - Attributes - ---------- - apiVersion : str, default is Undefined, required - api version - kind : str, default is Undefined, required - kind - name : str, default is Undefined, optional - name - """ - - - apiVersion: str - - kind: str - - name?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecHa: - r""" - Stack-wide HA. Bumps instances to `replicas` (default 3) and adds zonal topology spread. - - Attributes - ---------- - enabled : bool, default is Undefined, optional - enabled - replicas : int, default is 3, optional - replicas - topologySpreadByZone : bool, default is True, optional - topology spread by zone - """ - - - enabled?: bool = False - - replicas?: int = 3 - - topologySpreadByZone?: bool = True - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecKubernetesProviderConfigRef: - r""" - Reference to the Kubernetes ProviderConfig used to apply the Cluster CR + ExternalSecret. Defaults to clusterName. - - Attributes - ---------- - kind : str, default is Undefined, optional - kind - name : str, default is Undefined, optional - name - """ - - - kind?: "ProviderConfig" | "ClusterProviderConfig" - - name?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecMonitoring: - r""" - Add Prometheus scrape configuration. CNPG operator handles the actual PodMonitor creation when `monitoring.enablePodMonitor` is set on the Cluster CR. - - Attributes - ---------- - enabled : bool, default is True, optional - enabled - """ - - - enabled?: bool = True - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecPostgresql: - r""" - Postgres version + tuning parameters. - - Attributes - ---------- - parameters : {str:str}, default is Undefined, optional - postgres.conf parameters (e.g. `shared_buffers`, `max_connections`). - version : str, default is "17", optional - Postgres major version. Defaults to "17". - """ - - - parameters?: {str:str} - - version?: str = "17" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecScaleToZero: - r""" - Auto-hibernate the cluster after `idleTimeout` of no client activity. Requires the cnpg-i-scale-to-zero plugin on the target cluster (provided by psql-stack). When `ha.enabled` is also true, the cluster will not actually scale to zero (HA implies always-available); a Warning condition is emitted on the XR but the annotation is rendered for tooling that reads it. - - Attributes - ---------- - enabled : bool, default is Undefined, optional - enabled - idleTimeout : str, default is "30m", optional - How long the cluster must be idle before hibernating. Defaults to "30m". - """ - - - enabled?: bool = False - - idleTimeout?: str = "30m" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecStorage: - r""" - Postgres data PVC sizing. Online expansion is supported by CNPG when allowVolumeExpansion is true on the StorageClass (gp3 default on EKS Auto Mode) — bumping `size` upward grows the volume in place with no downtime. - - Attributes - ---------- - class : str, default is Undefined, optional - StorageClass name. Empty = cluster default (gp3 on Auto Mode). - size : str, default is Undefined, required - PVC size, e.g. "10Gi". Required. - """ - - - class?: str = "" - - size: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuser: - r""" - Optional postgres superuser credentials. Omit the block to let CNPG auto-generate the superuser secret. - - Attributes - ---------- - externalSecret : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret, default is Undefined, optional - external secret - secretName : str, default is Undefined, optional - K8s Secret holding the superuser credentials. Defaults to "-superuser". - """ - - - externalSecret?: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret - - secretName?: str = "" - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecret: - r""" - Source the superuser credentials via External Secrets Operator. Omit to BYO the K8s Secret named `secretName`. - - Attributes - ---------- - secretRef : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef, default is Undefined, required - secret ref - secretStore : HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore, default is Undefined, required - secret store - """ - - - secretRef: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef - - secretStore: HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretRef: - r""" - hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret ref - - Attributes - ---------- - path : str, default is Undefined, required - path - """ - - - path: str - - -schema HopsOpsComAiV1alpha1PSQLClusterSpecSuperuserExternalSecretSecretStore: - r""" - hops ops com ai v1alpha1 p SQL cluster spec superuser external secret secret store - - Attributes - ---------- - kind : str, default is "ClusterSecretStore", optional - kind - name : str, default is Undefined, required - name - namespace : str, default is Undefined, optional - namespace - """ - - - kind?: "ClusterSecretStore" | "SecretStore" = "ClusterSecretStore" - - name: str - - namespace?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatus: - r""" - PSQLClusterStatus defines the observed state. - - Attributes - ---------- - app : HopsOpsComAiV1alpha1PSQLClusterStatusApp, default is Undefined, optional - app - clusterPhase : str, default is Undefined, optional - Current CNPG Cluster `.status.phase` (e.g. "Cluster in healthy state"). - conditions : [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0], default is Undefined, optional - Conditions of the resource. - crossplane : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane, default is Undefined, optional - crossplane - ready : bool, default is Undefined, optional - Overall readiness — true once the CNPG Cluster reaches `Cluster in healthy state` AND the ExternalSecret (if composed) is SecretSynced. - superuser : HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser, default is Undefined, optional - superuser - """ - - - app?: HopsOpsComAiV1alpha1PSQLClusterStatusApp - - clusterPhase?: str - - conditions?: [HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0] - - crossplane?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane - - ready?: bool - - superuser?: HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusApp: - r""" - Connection details for the application user. Other XRs can reference this to wire connection strings without hardcoding. - - Attributes - ---------- - database : str, default is Undefined, optional - Application database name. - host : str, default is Undefined, optional - Service hostname for read-write connections (CNPG `-rw` service). - port : int, default is Undefined, optional - Postgres port. - secretName : str, default is Undefined, optional - K8s Secret holding the app role's credentials. - """ - - - database?: str - - host?: str - - port?: int - - secretName?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusConditionsItems0: - r""" - hops ops com ai v1alpha1 p SQL cluster status conditions items0 - - Attributes - ---------- - lastTransitionTime : str, default is Undefined, required - last transition time - message : str, default is Undefined, optional - message - observedGeneration : int, default is Undefined, optional - observed generation - reason : str, default is Undefined, required - reason - status : str, default is Undefined, required - status - $type : str, default is Undefined, required - type - """ - - - lastTransitionTime: str - - message?: str - - observedGeneration?: int - - reason: str - - status: str - - $type: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplane: - r""" - Indicates how Crossplane is reconciling this composite resource - - Attributes - ---------- - connectionDetails : HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails, default is Undefined, optional - connection details - """ - - - connectionDetails?: HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusCrossplaneConnectionDetails: - r""" - hops ops com ai v1alpha1 p SQL cluster status crossplane connection details - - Attributes - ---------- - lastPublishedTime : str, default is Undefined, optional - last published time - """ - - - lastPublishedTime?: str - - -schema HopsOpsComAiV1alpha1PSQLClusterStatusSuperuser: - r""" - Connection details for the postgres superuser. - - Attributes - ---------- - secretName : str, default is Undefined, optional - K8s Secret holding the superuser credentials (CNPG-managed if `spec.superuser` was not set). - """ - - - secretName?: str - - diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k deleted file mode 100644 index 82fd889..0000000 --- a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/compositiontest.k +++ /dev/null @@ -1,113 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema CompositionTest: - r""" - CompositionTest defines the schema for the CompositionTest custom resource. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "CompositionTest", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV1alpha1CompositionTestSpec, default is Undefined, required - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "CompositionTest" = "CompositionTest" - - metadata?: v1.ObjectMeta - - spec: MetaDevUpboundIoV1alpha1CompositionTestSpec - - -schema MetaDevUpboundIoV1alpha1CompositionTestSpec: - r""" - CompositionTestSpec defines the specification for the CompositionTest custom resource. - - Attributes - ---------- - assertResources : [any], default is Undefined, optional - AssertResources defines assertions to validate resources after test completion. - Optional. - composition : any, default is Undefined, optional - Composition specifies the composition definition inline. - Optional. - compositionPath : str, default is Undefined, optional - Composition specifies the composition definition path. - Optional. - context : {str:any}, default is Undefined, optional - Context specifies context for the Function Pipeline inline as key-value pairs. - Keys are context keys, values are JSON data. - Optional. - extraResources : [any], default is Undefined, optional - ExtraResources specifies additional resources inline. - Optional. - functionCredentialsPath : str, default is Undefined, optional - FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. - Optional. - observedResources : [any], default is Undefined, optional - ObservedResources specifies additional observed resources inline. - Optional. - timeoutSeconds : int, default is 30, required - Timeout for the test in seconds - Required. Default is 30s. - validate : bool, default is Undefined, optional - Validate indicates whether to validate managed resources against schemas. - Optional. - xr : any, default is Undefined, optional - XR specifies the composite resource (XR) inline. - Mutually exclusive with XRPath. At least one of XR or XRPath must be specified. - xrPath : str, default is Undefined, optional - XRPath specifies the composite resource (XR) path. - Mutually exclusive with XR. At least one of XR or XRPath must be specified. - xrd : any, default is Undefined, optional - XRD specifies the XRD definition inline. - Optional. - xrdPath : str, default is Undefined, optional - XRD specifies the XRD definition path. - Optional. - """ - - - assertResources?: [any] - - composition?: any - - compositionPath?: str - - context?: {str:any} - - extraResources?: [any] - - functionCredentialsPath?: str - - observedResources?: [any] - - timeoutSeconds: int = 30 - - validate?: bool - - xr?: any - - xrPath?: str - - xrd?: any - - xrdPath?: str - - - check: - timeoutSeconds >= 1 - - diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k deleted file mode 100644 index bccda70..0000000 --- a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/e2etest.k +++ /dev/null @@ -1,167 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema E2ETest: - r""" - E2ETest defines the schema for the E2ETest custom resource used for e2e - testing of Crossplane configurations in controlplanes. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "E2ETest", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV1alpha1E2ETestSpec, default is Undefined, required - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "E2ETest" = "E2ETest" - - metadata?: v1.ObjectMeta - - spec: MetaDevUpboundIoV1alpha1E2ETestSpec - - -schema MetaDevUpboundIoV1alpha1E2ETestSpec: - r""" - E2ETestSpec defines the specification for e2e testing of Crossplane - configurations. It orchestrates the complete test lifecycle including setting - up controlplane, applying test resources in the correct order (InitResources - → Configuration → ExtraResources → Manifests), validating conditions, and - handling cleanup. This spec allows you to define e2e tests that verify your - Crossplane compositions, providers, and managed resources work correctly - together in a real controlplane environment. - - Attributes - ---------- - cleanupTimeoutSeconds : int, default is 600, optional - CleanupTimeoutSeconds defines the maximum duration in seconds for cleanup - operations after the test completes. This timeout applies to the deletion - of test resources and any associated managed resources. If not specified, - defaults to 600 seconds (10 minutes). Consider increasing this value for - tests with many resources or complex deletion dependencies. - crossplane : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane, default is Undefined, required - crossplane - defaultConditions : [str], default is Undefined, optional - DefaultConditions specifies the expected conditions that should be met - after the manifests are applied. These are validation checks that verify - the resources are functioning correctly. Each condition is a string - expression that will be evaluated against the deployed resources. Common - conditions include checking resource status for readiness - extraResources : [any], default is Undefined, optional - ExtraResources specifies additional Kubernetes resources that should be - created or updated after the configuration has been successfully applied. - These resources may depend on the primary configuration being in place. - Common use cases include ConfigMaps, Secrets, providerConfigs. Each - resource must be a valid Kubernetes object. - initResources : [any], default is Undefined, optional - InitResources specifies Kubernetes resources that must be created or - updated before the configuration is applied. These are typically - prerequisite resources that the configuration depends on. Common use - cases include ImageConfigs, DeploymentRuntimeConfigs, or any foundational - resources required for the configuration to work. Each resource must be a - valid Kubernetes object. - manifests : [any], default is Undefined, required - Manifests contains the Kubernetes resources that will be applied as part - of this e2e test. These are the primary resources being tested - they - will be created in the controlplane and then validated against the - conditions specified in DefaultConditions. Each manifest must be a valid - Kubernetes object. At least one manifest is required. Examples include - Claims, Composite Resources or any Kubernetes resource you want to test. - skipDelete : bool, default is Undefined, optional - If true, skip resource deletion after test - timeoutSeconds : int, default is Undefined, optional - TimeoutSeconds defines the maximum duration in seconds that the test is - allowed to run before being marked as failed. This includes time for - resource creation, condition checks, and any reconciliation processes. If - not specified, a default timeout will be used. Consider setting higher - values for tests involving complex resources or those requiring multiple - reconciliation cycles. - """ - - - cleanupTimeoutSeconds?: int = 600 - - crossplane: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane - - defaultConditions?: [str] - - extraResources?: [any] - - initResources?: [any] - - manifests: [any] - - skipDelete?: bool - - timeoutSeconds?: int - - - check: - cleanupTimeoutSeconds >= 1 if cleanupTimeoutSeconds not in [None, Undefined] - len(defaultConditions) >= 1 if defaultConditions - len(manifests) >= 1 - timeoutSeconds >= 1 if timeoutSeconds not in [None, Undefined] - - -schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplane: - r""" - Crossplane specifies the Crossplane configuration and settings required - for this test. This includes the version of Universal Crossplane to - install, and optional auto-upgrade settings. The configuration defined - here will be used to set up the controlplane before applying the test - manifests. - - Attributes - ---------- - autoUpgrade : MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade, default is Undefined, optional - auto upgrade - state : str, default is "Running", optional - State defines the state for crossplane and provider workloads. We support - the following states where 'Running' is the default: - - Running: Starts/Scales up all crossplane and provider workloads in the ControlPlane - - Paused: Pauses/Scales down all crossplane and provider workloads in the ControlPlane - version : str, default is Undefined, optional - Version is the version of Universal Crossplane to install. - """ - - - autoUpgrade?: MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade - - state?: "Running" | "Paused" = "Running" - - version?: str - - -schema MetaDevUpboundIoV1alpha1E2ETestSpecCrossplaneAutoUpgrade: - r""" - AutoUpgrades defines the auto upgrade configuration for Crossplane. - - Attributes - ---------- - channel : str, default is "Stable", optional - Channel defines the upgrade channels for Crossplane. We support the following channels where 'Stable' is the - default: - - None: disables auto-upgrades and keeps the control plane at its current version of Crossplane. - - Patch: automatically upgrades the control plane to the latest supported patch version when it - becomes available while keeping the minor version the same. - - Stable: automatically upgrades the control plane to the latest supported patch release on minor - version N-1, where N is the latest supported minor version. - - Rapid: automatically upgrades the cluster to the latest supported patch release on the latest - supported minor version. - """ - - - channel?: "None" | "Patch" | "Stable" | "Rapid" = "Stable" - - diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k deleted file mode 100644 index f9e8c95..0000000 --- a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/operationtest.k +++ /dev/null @@ -1,98 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema OperationTest: - r""" - OperationTest defines the schema for the OperationTest custom resource. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "OperationTest", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, required - metadata - spec : MetaDevUpboundIoV1alpha1OperationTestSpec, default is Undefined, required - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "OperationTest" = "OperationTest" - - metadata: v1.ObjectMeta - - spec: MetaDevUpboundIoV1alpha1OperationTestSpec - - -schema MetaDevUpboundIoV1alpha1OperationTestSpec: - r""" - OperationTestSpec defines the specification for the OperationTest custom resource. - - Attributes - ---------- - assertResources : [any], default is Undefined, optional - AssertResources defines assertions to validate resources after test completion. - Optional. - context : {str:any}, default is Undefined, optional - Context specifies context for the Function Pipeline inline as key-value pairs. - Keys are context keys, values are JSON data. - Optional. - functionCredentialsPath : str, default is Undefined, optional - FunctionCredentialsPath specifies a path to a credentials file to be passed to tests. - Optional. - operation : any, default is Undefined, required - Operation specifies the Operation definition inline. - Optional. - operationPath : str, default is Undefined, optional - OperationPath specifies the XRD definition path. - Optional. - requiredResources : [any], default is Undefined, optional - RequiredResources specifies additional required resources inline. - Optional. - requiredResourcesPath : str, default is Undefined, optional - RequiredResourcesPath specifies a path to required resources file. - Optional. - timeoutSeconds : int, default is 30, required - Timeout for the test in seconds - Required. Default is 30s. - watchedResource : any, default is Undefined, optional - WatchedResource specifies additional watched resource inline. - Optional. - watchedResourcePath : str, default is Undefined, optional - WatchedResourcePath specifies a path to watched resource file. - Optional. - """ - - - assertResources?: [any] - - context?: {str:any} - - functionCredentialsPath?: str - - operation: any - - operationPath?: str - - requiredResources?: [any] - - requiredResourcesPath?: str - - timeoutSeconds: int = 30 - - watchedResource?: any - - watchedResourcePath?: str - - - check: - timeoutSeconds >= 1 - - diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k b/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k deleted file mode 100644 index 334e249..0000000 --- a/tests/test-cluster/model/io/upbound/dev/meta/v1alpha1/project.k +++ /dev/null @@ -1,320 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema Project: - r""" - Project defines an Upbound Project, which can be built into a Crossplane - Configuration. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v1alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "Project", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV1alpha1ProjectSpec, default is Undefined, optional - spec - """ - - - apiVersion: "meta.dev.upbound.io/v1alpha1" = "meta.dev.upbound.io/v1alpha1" - - kind: "Project" = "Project" - - metadata?: v1.ObjectMeta - - spec?: MetaDevUpboundIoV1alpha1ProjectSpec - - -schema MetaDevUpboundIoV1alpha1ProjectSpec: - r""" - ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes - resource there is no Status, only Spec. - - Attributes - ---------- - apiDependencies : [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional - APIDependencies are the API dependencies for this project. - NOTE: This is an experimental feature and is subject to change. - architectures : [str], default is Undefined, optional - architectures - crossplane : MetaDevUpboundIoV1alpha1ProjectSpecCrossplane, default is Undefined, optional - crossplane - dependsOn : [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0], default is Undefined, optional - depends on - description : str, default is Undefined, optional - description - imageConfig : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0], default is Undefined, optional - image config - license : str, default is Undefined, optional - license - maintainer : str, default is Undefined, optional - maintainer - paths : MetaDevUpboundIoV1alpha1ProjectSpecPaths, default is Undefined, optional - paths - readme : str, default is Undefined, optional - readme - repository : str, default is Undefined, required - repository - source : str, default is Undefined, optional - source - """ - - - apiDependencies?: [MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0] - - architectures?: [str] - - crossplane?: MetaDevUpboundIoV1alpha1ProjectSpecCrossplane - - dependsOn?: [MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0] - - description?: str - - imageConfig?: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0] - - license?: str - - maintainer?: str - - paths?: MetaDevUpboundIoV1alpha1ProjectSpecPaths - - readme?: str - - repository: str - - source?: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0: - r""" - APIDependencies defines a reference to an external API dependency. - NOTE: This is an experimental feature and is subject to change. - - Attributes - ---------- - git : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional - git - http : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional - http - k8s : MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional - k8s - $type : str, default is Undefined, required - Type defines the type of API dependency. - """ - - - git?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git - - http?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP - - k8s?: MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s - - $type: "k8s" | "crd" - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0Git: - r""" - Git defines the git repository source for the API dependency. - - Attributes - ---------- - path : str, default is Undefined, optional - Path is the path within the repository to the API definition. - ref : str, default is Undefined, optional - Ref is the git reference (branch, tag, or commit SHA). - repository : str, default is Undefined, required - Repository is the git repository URL. - """ - - - path?: str - - ref?: str - - repository: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0HTTP: - r""" - HTTP defines the HTTP source for the API dependency. - - Attributes - ---------- - url : str, default is Undefined, required - URL is the HTTP/HTTPS URL to fetch the API dependency from. - """ - - - url: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecAPIDependenciesItems0K8s: - r""" - K8s defines the Kubernetes API version for the dependency. - - Attributes - ---------- - version : str, default is Undefined, required - Version is the Kubernetes API version (e.g., "v1.33.0"). - """ - - - version: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecCrossplane: - r""" - CrossplaneConstraints specifies a packages compatibility with Crossplane versions. - - Attributes - ---------- - version : str, default is Undefined, required - Semantic version constraints of Crossplane that package is compatible with. - """ - - - version: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecDependsOnItems0: - r""" - Dependency is a dependency on another package. A dependency can be of an - arbitrary API version and kind, but Crossplane expects package dependencies - to behave like a Crossplane package. Specifically it expects to be able to - create the dependency and set its spec.package field to a package OCI - reference. - - Attributes - ---------- - apiVersion : str, default is Undefined, optional - APIVersion of the dependency. - configuration : str, default is Undefined, optional - Configuration is the name of a Configuration package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - function : str, default is Undefined, optional - Function is the name of a Function package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - kind : str, default is Undefined, optional - Kind of the dependency. - package : str, default is Undefined, optional - Package OCI reference of the dependency. Only used when apiVersion and - kind are set. - Must be a fully qualified image name, including the registry, - repository, and tag. For example, "registry.example.com/repo/package:tag". - provider : str, default is Undefined, optional - Provider is the name of a Provider package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion and kind instead. - version : str, default is Undefined, required - Version is the semantic version constraints of the dependency image. - """ - - - apiVersion?: str - - configuration?: str - - function?: str - - kind?: str - - package?: str - - provider?: str - - version: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0: - r""" - ImageConfig defines a set of rules for matching and rewriting images. - - Attributes - ---------- - matchImages : [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required - MatchImages is a list of image matching rules that should be satisfied. - rewriteImage : MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required - rewrite image - """ - - - matchImages: [MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0] - - rewriteImage: MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage - - -schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0MatchImagesItems0: - r""" - ImageMatch defines a rule for matching image. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix that should be matched. - $type : str, default is "Prefix", optional - Type is the type of match. - """ - - - prefix: str - - $type?: "Prefix" = "Prefix" - - -schema MetaDevUpboundIoV1alpha1ProjectSpecImageConfigItems0RewriteImage: - r""" - RewriteImage defines how a matched image should be rewritten. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix to use when rewriting the image. - """ - - - prefix: str - - -schema MetaDevUpboundIoV1alpha1ProjectSpecPaths: - r""" - ProjectPaths configures the locations of various parts of the project, for - use at build time. - - Attributes - ---------- - apis : str, default is Undefined, optional - APIs is the directory holding the project's apis. If not - specified, it defaults to `apis/`. - examples : str, default is Undefined, optional - Examples is the directory holding the project's examples. If not - specified, it defaults to `examples/`. - functions : str, default is Undefined, optional - Functions is the directory holding the project's functions. If not - specified, it defaults to `functions/`. - tests : str, default is Undefined, optional - Tests is the directory holding the project's tests. If not - specified, it defaults to `tests/`. - """ - - - apis?: str - - examples?: str - - functions?: str - - tests?: str - - diff --git a/tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k b/tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k deleted file mode 100644 index 5eb6272..0000000 --- a/tests/test-cluster/model/io/upbound/dev/meta/v2alpha1/project.k +++ /dev/null @@ -1,325 +0,0 @@ -""" -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" -import k8s.apimachinery.pkg.apis.meta.v1 - - -schema Project: - r""" - Project defines an Upbound Project, which can be built into a Crossplane - Configuration. - - Attributes - ---------- - apiVersion : str, default is "meta.dev.upbound.io/v2alpha1", required - APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - kind : str, default is "Project", required - Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - metadata : v1.ObjectMeta, default is Undefined, optional - metadata - spec : MetaDevUpboundIoV2alpha1ProjectSpec, default is Undefined, optional - spec - """ - - - apiVersion: "meta.dev.upbound.io/v2alpha1" = "meta.dev.upbound.io/v2alpha1" - - kind: "Project" = "Project" - - metadata?: v1.ObjectMeta - - spec?: MetaDevUpboundIoV2alpha1ProjectSpec - - -schema MetaDevUpboundIoV2alpha1ProjectSpec: - r""" - ProjectSpec is the spec for a Project. Since a Project is not a Kubernetes - resource there is no Status, only Spec. - - Attributes - ---------- - apiDependencies : [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0], default is Undefined, optional - APIDependencies are the API dependencies for this project. - NOTE: This is an experimental feature and is subject to change. - architectures : [str], default is Undefined, optional - architectures - crossplane : MetaDevUpboundIoV2alpha1ProjectSpecCrossplane, default is Undefined, optional - crossplane - dependsOn : [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0], default is Undefined, optional - depends on - description : str, default is Undefined, optional - description - imageConfig : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0], default is Undefined, optional - image config - license : str, default is Undefined, optional - license - maintainer : str, default is Undefined, optional - maintainer - paths : MetaDevUpboundIoV2alpha1ProjectSpecPaths, default is Undefined, optional - paths - readme : str, default is Undefined, optional - readme - repository : str, default is Undefined, required - repository - source : str, default is Undefined, optional - source - """ - - - apiDependencies?: [MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0] - - architectures?: [str] - - crossplane?: MetaDevUpboundIoV2alpha1ProjectSpecCrossplane - - dependsOn?: [MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0] - - description?: str - - imageConfig?: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0] - - license?: str - - maintainer?: str - - paths?: MetaDevUpboundIoV2alpha1ProjectSpecPaths - - readme?: str - - repository: str - - source?: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0: - r""" - APIDependencies defines a reference to an external API dependency. - NOTE: This is an experimental feature and is subject to change. - - Attributes - ---------- - git : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git, default is Undefined, optional - git - http : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP, default is Undefined, optional - http - k8s : MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s, default is Undefined, optional - k8s - $type : str, default is Undefined, required - Type defines the type of API dependency. - """ - - - git?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git - - http?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP - - k8s?: MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s - - $type: "k8s" | "crd" - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0Git: - r""" - Git defines the git repository source for the API dependency. - - Attributes - ---------- - path : str, default is Undefined, optional - Path is the path within the repository to the API definition. - ref : str, default is Undefined, optional - Ref is the git reference (branch, tag, or commit SHA). - repository : str, default is Undefined, required - Repository is the git repository URL. - """ - - - path?: str - - ref?: str - - repository: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0HTTP: - r""" - HTTP defines the HTTP source for the API dependency. - - Attributes - ---------- - url : str, default is Undefined, required - URL is the HTTP/HTTPS URL to fetch the API dependency from. - """ - - - url: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecAPIDependenciesItems0K8s: - r""" - K8s defines the Kubernetes API version for the dependency. - - Attributes - ---------- - version : str, default is Undefined, required - Version is the Kubernetes API version (e.g., "v1.33.0"). - """ - - - version: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecCrossplane: - r""" - CrossplaneConstraints specifies a packages compatibility with Crossplane versions. - - Attributes - ---------- - version : str, default is Undefined, required - Semantic version constraints of Crossplane that package is compatible with. - """ - - - version: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecDependsOnItems0: - r""" - Dependency is a dependency on another package. A dependency can be of an - arbitrary API version and kind, but Crossplane expects package dependencies - to behave like a Crossplane package. Specifically it expects to be able to - create the dependency and set its spec.package field to a package OCI - reference. - - Attributes - ---------- - apiVersion : str, default is Undefined, optional - APIVersion of the dependency. - configuration : str, default is Undefined, optional - Configuration is the name of a Configuration package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - function : str, default is Undefined, optional - Function is the name of a Function package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion, kind, and package instead. - kind : str, default is Undefined, optional - Kind of the dependency. - package : str, default is Undefined, optional - Package OCI reference of the dependency. Only used when apiVersion and - kind are set. - Must be a fully qualified image name, including the registry, - repository, and tag. For example, "registry.example.com/repo/package:tag". - provider : str, default is Undefined, optional - Provider is the name of a Provider package image. - Must be a fully qualified image name, including the registry, - - Deprecated: Specify an apiVersion and kind instead. - version : str, default is Undefined, required - Version is the semantic version constraints of the dependency image. - """ - - - apiVersion?: str - - configuration?: str - - function?: str - - kind?: str - - package?: str - - provider?: str - - version: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0: - r""" - ImageConfig defines a set of rules for matching and rewriting images. - - Attributes - ---------- - matchImages : [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0], default is Undefined, required - MatchImages is a list of image matching rules that should be satisfied. - rewriteImage : MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage, default is Undefined, required - rewrite image - """ - - - matchImages: [MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0] - - rewriteImage: MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage - - -schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0MatchImagesItems0: - r""" - ImageMatch defines a rule for matching image. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix that should be matched. - $type : str, default is "Prefix", optional - Type is the type of match. - """ - - - prefix: str - - $type?: "Prefix" = "Prefix" - - -schema MetaDevUpboundIoV2alpha1ProjectSpecImageConfigItems0RewriteImage: - r""" - RewriteImage defines how a matched image should be rewritten. - - Attributes - ---------- - prefix : str, default is Undefined, required - Prefix is the prefix to use when rewriting the image. - """ - - - prefix: str - - -schema MetaDevUpboundIoV2alpha1ProjectSpecPaths: - r""" - ProjectPaths configures the locations of various parts of the project, for - use at build time. - - Attributes - ---------- - apis : str, default is Undefined, optional - APIs is the directory holding the project's apis. If not - specified, it defaults to `apis/`. - examples : str, default is Undefined, optional - Examples is the directory holding the project's examples. If not - specified, it defaults to `examples/`. - functions : str, default is Undefined, optional - Functions is the directory holding the project's functions. If not - specified, it defaults to `functions/`. - operations : str, default is Undefined, optional - Operations is the directory holding the project's operations. If not - specified, it defaults to `operations/`. - tests : str, default is Undefined, optional - Tests is the directory holding the project's tests. If not - specified, it defaults to `tests/`. - """ - - - apis?: str - - examples?: str - - functions?: str - - operations?: str - - tests?: str - - diff --git a/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k b/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k deleted file mode 100644 index 0705904..0000000 --- a/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/object_meta.k +++ /dev/null @@ -1,97 +0,0 @@ -""" -This is the object_meta module in k8s.apimachinery.pkg.apis.meta.v1 package. -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" - - -schema ObjectMeta: - r""" - ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create. - - Attributes - ---------- - annotations : {str:str}, default is Undefined, optional - Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations - clusterName : str, default is Undefined, optional - The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request. - creationTimestamp : str, default is Undefined, optional - CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. - - Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata - deletionGracePeriodSeconds : int, default is Undefined, optional - Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. - deletionTimestamp : str, default is Undefined, optional - DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested. - - Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata - finalizers : [str], default is Undefined, optional - Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. - generateName : str, default is Undefined, optional - GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server. - - If this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). - - Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency - generation : int, default is Undefined, optional - A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. - labels : {str:str}, default is Undefined, optional - Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels - managedFields : [ManagedFieldsEntry], default is Undefined, optional - ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. - name : str, default is Undefined, optional - Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names - namespace : str, default is Undefined, optional - Namespace defines the space within each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty. - - Must be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces - ownerReferences : [OwnerReference], default is Undefined, optional - List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. - resourceVersion : str, default is Undefined, optional - An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources. - - Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency - selfLink : str, default is Undefined, optional - SelfLink is a URL representing this object. Populated by the system. Read-only. - - DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release. - uid : str, default is Undefined, optional - UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations. - - Populated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids - """ - - - annotations?: {str:str} - - clusterName?: str - - creationTimestamp?: str - - deletionGracePeriodSeconds?: int - - deletionTimestamp?: str - - finalizers?: [str] - - generateName?: str - - generation?: int - - labels?: {str:str} - - managedFields?: any - - name?: str - - namespace?: str - - ownerReferences?: [OwnerReference] - - resourceVersion?: str - - selfLink?: str - - uid?: str - - diff --git a/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k b/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k deleted file mode 100644 index 3854a5c..0000000 --- a/tests/test-cluster/model/k8s/apimachinery/pkg/apis/meta/v1/owner_reference.k +++ /dev/null @@ -1,41 +0,0 @@ -""" -This is the owner_reference module in k8s.apimachinery.pkg.apis.meta.v1 package. -This file was generated by the KCL auto-gen tool. DO NOT EDIT. -Editing this file might prove futile when you re-run the KCL auto-gen generate command. -""" - - -schema OwnerReference: - r""" - OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field. - - Attributes - ---------- - apiVersion : str, default is Undefined, required - API version of the referent. - blockOwnerDeletion : bool, default is Undefined, optional - If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. - controller : bool, default is Undefined, optional - If true, this reference points to the managing controller. - kind : str, default is Undefined, required - Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - name : str, default is Undefined, required - Name of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#names - uid : str, default is Undefined, required - UID of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#uids - """ - - - apiVersion: str - - blockOwnerDeletion?: bool - - controller?: bool - - kind: str - - name: str - - uid: str - - diff --git a/tests/test-cluster/model/kcl.mod b/tests/test-cluster/model/kcl.mod deleted file mode 100644 index 9df41d7..0000000 --- a/tests/test-cluster/model/kcl.mod +++ /dev/null @@ -1,4 +0,0 @@ -[package] -name = "models" -edition = "v0.11.2" -version = "0.0.1" From f3d3034b51704c924123e3ee99f85ccbb3451883 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 15:03:52 -0500 Subject: [PATCH 40/44] fix(psqlcluster): match ExternalSecret status lookup to actual resource names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The state-status template looked up `external-secret` in the observed map, but 100-external-secret renders two distinct resources with suffixed names (`external-secret-app`, `external-secret-superuser`) when their respective config blocks are set. The lookup never matched, so $state.observed.externalSecret.ready was always false. Now: aggregate over both names. Ready=true when every present ES Object reports Ready=true, and Ready=true when neither is present (BYO / CNPG-managed-secret paths — nothing to wait on). Note: the field is currently unused (999-status doesn't surface it, no composition gates on it). Fixing now to match the documented intent so future consumers don't inherit the bug. --- .../cluster/010-state-status.yaml.gotmpl | 31 +++++++++++++------ 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/functions/cluster/010-state-status.yaml.gotmpl b/functions/cluster/010-state-status.yaml.gotmpl index d66a668..bc1b38a 100644 --- a/functions/cluster/010-state-status.yaml.gotmpl +++ b/functions/cluster/010-state-status.yaml.gotmpl @@ -22,18 +22,31 @@ {{- end }} # ============================================================================== -# ExternalSecret status — Object Ready=true means the ExternalSecret CR exists. -# We don't inspect the wrapped status further; SecretSynced is fast enough. +# ExternalSecret status — the composition emits two distinct Object resources +# (`external-secret-app` and `external-secret-superuser`) whenever the +# corresponding `*.externalSecret` block is set. Aggregate readiness across +# both: ready=true when every present ES Object reports Ready=true, and +# ready=true when neither exists (nothing to wait on, BYO/auto-gen path). +# Object Ready=true means the ExternalSecret CR was applied; SecretSynced +# follows fast enough that we don't inspect the wrapped ES status further. # ============================================================================== -{{- $esEntry := get $observed "external-secret" | default dict }} -{{- $esResource := $esEntry.resource | default dict }} -{{- $esConditions := (($esResource.status | default dict).conditions | default list) }} -{{- $esReady := false }} -{{- range $esConditions }} - {{- if and (eq .type "Ready") (eq .status "True") }} - {{- $esReady = true }} +{{- $esResourceNames := list "external-secret-app" "external-secret-superuser" }} +{{- $esPresent := 0 }} +{{- $esReadyCount := 0 }} +{{- range $name := $esResourceNames }} + {{- $entry := get $observed $name | default dict }} + {{- if $entry }} + {{- $esPresent = add $esPresent 1 }} + {{- $resource := $entry.resource | default dict }} + {{- $conditions := (($resource.status | default dict).conditions | default list) }} + {{- range $conditions }} + {{- if and (eq .type "Ready") (eq .status "True") }} + {{- $esReadyCount = add $esReadyCount 1 }} + {{- end }} + {{- end }} {{- end }} {{- end }} +{{- $esReady := eq $esPresent $esReadyCount }} {{- $state = set $state "observed" (dict "cluster" (dict "ready" $clusterReady "phase" $clusterPhase) From 7f79901a85588a4244592fa10ba5a605732f8362 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 15:08:03 -0500 Subject: [PATCH 41/44] fix(make): use defined Makefile vars in validate:% recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The validate:% recipe was using $$definition, $$composition, $$api_dir shell variables that were never initialized — `make validate:minimal` silently fed empty strings to `up composition render` and `crossplane beta validate`, so the recipe never produced a real validation result. Aligns validate:% with render:%: both now use the top-level Makefile variables ($(DEFINITION), $(COMPOSITION), $(XRD_DIR)) which point at apis/psqlstacks. Single-target shorthand stays psqlstacks-only as documented in README. The :all targets keep their derive-per-example shell logic for the multi-API case. --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 62fc7a9..2e84af2 100644 --- a/Makefile +++ b/Makefile @@ -112,9 +112,9 @@ render\:%: validate\:%: @example="examples/psqlstacks/$*.yaml"; \ - up composition render --xrd=$$definition $$composition $$example \ + up composition render --xrd=$(DEFINITION) $(COMPOSITION) $$example \ --include-full-xr --quiet | \ - crossplane beta validate $$api_dir --error-on-missing-schemas - + crossplane beta validate $(XRD_DIR) --error-on-missing-schemas - test: up test run $(RENDER_TESTS) From 45619d51b2c8a44b2f051fedb4941a3e55b5b64b Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 15:17:14 -0500 Subject: [PATCH 42/44] test(psqlcluster): assert full ExternalSecret data mapping MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Existing test 7 only asserted the ExternalSecret existed with the right secretStoreRef and target.name — it didn't lock in the data[] mappings or the basic-auth target template. The current gotmpl correctly maps both username and password as separate remoteRef entries (with `property: username` / `property: password`) extracted from the same JSON blob at secretRef.path, then synthesizes a kubernetes.io/basic-auth Secret. Asserting the full shape so a regression that drops one key fails the test instead of silently shipping a half-populated Secret. --- tests/test-cluster/main.k | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/tests/test-cluster/main.k b/tests/test-cluster/main.k index dbb1905..934d4c8 100644 --- a/tests/test-cluster/main.k +++ b/tests/test-cluster/main.k @@ -294,7 +294,40 @@ _items = [ name = "hops-aws-secrets-manager" kind = "ClusterSecretStore" } - target.name = "es-app-app" + # Both username + password must be mapped from the + # JSON blob at secretRef.path. CNPG reads this + # Secret as kubernetes.io/basic-auth — a single + # remoteRef without `property` would leave one of + # the two keys empty and CNPG would fail to + # bootstrap the role. Lock in the full shape so + # future drift is caught. + data = [ + { + secretKey = "username" + remoteRef = { + key = "my-cluster/es-app" + property = "username" + } + } + { + secretKey = "password" + remoteRef = { + key = "my-cluster/es-app" + property = "password" + } + } + ] + target = { + name = "es-app-app" + creationPolicy = "Owner" + template = { + type = "kubernetes.io/basic-auth" + data = { + username = "{{ .username }}" + password = "{{ .password }}" + } + } + } } } } From e0399959023ed7737efec3bc9a9aaadbe7f42428 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 15:22:48 -0500 Subject: [PATCH 43/44] fix(psqlcluster): bump ExternalSecret to external-secrets.io/v1 ESO 0.10+ moved to v1 GA, and recent ESO charts (the version cert-stack/aws-secret-stack already use) drop the v1beta1 served-version: applies fail with `no matches for kind "ExternalSecret" in version "external-secrets.io/v1beta1"`. Schema shape (data[].remoteRef.{key,property} + target.template) is identical between v1beta1 and v1, so this is a pure apiVersion bump. Aligns with the other stacks that already use v1 (auth, gitops, cloudflare/dns, aws/secret). --- functions/cluster/100-external-secret.yaml.gotmpl | 2 +- tests/test-cluster/main.k | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/functions/cluster/100-external-secret.yaml.gotmpl b/functions/cluster/100-external-secret.yaml.gotmpl index d65dc01..b3b2bf3 100644 --- a/functions/cluster/100-external-secret.yaml.gotmpl +++ b/functions/cluster/100-external-secret.yaml.gotmpl @@ -26,7 +26,7 @@ spec: managementPolicies: {{ .ManagementPolicies | toJson }} forProvider: manifest: - apiVersion: external-secrets.io/v1beta1 + apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: {{ .SecretName }} diff --git a/tests/test-cluster/main.k b/tests/test-cluster/main.k index 934d4c8..c0dcd43 100644 --- a/tests/test-cluster/main.k +++ b/tests/test-cluster/main.k @@ -285,7 +285,7 @@ _items = [ kind = "Object" metadata.name = "es-app-external-secret-app" spec.forProvider.manifest = { - apiVersion = "external-secrets.io/v1beta1" + apiVersion = "external-secrets.io/v1" kind = "ExternalSecret" metadata.name = "es-app-app" spec = { From 6b8e83c68e4cc203bf6870d49abc1152e421ec56 Mon Sep 17 00:00:00 2001 From: Patrick Lee Scott Date: Fri, 8 May 2026 15:26:20 -0500 Subject: [PATCH 44/44] docs(psqlcluster): mirror externalSecret descriptions on superuser block MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The superuser.externalSecret schema had the same shape as app.externalSecret but no field descriptions, so kubectl explain and generated docs gave inconsistent guidance for what's the same contract. Adds the missing descriptions (secretStore.{kind,name,namespace} and secretRef.{path}) verbatim from app.externalSecret. Pure documentation change — no behavior, default, or required-fields shift. --- apis/psqlclusters/definition.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/apis/psqlclusters/definition.yaml b/apis/psqlclusters/definition.yaml index a82cc85..8dbdcc0 100644 --- a/apis/psqlclusters/definition.yaml +++ b/apis/psqlclusters/definition.yaml @@ -188,24 +188,30 @@ spec: type: object properties: secretStore: + description: Reference to the SecretStore or ClusterSecretStore providing the value. type: object properties: kind: + description: Store kind. Use ClusterSecretStore for cluster-scoped stores (the platform default), SecretStore for namespaced stores in this XR's namespace. type: string enum: - ClusterSecretStore - SecretStore default: ClusterSecretStore name: + description: Name of the SecretStore or ClusterSecretStore. type: string namespace: + description: Namespace of the SecretStore (ignored for ClusterSecretStore). Defaults to the PSQLCluster's namespace. type: string required: - name secretRef: + description: Where in the secrets backend the credentials live. The remote value must be a JSON blob with `username` and `password` properties — both are extracted into the resulting K8s Secret. type: object properties: path: + description: Path/key in the secrets backend (e.g. AWS Secrets Manager `my-cluster/superuser`). type: string required: - path