Can't deploy latest release on GKE

[jairojunior]

While generating the manifests to deploy ww-gitops using gitops CLI[1], can't install HelmRelease as the Pod enters a CrashLoopBackOff.

[1]

gitops create dashboard ww-gitops \
  --password=$PASSWORD \
  --export > ./clusters/development/weave-gitops-dashboard.yaml

Environment

• Weave-Gitops Version v0.30.0
• Flux Version 2.0.1
• Kubernetes version v1.27.3-gke.100

Additional Context (screenshots, logs, etc)

2023-08-22T17:13:16.761Z        INFO    gitops  cmd/cmd.go:135  Version {"version": "v0.30.0", "git-commit": "8d13b121", "branch": "HEAD", "buildtime": "2023-08-16_15:53:45"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d2bec6]
                                                         
goroutine 1 [running]:              
k8s.io/client-go/discovery.convertAPIResource({{0xc000118850, 0xa}, 0x0, {0xc000118860, 0x7}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:114 +0x1e6
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000548b28, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:95 +0x845
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0000b64f8, 0x15}, {0xc000359440, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:49 +0x17a
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xc0000a4d50)
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:328 +0x838
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc0000a4d50)
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:203 +0x1fc
k8s.io/client-go/discovery.ServerGroupsAndResources({0x34f5278, 0xc0000a4d50})
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:413 +0x18b
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:376 +0x91
k8s.io/client-go/discovery.withRetries(0x2, 0xc00071c458) 
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:651 +0xec
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0xc0000a4d50)
        /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:375 +0xb9
k8s.io/client-go/restmapper.GetAPIGroupResources({0x34f5278, 0xc0000a4d50})
        /go/pkg/mod/k8s.io/client-go@v0.26.3/restmapper/discovery.go:148 +0xa2
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:94 +0x67
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(0xc000469400)
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:130 +0x44
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000400fc0, {0x0, 0x0, 0x0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:110 +0x405
sigs.k8s.io/controller-runtime/pkg/client.newClient(0xc000400fc0, {0xc0000df880, {0x0, 0x0}, {0x0, 0x0}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/client.go:109 +0x1b8
sigs.k8s.io/controller-runtime/pkg/client.New(0xc000526b40, {0xc0000df880, {0x0, 0x0}, {0x0, 0x0}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/client.go:77 +0x99
github.com/weaveworks/weave-gitops/cmd/gitops-server/cmd.runCmd(0xc000004900, {0xc000477020, 0x0, 0x3})
        /app/cmd/gitops-server/cmd/cmd.go:164 +0x97a
github.com/spf13/cobra.(*Command).execute(0xc000004900, {0xc000050090, 0x3, 0x3})
        /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0xd02
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004900)
        /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x991
github.com/spf13/cobra.(*Command).Execute(0xc000004900)
        /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 +0x38
main.main()
        /app/cmd/gitops-server/main.go:11 +0x35```

[yiannistri]

@jairojunior we have fixed a related bug today in #3959 Can you confirm whether the apiVersion shown for the HelmRepository in the generated file is source.toolkit.fluxcd.io/v1 ?

[jairojunior]

@yiannistri I forgot to mention that I fixed that before applying the manifests, hence the CrashLoopBackOff after installing the release.

[rileyai-dev]

@yiannistri I have the same issue (and same versions of everything) but the HelmRepository is set to version: v1beta2 (but its age is 200 days so might be from an older release...)

[yiannistri]

I will attempt to reproduce this one today and report back.

[rileyai-dev]

@yiannistri Were you able to reproduce the issue?

[yiannistri]

I was not able to reproduce this issue. I've tried the following steps:

> gcloud container clusters create issue-3960 --num-nodes 2 --machine-type e2-medium
NAME        LOCATION        MASTER_VERSION  MASTER_IP     MACHINE_TYPE  NODE_VERSION    NUM_NODES  STATUS
issue-3960  europe-west1-b  1.27.3-gke.100  34.77.78.120  e2-medium     1.27.3-gke.100  2          RUNNING
> flux bootstrap github --owner yiannistri --repository issue-3960 --personal
► connecting to github.com
✔ repository "https://github.com/yiannistri/issue-3960" created
► cloning branch "main" from Git repository "https://github.com/yiannistri/issue-3960.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed sync manifests to "main" ("cf0ac735c16b0a99168a9133b56909f3ca8273df")
► pushing component manifests to "https://github.com/yiannistri/issue-3960.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBA+P6TTYOjOMdB2EMwK74qpWqk5LEPmgw2u2sE/NnJwLHQWfq9VfAQ9UeqnOupQE0iq/etJZZI1KOmC8dYsJ5NpG84oHIkzL44xtFti7lJethoqjnHjRpigVvKMczGu/Gg==
✔ configured deploy key "flux-system-main-flux-system" for "https://github.com/yiannistri/issue-3960"
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("6e2187eb722a4785b8110f0062168476a12e69b7")
► pushing sync manifests to "https://github.com/yiannistri/issue-3960.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
> flux version
flux: v2.0.1
helm-controller: v0.35.0
kustomize-controller: v1.0.1
notification-controller: v1.0.0
source-controller: v1.0.1
> gitops create dashboard ww-gitops --password=$PASSWORD --export  > ww-gitops.yaml
> gitops version
Current Version: 0.30.0
GitCommit: 8d13b121c8fdce8904093c2d74c0a9b440a275d2
BuildTime: 2023-08-16T15:40:19Z
Branch: releases/v0.30.0

Then I changed the HelmRepository apiVersion and pushed the following YAML to my GitOps repo:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  annotations:
    metadata.weave.works/description: This is the source location for the Weave GitOps
      Dashboard's helm chart.
  labels:
    app.kubernetes.io/component: ui
    app.kubernetes.io/created-by: weave-gitops-cli
    app.kubernetes.io/name: weave-gitops-dashboard
    app.kubernetes.io/part-of: weave-gitops
  name: ww-gitops
  namespace: flux-system
spec:
  interval: 1h0m0s
  type: oci
  url: oci://ghcr.io/weaveworks/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  annotations:
    metadata.weave.works/description: This is the Weave GitOps Dashboard.  It provides
      a simple way to get insights into your GitOps workloads.
  name: ww-gitops
  namespace: flux-system
spec:
  chart:
    spec:
      chart: weave-gitops
      sourceRef:
        kind: HelmRepository
        name: ww-gitops
  interval: 1h0m0s
  values:
    adminUser:
      create: true
      passwordHash: $2a$10$h791wKMCvBbDC.lDh3n65OL6wLdQRiWqH2f1LVC7qO6zS3ovX4Xki
      username: admin

Once Flux reconciled the HelmRepository/HelmRelease I could see the pods started:

kubectl get pods -n flux-system
NAME                                      READY   STATUS    RESTARTS   AGE
helm-controller-74b5f87d94-trgkn          1/1     Running   0          11m
kustomize-controller-59d4cb8bc6-mjcx5     1/1     Running   0          11m
notification-controller-b7d8566b7-jp754   1/1     Running   0          11m
source-controller-645656595b-d56m5        1/1     Running   0          11m
ww-gitops-weave-gitops-bd77c7d99-8gbrx    1/1     Running   0          2m10s

[yiannistri]

The original error reported happens when calling config.GetConfig() which should create a Kubernetes client and pass it on to the auth server which uses the client to lookup the OIDC secret. But if something fails it should return an error and exit rather than panic.

[rileyai-dev]

We got the same panic error and the only difference we have from @jairojunior is that we are on v1.27.4-eks-2d98532

Everything happened when we upgraded kubernetes from 1.26 to 1.27. We still have our prod environment on the 1.26 version, we are waiting this to be fixed before we upgrade it.

[yiannistri]

Ok digging a bit more, I suspect this panic is related to kubernetes/kubernetes#119840. Let me check what libs we can update to resolve this.