Skip to content

Add Helm template storage backend logic#685

Merged
mszostok merged 1 commit intocapactio:mainfrom
mszostok:helm-storage/templates
Mar 24, 2022
Merged

Add Helm template storage backend logic#685
mszostok merged 1 commit intocapactio:mainfrom
mszostok:helm-storage/templates

Conversation

@mszostok
Copy link
Copy Markdown
Collaborator

@mszostok mszostok commented Mar 22, 2022

Description

Changes proposed in this pull request:

  • Add Helm template storage backend logic

Testing

  1. Create cluster:
    export DISABLE_MONITORING_INSTALLATION=true
    export ENABLE_POPULATOR=true
    export CLUSTER_TYPE=k3d
    export DISABLE_K3D_REGISTRY=false
    make dev-cluster
    
  2. Install Helm backend via UI
  3. Upgrade Helm backend release (new chart is not yet released):
    helm upgrade helm-storage-backend-1648070484 -n default ./deploy/kubernetes/charts/helm-storage-backend --set=global.containerRegistry.path="ghcr.io/capactio/pr" --set=global.containerRegistry.overrideTag="PR-685" --wait --reuse-values
    Update helm release name.
  4. Until Handle multiple backends for the TypeInstance #657 is not merge, run:
    kubectl set image deploy/capact-hub-local hub-local=ghcr.io/capactio/pr/hub-js:PR-657
    
  5. Install sample Helm release via Runner:
    RUNNER_CONTEXT_PATH=cmd/helm-runner/example-input/context.yaml \
     RUNNER_ARGS_PATH=cmd/helm-runner/example-input/install-args.yaml \
     RUNNER_LOGGER_DEV_MODE=true \
     RUNNER_COMMAND="install" \
     go run cmd/helm-runner/main.go
  6. Get ID of cap.type.helm.template.storage:0.1.0
  7. Run test:
    BACKEND_ID="ecdfece6-da75-483f-a3ca-24246ed3308d" go test ./pkg/hub/client/ -run TestThatShowcaseExternalStorage -v -count 1

Unfortunately, in my case, it's quite slow. Probably we will need to debug it further. Currently, for my setup to work properly, it requires 1CPU.

The general problem is that the PostgreSQL has as a dependency “common” chart (small size). We need to download it (in general deps charts) as it won't be available by default. The LocateChart method from Helm project, that is able to download this chart, starts with downloading an index file which in case of Bitnami has >3MB. This needs to be decoded and as a result it requires a much higher CPU limit.

You could say, that this is a special case, but unfortunately the PostgreSQL is quite often used in our examples. It needs to work properly.

Additionally, Helm methods don't support context, so we are not able to cancel any ongoing operation even if user cancelled the request and no one will wait for the response.

Related issue(s)

Fix #672

@mszostok mszostok added enhancement New feature or request area/engine Relates to Engine area/hub Relates to Hub labels Mar 22, 2022
@mszostok mszostok force-pushed the helm-storage/templates branch 6 times, most recently from acb8f0f to 4ab4266 Compare March 23, 2022 22:55
@pkosiec pkosiec self-assigned this Mar 24, 2022
Copy link
Copy Markdown
Collaborator

@pkosiec pkosiec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well, I've just tested that E2E on my PR (capactio/hub-manifests#64). Just a few minor comments 🙂

Comment thread deploy/kubernetes/charts/helm-storage-backend/values.yaml
Comment thread internal/helm-storage-backend/template.go Outdated
Comment thread internal/helm-storage-backend/template_test.go
@mszostok mszostok force-pushed the helm-storage/templates branch 2 times, most recently from d789d38 to bd473a2 Compare March 24, 2022 22:24
@mszostok mszostok force-pushed the helm-storage/templates branch from bd473a2 to b409504 Compare March 24, 2022 22:27
@mszostok mszostok enabled auto-merge (squash) March 24, 2022 22:28
@mszostok mszostok merged commit bf8d006 into capactio:main Mar 24, 2022
@mszostok mszostok deleted the helm-storage/templates branch March 28, 2022 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/engine Relates to Engine area/hub Relates to Hub enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Helm template handler for Helm storage backend

2 participants