Handle multiple backends for the TypeInstance by mkuziemko · Pull Request #657 · capactio/capact

mkuziemko · 2022-03-10T10:54:41Z

Description

Changes proposed in this pull request:

Argo action in case of the artifact:

value: 
    key: test
backend:
    context:
        (...)

will create an appropriate TypeInstance object and create a mutation.

modify Helm and Terraform runner to add value to output,
modify Jinja with the new configuration field: strip-value: true,
modify merger to remove value if exists in the artifacts.

Testing

Mattermost installation

Check out this branch.
Create a cluster with make dev-cluster
Set manifest location to my fork

kubectl set env deploy/capact-hub-public -n capact-system -c hub-public-populator MANIFESTS_SOURCES="github.com/mkuziemko/hub-manifests?ref=context_handling"

Wait until the manifests are populated
Use port-foward for capact-gateway :

kubectl port-forward svc/capact-gateway 8080:80 -n capt-system

Create parameters for Mattermost installation:

cat <<EOF > /tmp/mattermost-install.yaml
input-parameters:
  host: mattermost.capact.local
EOF

Create action

capact action create --name mattermost-install cap.interface.productivity.mattermost.install --parameters-from-file /tmp/mattermost-install.yaml

Use capact act run mattermost-install and capact act watch mattermost-install to observe the status of the running action.
When it is finished, you can delete the action using capact act delete mattermost-install and helm charts.

PostgreSQL installation

Prepare inputs for PostgreSQL:

cat <<EOF > /tmp/postgresql-install.yaml
input-parameters:
   superuser:
       username: "username"
       password: "password"
   defaultDBName: "default_db"
EOF

Create the action:

capact action create --name test-1 cap.interface.database.postgresql.install --parameters-from-file /tmp/postgresql-install.yaml

Run and watch the action:

capact act run test-1
capact act watch test-1

Run integration tests

Currently, the tests are failing due to a lack of storage backend integration. It will be fixed by: #655

Note: No longer valid. The PR was merged and this branch was updated. We can run tests by simple: make test-integration

Install secret storage backend:

helm install dotenv -n capact-system ./deploy/kubernetes/charts/secret-storage-backend --set=global.containerRegistry.path="ghcr.io/capactio/pr" --set=global.containerRegistry.overrideTag="PR-653" --set=supportedProviders={aws_secretsmanager,dotenv} --wait

Comment out line from e2e_suite_test.go:

waitTillServiceEndpointsAreReady()

Create envs for test:

export STATUS_ENDPOINTS="not-needed"
export GATEWAY_USERNAME="graphql"
export GATEWAY_PASSWORD="t0p_s3cr3t"
export GATEWAY_ENDPOINT="https://gateway.capact.local/graphql"

Run new scenario:

ginkgo -tags="integration" -v --focus="should propagate context provider to storage backend" ./test/e2e/...
ginkgo -tags="integration" -v --focus="should fail due to incorrect storage provider" ./test/e2e/...

Related issue(s)

Handle multiple backends for the TypeInstance upload from workflow #634

mkuziemko · 2022-03-10T11:58:12Z

+			typeInstance.Value = typeInstanceValue
+		} else {
+			// for backward compatibility, if there is an artifact without value/backend syntax,
+			// treat it as a value for TypeInstance


it is mainly for the case like here

mkuziemko · 2022-03-10T22:28:29Z

@@ -50,7 +50,7 @@ readonly CAPACT_USE_TEST_SETUP="false"
 #


will be removed before merge

mszostok

In general, our contract is not implemented. So I didn't run the test as it shouldn't work that way.

mszostok

Overall it LGTM 👍 But please address my comments and additionally find in our code base all TODO(https://github.com/capactio/capact/issues/634) and fix them. Thanks!

mszostok

I saw the WIP label, so it's probably not for review, but I already had some comments so I posted them but I understand that you may already addressed them locally :)

Please also fix e2e tests (dotenv storage backend definition). Thanks!

mszostok · 2022-03-23T12:58:45Z

+
+		typeInstance.Value = unmarshalledTI.Value
+		if unmarshalledTI.Backend != nil {
+			if typeInstance.Backend != nil {


nit: IMO we should return an error when typeInstance.Backend.ID is not set. As setting context without backend ID means that it will be stored in built-in storage and this storage doesn't allow context by definition.

mszostok · 2022-03-23T14:53:28Z

  filename=$(basename -- "$filename")
  prefix="${filename%.*}"
+
+  # remove value key if exists


For consistency reason, it would be good to also have unpackValue conf option here. When I went through our manifests, it was hard to understand why it works in this way, as this is a "hidden" functionality.

But let's do that on a new follow-up PR. It's not a blocker for merging this one 👍

mkuziemko added area/engine Relates to Engine area/hub Relates to Hub labels Mar 10, 2022

mkuziemko requested review from mszostok, pkosiec and tmitchell as code owners March 10, 2022 10:54

github-actions Bot reviewed Mar 10, 2022

View reviewed changes

Comment thread pkg/runner/common.go Outdated

mkuziemko mentioned this pull request Mar 10, 2022

Support for handle multiple backends capactio/hub-manifests#60

Merged

mkuziemko force-pushed the add_context_to_backend branch from 9a22c5d to e134b65 Compare March 10, 2022 11:08

Handle multiple backends for the TypeInstance

15698eb

mkuziemko force-pushed the add_context_to_backend branch from e134b65 to 15698eb Compare March 10, 2022 11:17

mkuziemko commented Mar 10, 2022

View reviewed changes

mkuziemko mentioned this pull request Mar 10, 2022

Handle multiple backends for the TypeInstance upload from workflow #634

Closed

Mateusz Kuziemko added 2 commits March 10, 2022 17:29

Merge branch 'main' into add_context_to_backend

49634a3

Fix tests after merge

d87db66

mkuziemko commented Mar 10, 2022

View reviewed changes

mszostok self-assigned this Mar 11, 2022

mszostok added a commit to mszostok/capact that referenced this pull request Mar 11, 2022

Add fake value expectation in e2e test till capactio#657 is not resolved

e2735a7

mszostok mentioned this pull request Mar 11, 2022

Add fake value expectation in e2e test till #657 is not resolved #660

Merged

mszostok added a commit that referenced this pull request Mar 11, 2022

Add fake value expectation in e2e test till #657 is not resolved (#660)

834cd44

mszostok reviewed Mar 13, 2022

View reviewed changes

Merge branch 'main' into add_context_to_backend

bbfc2b8

mkuziemko added the WIP Work in progress label Mar 15, 2022

mkuziemko force-pushed the add_context_to_backend branch 3 times, most recently from efe5dbb to d00eed2 Compare March 15, 2022 07:52

Fixes after review

b5eb0f2

mkuziemko force-pushed the add_context_to_backend branch from d00eed2 to b5eb0f2 Compare March 15, 2022 18:04

mkuziemko removed the WIP Work in progress label Mar 15, 2022

mszostok reviewed Mar 16, 2022

View reviewed changes

Comment thread pkg/argo-actions/update_type_instances.go Outdated

Comment thread pkg/argo-actions/update_type_instances.go Outdated

Comment thread pkg/argo-actions/upload_type_instances.go Outdated

Comment thread pkg/runner/common.go Outdated

Comment thread test/e2e/action_test.go Outdated

Refactor integration tests

abe8522

Merge branch 'main' into add_context_to_backend

5cd86f4

pkosiec mentioned this pull request Mar 17, 2022

Update Content development guide capactio/website#114

Merged

Mateusz Kuziemko added 2 commits March 18, 2022 08:49

fix typo

6d0e3f6

fix conflicts after merge

37c4a50

mkuziemko added the WIP Work in progress label Mar 18, 2022

mszostok reviewed Mar 20, 2022

View reviewed changes

Comment thread test/e2e/action_test.go Outdated

Comment thread test/e2e/action_test.go Outdated

Comment thread test/e2e/action_test.go Outdated

Comment thread test/e2e/action_test.go

mszostok mentioned this pull request Mar 21, 2022

Fix fetching context from neo4j db #681

Closed

mkuziemko force-pushed the add_context_to_backend branch from 946969c to acc2b19 Compare March 21, 2022 13:07

Fix internal storage TI

ac92303

mkuziemko force-pushed the add_context_to_backend branch from acc2b19 to ac92303 Compare March 21, 2022 13:13

Add pkg/hub/api/graphql/local/schema_gen.go

7f663c7

mkuziemko removed the WIP Work in progress label Mar 21, 2022

pkosiec added the enhancement New feature or request label Mar 23, 2022

mszostok approved these changes Mar 23, 2022

View reviewed changes

mszostok suggested changes Mar 23, 2022

View reviewed changes

Comment thread pkg/argo-actions/download_type_instances.go

Fix helm upgrade command

e902da7

mszostok mentioned this pull request Mar 23, 2022

Add Helm template storage backend logic #685

Merged

mszostok approved these changes Mar 24, 2022

View reviewed changes

Run integration tests in main

dca5f7c

mkuziemko merged commit 6b60379 into capactio:main Mar 24, 2022

mkuziemko deleted the add_context_to_backend branch March 24, 2022 11:09

Conversation

mkuziemko commented Mar 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Related issue(s)

Uh oh!

Uh oh!

mkuziemko Mar 10, 2022

Choose a reason for hiding this comment

Uh oh!

mkuziemko Mar 10, 2022

Choose a reason for hiding this comment

Uh oh!

mszostok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mszostok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mszostok left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mszostok Mar 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mszostok Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mkuziemko commented Mar 10, 2022 •

edited

Loading

mszostok left a comment •

edited

Loading

mszostok Mar 23, 2022 •

edited

Loading