Adjust all Implementation manifests to use latest workflow syntax and storage backends

[pkosiec]

Description

Blocked by #634, #650, #665

• Adjust all Implementation manifests inside capactio/hub-manifests to use the latest workflow syntax (Argo workflows artifact content) - breaking change introduced in Handle multiple backends for the TypeInstance upload from workflow #634
• Use Helm storage for all Helm installation (introduced in Implement Helm storage backend logic #650)
• Use dynamic TypeInstance projection for output TypeInstances (Implement Go Template storage backend service #665 and Implement Helm storage backend logic #650)
• Update RocketChat and Mattermost installation tutorial (add prerequisite for Helm storage)

Additional scope:

• Move outputting static Helm Release logic from Helm Runner inputs to env variables
• Remove output.useHelmReleaseStorage from helm.install and add env variable to Helm Runner container. We have now two Implementations (install and install-static, so we can distinguish them). However, all manifests that use Helm runner should be updated to always assume they use the new helm.install which outputs dynamic Helm Release TI. That means:

capact-outputTypeInstances:
  - name: helm-release
     from: helm-release

Needs to be changed to:

capact-outputTypeInstances:
  - name: helm-release
     from: helm-release
     backend: helm-release-storage # from requires section

• Remove output.useHelmReleaseStorage from helm.upgrade input Type - detect input TypeInstance in Helm Runner and output the same shape
• Consider making a dedicated Interface and Implementation for the ti-value-fetcher.

Timebox for 3SP.

Related issues

See epic #604 for reason and use cases.

[mszostok]

Testing scenarios

AWS:

• Elasticsearch - tested separately ✅
• Redis - tested separately ✅
• RDS - tested as a part of Mattermost install ✅
• Secrets Manager - tested manually ✅

GCP:

• CloudSQL - tested as a part of Concourse install ✅

Bitnami:

• MinIO - tested separately ✅
• PostgreSQL - tested as a part of Mattermost install ✅
• Redis - tested separately ✅
• MongoDB - tested as a part of RocketChat install ✅

Elastic:

• Elasticsearch - tested separately ✅ - as the only one I left it with the static Helm storages

Capact:

• Upgrade - the Helm storage is not available when the Capact is installed, as a result, we cannot change Upgrade action to dynamic Helm storage.

Applications:

1. Install Mattermost
   1. RDS - found issue that will be solved by Storage backend ID for TypeInstance from nested workflow is lost during TypeInstance upload #705
   2. on-prem PostgreSQL ✅
2. Install RocketChat
   1. on-prem MongoDB ✅
3. Install Concourse
   1. CloudSQL - in progress ✅

Helm Storage

Found Issues:

• We have memory issue, if we have multiple TypeInstances with Template backend that refers to the bitnami Helm chart. Additionally, executing capact ti get -oyaml causes that the Helm Template backend pod is OOMKilled because in a single request we fetch all TypeInstance, but the same instance is “fetched” multiple times because of uses relation and latest previous revisions revision field. The Helm Template backend probably doesn't have memory leak as the resources are released. Later it has higher memory usage, but this could be due to memory management (because of performance reason sometimes app doesn't release all memory until it is asked explicitly by the system)

  

I installed the Helm storage via Dashboard and applied such Global Policy:

cat > /tmp/helm-storage-policy.yaml << ENDOFFILE
interface:
  default: # properties applied to all rules above
     inject:
       requiredTypeInstances:
          - id: "3e4af86a-ab76-48c5-8799-1bcb0d994018"
            description: "Helm template"
          - id: "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
            description: "Helm release"
  rules:
      - interface:
          path: cap.*
        oneOf:
        - implementationConstraints:
            requires:
            - path: cap.core.type.platform.kubernetes
        - implementationConstraints: {}
ENDOFFILE
capact policy apply -f /tmp/helm-storage-policy.yaml

Elasticsearch

Found issues:

• The host parameters is ignored in AWS Implementation. However, the output TypeInstance contains the Elastic URL from AWS ELB which is the valid one.
• The version is different. In AWS Implementation I changed it from number to string, however we are not able to enforce the SemVer format, we got 7.10 but we want 7.10.0.
  I decided to remove the SemVer regex. We should not force it on AWS Implementation.

AWS

1. Create TypeInstance with AWS Credentials.
2. Create Action Policy to prefer AWS Implementation:

   export AWS_TI_ID="76db17cc-6a72-4d3a-99d7-4c6cbf887446"
   cat > /tmp/action-policy.yaml << ENDOFFILE
   interface:
     rules:
       - interface:
           path: "cap.interface.analytics.elasticsearch.install"
         oneOf:
           - implementationConstraints:
               attributes:
                 - path: "cap.attribute.cloud.provider.aws"
             inject:
               requiredTypeInstances:
               - id: ${AWS_TI_ID}
                 description: "AWS credentials"
       - interface:
           path: "cap.*"
         oneOf:
           - implementationConstraints:
               requires:
                 - path: "cap.core.type.platform.kubernetes"
           - implementationConstraints: {}
   ENDOFFILE

3. Create input parameters:

   cat > /tmp/aws-elastic.yaml << ENDOFFILE
   input-parameters:
     host: "aws-elastic.capact.local" # this is ignored by AWS implementation
     replicas: "1"
   ENDOFFILE

4. Create an Action:

   capact act create cap.interface.analytics.elasticsearch.install --name "aws-elastic" --parameters-from-file /tmp/aws-elastic.yaml --action-policy-from-file /tmp/action-policy.yaml

5. Run and watch Action:

   capact act run aws-elastic
   capact act watch aws-elastic

Test

1. Get TypeInstance details:

   capact ti get 0fbb0212-bc2e-47de-a971-dbcbf46b39af -ojson | jq -r '.[0].latestResourceVersion.spec.value'
   

   Expected output:

   {
     "basicAuthSecretName": "elastic-basic-authr8gx5",
     "host": "search-elastic-rntjuhjurdwz-wubimtrl2egzfwzrufcxcqc3bu.eu-west-1.es.amazonaws.com",
     "port": 443,
     "protocol": "https",
     "version": "7.10"
   }

2. Check health:

   export USER=$(kubectl -n default get secret elastic-basic-authr8gx5  -ogo-template='{{.data.username | base64decode }}')
   export PASS=$(kubectl -n default get secret elastic-basic-authr8gx5  -ogo-template='{{.data.password | base64decode }}')
   
   curl -u $USER:$PASS https://search-elastic-rntjuhjurdwz-wubimtrl2egzfwzrufcxcqc3bu.eu-west-1.es.amazonaws.com/_cat/health

Delete

To delete resource on AWS side, run:

curl -LO https://storage.googleapis.com/capactio-terraform-modules/terraform.aws.elasticsearch.provision/0.1.0/module.tgz
tar zxvf module.tgz

capact typeinstance get 0fbb0212-bc2e-47de-a971-dbcbf46b39af -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
capact typeinstance get 0fbb0212-bc2e-47de-a971-dbcbf46b39af -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars

terraform init
terraform destroy # region eu-west-1

Helm chart

Install it via Dashboard.

Test

1. Get TypeInstance details:

   capact ti get b02c9803-f7f3-4c7d-b663-9b4c634986e6 -ojson | jq -r '.[0].latestResourceVersion.spec.value'

   Expected output:

   {
   "basicAuthSecretName": "elastic-basic-authmxpvp",
   "host": "elastic.capact.local",
   "port": 80,
   "protocol": "http",
   "version": "7.4.1"
   }

2. Check health:

   export USER=$(kubectl -n default get secret elastic-basic-authmxpvp  -ogo-template='{{.data.username | base64decode }}')
   export PASS=$(kubectl -n default get secret elastic-basic-authmxpvp  -ogo-template='{{.data.password | base64decode }}')
   
   curl -u $USER:$PASS https://elastic.capact.local/_cat/health

Delete

It's not needed. Resources will be removed on cluster deprovisioning.

Redis

Found issues:

• high memory consumption on Helm Template storage > 1.5G.
• In AWS Implementation the vpc_id, subnet_ids and node_type are required on additional parameters. This doesn't seem to be right. Our Action will be rendered but during execution it will fail with such error:

  2022-04-21T12:26:27.707Z	DEBUG	runner.terraform	terraform/terraform.go:103	Terraform output	{"output": "var.node_type\n  The compute and memory capacity of the nodes in the node group.\n\n  Enter a value: var.subnet_ids\n  List of VPC Subnet IDs for the cache subnet group.\n\n  Enter a value: var.vpc_id\n  VPC Id to associate with Redis ElastiCache.\n\n  Enter a value: \nError: No value for required variable\n\n  on variables.tf line 18:\n  18: variable \"node_type\" {\n\nThe root module input variable \"node_type\" is not set, and has no default\nvalue. Use a -var or -var-file command line argument to provide a value for\nthis variable.\n\nError: No value for required variable\n\n  on variables.tf line 23:\n  23: variable \"subnet_ids\" {\n\nThe root module input variable \"subnet_ids\" is not set, and has no default\nvalue. Use a -var or -var-file command line argument to provide a value for\nthis variable.\n\nError: No value for required variable\n\n  on variables.tf line 28:\n  28: variable \"vpc_id\" {\n\nThe root module input variable \"vpc_id\" is not set, and has no default value.\nUse a -var or -var-file command line argument to provide a value for this\nvariable.\n"}
  2022-04-21T12:26:27.708Z	ERROR	runner.terraform	runner/manager.go:77	while waiting for runner completion	{"runner": "terraform", "dryRun": false, "error": "terraform failed to finish: while running terraform: exit status 1", "errorVerbose": "exit status 1\nwhile running terraform\ncapact.io/capact/pkg/runner/terraform.(*terraform).Start.func1\n\t/capact.io/capact/pkg/runner/terraform/terraform.go:53\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nterraform failed to finish\ncapact.io/capact/pkg/runner/terraform.(*terraformRunner).WaitForCompletion\n\t/capact.io/capact/pkg/runner/terraform/runner.go:100\ncapact.io/capact/pkg/runner.(*Manager).Execute\n\t/capact.io/capact/pkg/runner/manager.go:75\nmain.main\n\t/capact.io/capact/cmd/terraform-runner/main.go:29\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}
  capact.io/capact/pkg/runner.(*Manager).Execute
    /capact.io/capact/pkg/runner/manager.go:77
  main.main
    /capact.io/capact/cmd/terraform-runner/main.go:29
  runtime.main
    /usr/local/go/src/runtime/proc.go:225
  2022/04/21 12:26:27 while executing runner: while waiting for completion: terraform failed to finish: while running terraform: exit status 1
  

  However, I didn't fix that. I just provided the defaults for a given region.
• In AWS Implementation the input password must contain from 16 to 128 alphanumeric characters or symbols (excluding @, ", and /). I added "minLength": 16, property to JSON Schema. We can also prepare the regex for it but this would be more problematic to maintain.

AWS

1. Create TypeInstance with AWS Credentials.

2. Create Action Policy to prefer AWS Implementation

   export AWS_TI_ID="76db17cc-6a72-4d3a-99d7-4c6cbf887446"
   cat > /tmp/action-policy.yaml << ENDOFFILE
   interface:
     rules:
       - interface:
           path: "cap.interface.database.redis.install"
         oneOf:
           - implementationConstraints:
               attributes:
                 - path: "cap.attribute.cloud.provider.aws"
             inject:
               requiredTypeInstances:
               - id: ${AWS_TI_ID}
                 description: "AWS credentials"
               additionalParameters:
                 - name: additional-parameters
                   value: # default for region eu-west-1
                     vpc_id: vpc-03c9859997d2e6850
                     subnet_ids:
                       - subnet-0286ffe48cefeb287
                       - subnet-0608e53e751752501
                       - subnet-023e6ddbe6ffa9583
                     node_type: cache.t3.small
       - interface:
           path: "cap.*"
         oneOf:
           - implementationConstraints:
               requires:
                 - path: "cap.core.type.platform.kubernetes"
           - implementationConstraints: {}
   ENDOFFILE

3. Create input parameters:

   cat > /tmp/aws-redis.yaml << ENDOFFILE
   input-parameters:
     name: "aws-redis-instance"
     password: ...trimmed...
   ENDOFFILE

4. Create an Action:

   capact act create cap.interface.database.redis.install --name "aws-redis" --parameters-from-file /tmp/aws-redis.yaml --action-policy-from-file /tmp/action-policy.yaml

5. Run and watch Action:

   capact act run aws-redis
   capact act watch aws-redis

Test

1. Get Action output:

   capact act get aws-redis -ojson | jq -r '.Actions[0].output.typeInstances'

2. Get TypeInstance details:

   capact ti get cb17e406-26ad-4c25-8996-1d23879fc8f7 -ojson | jq -r '.[0].latestResourceVersion.spec.value'
   

   Expected output:

   {
     "host": "master.aws-redis-instance-redis.jin6nd.euw1.cache.amazonaws.com",
     "password": "...trimmed...",
     "port": 6379
   }

I tried to reach the Redis instance via redis-cli but without success. However, we can see that it is up and running on AWS console. I also created a dedicated EC2 and check the connection between EC2 and Redis node, see: https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1#NetworkPath:pathId=nip-0e75c9c53af9c2a8f. As you can see the connection is possible. Maybe Redis itself deny the request. I ran out of time for further debugging. We also don't have a case where Redis is used.

Delete

To delete resource on AWS side, run:

git clone --depth 1 https://github.com/umotif-public/terraform-aws-elasticache-redis.git
cd terraform-aws-elasticache-redis

capact typeinstance get 9a1b2b10-539f-4abf-9e05-5cf3e41ac200 -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
capact typeinstance get 9a1b2b10-539f-4abf-9e05-5cf3e41ac200 -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars

terraform init
terraform destroy # region eu-west-1

Next delete the EC2 https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#InstanceDetails:instanceId=i-0c21aa8a04b0f7bde and analysis https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1#NetworkPath:pathId=nip-0e75c9c53af9c2a8f

Helm chart

1. Install Helm Storage and configure Global Policy
2. Install Redis via Dashboard. The Helm Implementation will be selected.

Test

1. Get Action output:

   capact act get local-redis -ojson | jq -r '.Actions[0].output'

   Expected output:

   {
     "typeInstances": [
       {
         "backend": {
           "abstract": false,
           "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
         },
         "id": "bcad8324-5148-4a83-aab3-b2a0da7afd87",
         "typeRef": {
           "path": "cap.type.database.redis.config",
           "revision": "0.1.0"
         }
       },
       {
         "backend": {
           "abstract": false,
           "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
         },
         "id": "5a7ec310-94ab-4b57-a5ee-ad241c128e99",
         "typeRef": {
           "path": "cap.type.helm.chart.release",
           "revision": "0.1.0"
         }
       }
     ]
   }

   As you can see, data is stored in external backends.

Rocket.Chat

Helm chart

1. Install Helm Storage and configure Global Policy
2. Install Rocket.Chat via Dashboard. The Helm Implementation will be selected.

Test

1. Create tunel:

   ssh -f -M -N -S /tmp/rocket.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:8088:rocket.capact.local:443

2. Update /etc/hosts on your localhost:

   127.0.0.1 rocket.capact.local

3. Navigate to https://rocket.capact.local:8088/

   Username: admin
   Password: admin
   

4. Get Action output:

   capact act get local-rocket -ojson | jq -r '.Actions[0].output.typeInstances'

   Expected output:

   [
     {
       "backend": {
         "abstract": false,
         "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
       },
       "id": "74502708-331e-47ea-beaa-9fb2a1d2764b",
       "typeRef": {
         "path": "cap.type.database.mongodb.config",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
       },
       "id": "1448c96d-8391-4752-9bce-2e6d0ee6fbc0",
       "typeRef": {
         "path": "cap.type.helm.chart.release",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
       },
       "id": "83ab4f5c-9190-4bec-baba-3fcdea85d665",
       "typeRef": {
         "path": "cap.type.helm.chart.release",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
       },
       "id": "365ce52c-1bca-463c-b76b-af32cb0b15a6",
       "typeRef": {
         "path": "cap.type.productivity.rocketchat.config",
         "revision": "0.1.0"
       }
     }
   ]

   As you can see, data is stored in external backends.

5. You can check all of them by running:

   capact ti get "74502708-331e-47ea-beaa-9fb2a1d2764b" "1448c96d-8391-4752-9bce-2e6d0ee6fbc0" "83ab4f5c-9190-4bec-baba-3fcdea85d665" "365ce52c-1bca-463c-b76b-af32cb0b15a6" -ojson | jq -r '.[].latestResourceVersion.spec.value'

   NOTE: This may take up to 1 min.

Delete

It's not needed. Resources will be removed on cluster deprovisioning.

Mattermost

PSQL Helm chart

1. Install Helm Storage and configure Global Policy
2. Install Mattermost via Dashboard. The Helm Implementation will be selected.

Test

1. Create tunel:

   ssh -f -M -N -S /tmp/local-mattermost.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:8089:mattermost.capact.local:443

2. Update /etc/hosts on localhost:

   127.0.0.1 mattermost.capact.local

3. Navigate to: https://mattermost.capact.local:8089/

   Username: admin
   Password: admin
   

4. Get Action output:

   capact act get local-mattermost -ojson | jq -r '.Actions[0].output.typeInstances'

   Expected output:

   [
     {
       "backend": {
         "abstract": false,
         "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
       },
       "id": "e71d0e84-051d-4c99-a682-30673d5167a5",
       "typeRef": {
         "path": "cap.type.database.postgresql.config",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": true,
         "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
       },
       "id": "642a5244-0be6-40bb-99f3-06c71e23d2f8",
       "typeRef": {
         "path": "cap.type.database.postgresql.database",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": true,
         "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
       },
       "id": "2c5c7fe1-589a-4343-a534-92923c771c7f",
       "typeRef": {
         "path": "cap.type.database.postgresql.user",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
       },
       "id": "ca43ae2d-f112-4719-90ae-3b186160e45f",
       "typeRef": {
         "path": "cap.type.helm.chart.release",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
       },
       "id": "db12a941-6e0c-4c12-88b5-90e8cd26ec93",
       "typeRef": {
         "path": "cap.type.helm.chart.release",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
       },
       "id": "ba5379b0-2cee-411a-9ad8-03364520e53c",
       "typeRef": {
         "path": "cap.type.productivity.mattermost.config",
         "revision": "0.1.0"
       }
     }
   ]

   As you can see, data is stored in external backends.

5. You can check Mattermost config by running:

   capact ti get -ojson ba5379b0-2cee-411a-9ad8-03364520e53c | jq -r '.[0].latestResourceVersion.spec.value'

   NOTE: This may take up to 30sec.

6. Clean-up:

   ssh -S /tmp/local-mattermost.vm.sock -O exit 18.184.179.238

Delete

It's not needed. Resources will be removed on cluster deprovisioning.

RDS

Found issues:

• We have such property port: {{ .port }} in goTemplate property. Unfortunately this generates such problem:

  2022/04/21 14:26:44 while executing runner: while reading runner input: while unmarshaling data from file "/terraform-args": error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".port":interface {}(nil)}

  However, we cannot quote it as in the output we expect the number and not string. To fix this problem, I had to address this issue: Change output.goTemplate in Terraform-based Implementations to be of type string #504.
• Another problem was the PostreSQL version:

  Warning: Argument is deprecated
  
    with module.db.module.db_instance.aws_db_instance.this,
    on .terraform/modules/db/modules/db_instance/main.tf line 34, in resource \"aws_db_instance\" \"this\":
    34:   name                                = var.name
  
  Use db_name instead
  
  (and 3 more similar warnings elsewhere)
  
  Error: Error creating DB Instance: InvalidParameterCombination: Cannot find version 11.10 for postgres
  \tstatus code: 400, request id: 0bb0f5eb-740f-4cf4-b5d6-7a7475cb7036
  
    with module.db.module.db_instance.aws_db_instance.this[0],
    on .terraform/modules/db/modules/db_instance/main.tf line 20, in resource \"aws_db_instance\" \"this\":
    20: resource \"aws_db_instance\" \"this\" {
  
  "}

  Was fixed by changing the default version directly in Implementation manifest.
• Two Mattermost installation cannot be in the same Namespace. There is a name conflict with created SA.
• For now we are not able to change the PostgreSQL Implemenation to RDS due to Storage backend ID for TypeInstance from nested workflow is lost during TypeInstance upload #705

                # Install DB
                - - name: install-db
                    capact-when: postgresql == nil
                    capact-action: postgresql.install
                    capact-outputTypeInstances:
                      - name: postgresql
                        from: postgresql
                        # Affected by https://github.com/capactio/capact/issues/705
                        # This needs to be removed to allow other non-Helm based Implementation
                        backend: helm-template-storage

  However, the installation works propertly. It's only a matter of outputting the TypeInstances.

Follow Install Mattermost with an external AWS RDS database tutorial. In my case it was:

1. Policy:

   export AWS_TI_ID="76db17cc-6a72-4d3a-99d7-4c6cbf887446"
   cat > /tmp/policy.yaml << ENDOFFILE
   interface:
     default:
       inject:
         requiredTypeInstances:
           - id: "3e4af86a-ab76-48c5-8799-1bcb0d994018"
             description: "Helm template"
           - id: "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
             description: "Helm release"
     rules:
       - interface:
           path: cap.interface.database.postgresql.install
         oneOf:
            - implementationConstraints:
                attributes:
                  - path: "cap.attribute.cloud.provider.aws"
              inject:
                requiredTypeInstances:
                - id: ${AWS_TI_ID}
                  description: "AWS credentials"
       - interface:
           path: cap.*
         oneOf:
           - implementationConstraints:
               requires:
                 - path: "cap.core.type.platform.kubernetes"
           - implementationConstraints: {} # fallback to any Implementation
   ENDOFFILE

2. Input params:

   cat <<EOF > /tmp/mattermost-install.yaml
   input-parameters:
     host: rds-mattermost.capact.local
   EOF

3. Action:

   kubectl create ns  rds-mattermost
   capact action create --namespace rds-mattermost --name rds-mattermost cap.interface.productivity.mattermost.install --parameters-from-file /tmp/mattermost-install.yaml --action-policy-from-file /tmp/policy.yaml

Delete

To delete resource on AWS side, run:

curl -LO https://storage.googleapis.com/capactio-terraform-modules/terraform.aws.rds.postgresql.provision/0.1.0/module.tgz
tar zxvf module.tgz

capact typeinstance get 681bd80b-1079-4c7d-b0ae-45fffb93c548 -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
capact typeinstance get 681bd80b-1079-4c7d-b0ae-45fffb93c548 -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars

terraform init
terraform destroy # region eu-west-1

Concourse

Found issues:

• Missing unpackValue: true for rendering GCP project name.
• Wrong input credentials syntax

nError: Attempted to load application default credentials since neither `credentials` nor `access_token` was set in the provider block.  No credentials loaded. To use your gcloud credentials, run 'gcloud auth application-default login'.  Original error: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: missing 'type' field in credentials

  with provider[\"registry.terraform.io/hashicorp/google\"],
    on main.tf line 46, in provider \"google\":
      46: provider \"google\" {

        "
      }

• Argo race issue:

       └───✖ helm-install                                                               concourse-install-helm-install-helm                                                                                    child 'concourse-3698900586' failed
           ├───✔ inject-input-type-instances-e0489c2c-8005-42f4-80ae-538c0c7d7cc2-step  inject-input-type-instances-e0489c2c-8005-42f4-80ae-538c0c7d7cc2                       concourse-2371691802  2s
           ├───✔ helm-install                                                           concourse-install-helm-install-helm-install                                            concourse-19934320    2m
           └───⚠ resolve-helm-rel-value                                                 concourse-install-helm-install-resolve-ti-art-value                                    concourse-3698900586  6s        Error (exit code 1): cannot enter chroot for container named "main": no PID known - maybe short running container

  To solve that issue, I modified the value-fetcher to run for at least 1 sec.

Cloud SQL

1. Create TypeInstance with GCP Credentials.

2. Action Policy:

   cat > /tmp/cloudsqlpolicy.yaml << ENDOFFILE
   interface:
     default:
       inject:
         requiredTypeInstances:
           - id: "3e4af86a-ab76-48c5-8799-1bcb0d994018"
             description: "Helm template"
           - id: "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
             description: "Helm release"
     rules:
       - interface:
           path: cap.interface.database.postgresql.install
         oneOf:
           - implementationConstraints:
               attributes:
                 - path: "cap.attribute.cloud.provider.gcp"
               requires:
                 - path: "cap.type.gcp.auth.service-account"
             inject:
               requiredTypeInstances:
                 - id: 29f9a769-efa6-4880-854f-187a6b682e18
                   description: "GCP Service Account"
       - interface:
           path: cap.*
         oneOf:
           - implementationConstraints:
               requires:
                 - path: "cap.core.type.platform.kubernetes"
           - implementationConstraints: {} # fallback to any Implementation
   ENDOFFILE

3. Input params:

   cat <<EOF > /tmp/concourse-install.yaml
   input-parameters:
     host: concourse.capact.local
   EOF

4. Action:

   kubectl create ns  cloud-concourse
   capact action create --namespace cloud-concourse --name concourse cap.interface.automation.concourse.install --parameters-from-file /tmp/concourse-install.yaml --action-policy-from-file /tmp/cloudsqlpolicy.yaml

Test

1. Create tunel:

   ssh -f -M -N -S /tmp/cloud-concourse.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:8022:concourse.capact.local:443

2. Update /etc/hosts on localhost:

    127.0.0.1 concourse.capact.local

3. Navigate to https://concourse.capact.local:8022/.

4. Get Action output:

   capact act get concourse -n cloud-concourse -ojson | jq -r '.Actions[0].output.typeInstances'

   Expected output:

   [
      {
        "backend": {
          "abstract": false,
          "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
        },
        "id": "84af3e01-0c6a-4046-adde-bf7b9ac79e05",
        "typeRef": {
          "path": "cap.type.automation.concourse.config",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "e55db908-a1d2-4e6c-81d4-e8676d6d1bf5",
        "typeRef": {
          "path": "cap.type.database.postgresql.config",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "d8bcdb6b-f857-4c38-bffe-b20dae9de914",
        "typeRef": {
          "path": "cap.type.database.postgresql.database",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "eca6ca74-75d0-405a-a4a8-ad5be04fa91b",
        "typeRef": {
          "path": "cap.type.database.postgresql.user",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": false,
          "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
        },
        "id": "7a8132b1-8c41-492b-8b4e-178640926882",
        "typeRef": {
          "path": "cap.type.helm.chart.release",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "fec082e7-6b03-4f32-865f-922809959d18",
        "typeRef": {
          "path": "cap.type.terraform.release",
          "revision": "0.1.0"
        }
      },
      {
        "backend": {
          "abstract": true,
          "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        },
        "id": "e486b409-622d-4978-bc12-0eba8ea3ea05",
        "typeRef": {
          "path": "cap.type.terraform.state",
          "revision": "0.1.0"
        }
      }
    ]

   As you can see, data is stored in external backends.

5. You can check Concourse config by running:

    capact ti get -ojson 84af3e01-0c6a-4046-adde-bf7b9ac79e05 | jq -r '.[0].latestResourceVersion.spec.value'

   NOTE: This may take up to 30sec.

6. Clean-up:

    ssh -S /tmp/cloud-concourse.vm.sock -O exit 18.184.179.238

Delete

To delete resource on GCP side:

1. Delete the Cloud SQL manually from GCP dashboard, navigate to https://console.cloud.google.com/sql/instances?project=capact.

2. Ensure that all resources are deleted, run:

   curl -LO https://storage.googleapis.com/capactio-terraform-modules/terraform.gcp.cloudsql.postgresql.install/0.2.0/module.tgz
   tar zxvf module.tgz
   
   capact typeinstance get abe156ca-dfd0-4c05-b213-7bd880865dce -ojson | jq -r '.[0].latestResourceVersion.spec.value.state' | base64 -d | sed -E 's/(.*"filename": ")(\/root)(.*)/\1~\3/g' > terraform.tfstate
   capact typeinstance get abe156ca-dfd0-4c05-b213-7bd880865dce -ojson | jq -r '.[0].latestResourceVersion.spec.value.variables' | base64 -d > terraform.tfvars
   
   terraform init
   terraform destroy # region eu-west-1

NOTE Unfortunately running the TF is not possible as the deletion_protection is enabled. You can run TF to change that setting but it's easier to do that via GCP dashboard.

MinIO

Issues:

• Argo time race:

           └───⚠ resolve-helm-rel-value                                                 deploy-helm-install-resolve-ti-art-value                          local-minio-3788856621  6s        Error (exit code 1): cannot enter chroot for container named "main": no PID known - maybe short running container

  This happens when the value is already provided:

  $ kc logs local-minio-3788856621 main
  2022-04-27T09:42:53.763Z	INFO	ti-value-fetcher	ti-value-fetcher/fetcher.go:48	Value already provided. Skipping...

  To solve that issue, I modified the value-fetcher to run for at least 1 sec.

Helm chart

1. Install Helm Storage and configure Global Policy
2. Install MinIO via Dashboard. The Helm Implementation will be selected.

Test

1. Get Action output:

   capact act get local-minio -ojson | jq -r '.Actions[0].output.typeInstances'

   Expected output:

   [
     {
       "backend": {
         "abstract": false,
         "id": "3e4af86a-ab76-48c5-8799-1bcb0d994018"
       },
       "id": "6105fad9-94e0-465c-a690-6e979e3118cb",
       "typeRef": {
         "path": "cap.type.database.object.aws.s3.config",
         "revision": "0.1.0"
       }
     },
     {
       "backend": {
         "abstract": false,
         "id": "b852a7b9-36bc-48bc-b655-a657ec61bcd8"
       },
       "id": "73716174-7fc0-4e7f-8236-1b1965133ea6",
       "typeRef": {
         "path": "cap.type.helm.chart.release",
         "revision": "0.1.0"
       }
     }
   ]

   As you can see, data is stored in external backends.
2. You can check MinIO config by running:

    capact ti get -ojson 6105fad9-94e0-465c-a690-6e979e3118cb | jq -r '.[0].latestResourceVersion.spec.value'

   NOTE: This may take up to 30sec.

3. Run MinIO client Pod:

   kubectl run my-mc --rm -i --tty --image minio/mc:latest --command -- bash

4. Login:

   mc alias set myminio/ http://local-s3-bucket.default.svc.cluster.local:9000 hsltgwucrnfsnwsi zwmbsifbrtdzohgj

5. Create, list and delete bucket:

   mc mb myminio/test
   mc ls myminio/
   mc rb myminio/test

Appendix

Forward Gateway

Create local tunel to Gateway.

1. Run on localhost:

   ssh -f -M -N -S /tmp/gateway.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:8082:gateway.capact.local:443

2. Log in into Gateway:

   capact login https://gateway.capact.local:8082 -u graphql -p t0p_s3cr3t

Forward Dashboard

Create local tunel to Dashboard.

1. Run on localhost:

   ssh -f -M -N -S /tmp/dashboard.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:8081:dashboard.capact.local:443

2. Navigate to https://dashboard.capact.local:8081.

Forward Neo4j browser

1. Run on VM:

   kubectl -n capact-system port-forward svc/neo4j-neo4j 7687 7474

2. Run on localhost:

   ssh -f -M -N -S /tmp/neo1.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:7687:127.0.0.1:7687
   ssh -f -M -N -S /tmp/neo2.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:7474:127.0.0.1:7474

3. Clean-up:

   ssh -S /tmp/neo1.vm.sock -O exit 18.184.179.238
   ssh -S /tmp/neo2.vm.sock -O exit 18.184.179.238

Forward Grafana

1. Run on VM:

   export CAPACT_NAMESPACE="capact-system"
   export USERNAME=$(kubectl -n "${CAPACT_NAMESPACE}" get secrets monitoring-grafana -ojsonpath="{.data.admin-user}" | base64 -d)
   export PASSWORD=$(kubectl -n "${CAPACT_NAMESPACE}" get secrets monitoring-grafana -ojsonpath="{.data.admin-password}" | base64 -d)
   
   echo "URL: http://localhost:3000"
   echo "Username: $USERNAME"
   echo "Password: $PASSWORD"
   
   kubectl -n "${CAPACT_NAMESPACE}" port-forward svc/monitoring-grafana 3000:80

2. Run on localhost:

   ssh -f -M -N -S /tmp/grafana.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:3000:127.0.0.1:3000

3. Clean-up:

   ssh -S /tmp/grafana.vm.sock -O exit 18.184.179.238

Force manifest repulling

kc exec -it -n capact-system capact-hub-public-5c45558575-7vhlj -c hub-public-populator -- pkill sleep
kc logs -n capact-system capact-hub-public-5c45558575-7vhlj -c hub-public-populator -f

Forward Argo

1. Run on VM:

   kubectl port-forward -n capact-system svc/argo-argo-workflows-server 2746

2. Run on localhost:

   ssh -f -M -N -S /tmp/argo.vm.sock -i ~/.ssh/provision-vm-bigbang.pem ubuntu@18.184.179.238 -L 127.0.0.1:2746:127.0.0.1:2746

3. Clean-up:

   ssh -S /tmp/argo.vm.sock -O exit 18.184.179.238

[mszostok]

PRs:

• Narrow-down the download TypeInstance details #707
• Update manifests hub-manifests#68
• Add prereq to Rocket.Chat doc, update goTemplate syntax website#122