Support dynamic TypeInstances in umbrella workflow

[pkosiec]

Description

This is an outcome of #688, discussed and agreed with @mszostok. The solution is to use value field to store the rendered value, even for dynamic TypeInstances (the ones which has dynamic values fetched based on context).

AC

• Split gRPC API (resolved in Describe approach for dynamic TypeInstances in nested workflows and update gRPC API #696)

  • Dynamic and static Storage

  • Update code in backends and Hub

  • Add new gRPC method to the dynamic (context-only) Storage:

    message GetPreCreateValueRequest {
      // no TypeInstance ID and resourceVersion
      bytes context = 1;
    }
    
    message GetPreCreateValueResponse {
      optional bytes value = 1;
    }
    
    service StorageBackend {
      rpc GetPreCreateValue(GetPreCreateValueRequest) returns (GetPreCreateValueResponse);
    }

• Update JSON schema for Storages (instead of acceptValue, there will be an enum)

  • Not valid anymore, as there is no nice way of services composition in ProtoBuf. Every service would need to implement multiple gRPC servers and multiplex them (e.g. dynamic TI + locking TI)

• Update Mattermost installation manifest and fully use Helm template backend for PostgreSQL (Use Helm template storage backend for PostgreSQL and Mattermost installation hub-manifests#66)

  • Add generic container to enrich TypeInstance (run GetPreCreateValue) at the end of a given workflow
    • Manually added by Content Developer
    • Add the following steps after Helm installation:

  - - name: resolve-dynamic-ti-value
      template: resolve-dynamic-ti-value
      arguments:
      - name: input-artifact
        from: "{{steps.resolve-dynamic-ti.outputs.artifacts.postgresql}}"
      - name: backend
        # fortunately it is already injected into workflow
        from: "{{workflow.outputs.artifacts.helm-template-storage}}"

• Modify TypeInstance uploader: TypeInstance upload would ignore TypeInstance value if a given Storage Backend (we have its ID) is dynamic (accepts only context)

  • some additional logging could be helpful

• Support GetPreCreate for Helm storage backends

• Update documentation on website (resolved in Describe Storage Backend ProtoBuf API changes and TypeInstance Value Fetcher website#119)

Reason

Previously, every downloaded TypeInstance in a workflow had a form of an artifact with the TypeInstance value.
Now, every TypeInstance have:

value: {} # static value, or resolved value based on context
backend:
  context: {} # additional context for storage backend

Jinja2 templating and Artifact Merger already support unpacking value.

Sometimes the artifact could contain just the backend.context. For example, if we took an artifact prepared for Helm Template storage:

backend:
  context:
    chartLocation: https://charts.bitnami.com/bitnami
    driver: secrets
    name: postgresql-1648472938
    namespace: default
value: null

The problem is that some of further workflow steps might need actual values, and the value is null.

Related issues

See #688