Implement Delegated Storage

[pkosiec]

Description

Implement Delegated Storage according to the accepted proposal.

• Implement additional references to parent nodes for Type #601 [3MD]
  - Make sure Populator supports it
  - Support it in GraphQL API
  - Find Types based on prefix of parent nodes (consider refactoring Cypher query to TypeScript)
• Validate TypeInstance input during TypeInstance creation #544 [3MD]
• Investigate existing solutions to speed up Local Hub rewrite #626 [2MD]
• Rewrite Local Hub to Go #605 [13MD] dropped
• Implement gRPC delegation in Local Hub #645 [13MD]
• Handle multiple backends for the TypeInstance upload from workflow #634
• Implement Helm storage backend logic #650
• Adjust all Implementation manifests to use latest workflow syntax and storage backends #666
• Add e2e support for setting backend.id (Policy, Workflow, GraphQL) #624
  • Add new properties
  • (new scope) Extend capact-outputTypeInstances syntax
  • Handle common TypeInstance injections
• Add common TypeInstance injection #635
• Update Content development guide after delegated storage #667
• Implement storage backends
  • Vault [builtin: 5MD / external: 8MD - to drop in minimal scope]
  • AWS Secrets Manager
  • Prerequisite: Helm Runner
• Validate Type for storage backends #691
• Improve readability of context schema  #687

NEW

• Prevent TypeInstance deletion when it's used by others #632
• Implement AWS Secrets Manager storage backend logic #637
  • Configure Secret storage backend to be used with Local Hub #647
• Investigate dynamic TypeInstances support in umbrella workflows #688
• Support dynamic TypeInstances in umbrella workflow #695

Total SP:
- Minimal scope: 46 SP + padding 15% = 53 SP (~ 7 weeks)
- Target architecture: 74 SP + padding 15% = 85 SP

Additional subtasks (not required for minimal scope)

• Investigate our approach to implementation [3MD - to drop in minimal scope]
  taking into account the Potential feature evolution which may impact implementation
• Implement Go Template storage backend service #665
• Add an option to update backend for already existing TypeInstance #636

AC

• in each sub-task check for TODO(https://github.com/capactio/capact/issues/604) and if applicable, resolve it.

Reason

We want to be able to consume and save data into external backends. In that way we can integrate easily with 3rd party applications/services/platforms. We will solve also the security issues as one of the delegated storage can be e.g., Vault. Another benefit is to get rid of Neo4j and make Capact lightweight.

Why it's worth to replace the Neo4j:

• It’s a resource intensive
• Better local dev/test experience with lightweight alternative (not in Java)
• Option to create interactive tutorials, e.g. using katacoda: https://www.katacoda.com/. In that way, you can try our solution e2e without installing anything on your local machine. Currently, neo4j takes too much CPU and memory.
• GPL3 license will scare people away
• Not fully optimized for our use case (we got it working but it’s not ideal)

SCOPE: Only the Local Hub. Probably in the next phases this can be integrated/reused to other parts of our system, e.g., Public Hub.