Decouple 3rd party integrations from Concourse #57
Conversation
|
|
||
| Define interfaces and externalize the integrations such that adding new integrations or updating existing integrations doesn't require a new version of Concourse. | ||
|
|
||
| One way to achieve this would be to use [gRP](https://grpc.io/) where each integration type would have a `service` defining the interface. |
There was a problem hiding this comment.
@aoldershaw also brought my attention to doing something similar for the build events storage
|
|
||
| # Open Questions | ||
|
|
||
| How do we manage changes to the interface ? |
There was a problem hiding this comment.
I think we should follow Terraform's lead on this, particularly wrt how they support providers (and provisioners I believe). They use their in-house https://github.com/hashicorp/go-plugin, which supports gRPC-based plugins. From what I can tell, the benefit to using go-plugin over gRPC alone is that it supports some form of plugin versioning (I haven't looked to much into it) on both the client and server side.
Worth taking a look at how it's used in Terraform: https://github.com/hashicorp/terraform/tree/master/plugin
There was a problem hiding this comment.
cool find. will take a look 👀
There was a problem hiding this comment.
For completeness, the Terraform plugin code has been extracted into a separate project. The latest version of the code can be found here: https://github.com/hashicorp/terraform-plugin-sdk
|
|
||
| There are many examples of successfully doing this in the larger community such as the [containerd project](https://www.slideshare.net/Docker/driving-containerd-oprations-with-grpc). | ||
|
|
||
| # Open Questions |
There was a problem hiding this comment.
One question I have is how we would go about supporting plugin configuration. Currently, we have specific flags for e.g. each credential manager. One approach is to simply pass along environment variables with a specific prefix (e.g. CONCOURSE_CREDS_*) to the plugin, and allow the plugin to interpret them - but we lose the ability to see what config options are available from concourse web -help (which is maybe okay?)
One idea to remediate this is to come up with some interface for plugins to expose their configuration options to Concourse (via gRPC)
There was a problem hiding this comment.
Yep, the approach of passing along the env vars with a prefix makes sense (there is precedence here with Garden + Concourse already) and would keep the plugin and Concourse decoupled.
And subsequently tackling a way to surface the plugin configuration options to the user via a flavor ofconcourse -h would be a awesome enhancement
| @@ -0,0 +1,50 @@ | |||
| # Summary | |||
|
|
|||
| Concourse has several types of integrations such as metrics and credential stores. For each type, there are several implementations such as Datadog, New Relic, Prometheus, etc. | |||
There was a problem hiding this comment.
It's probably worth cross-referencing var sources (#39) and prototypes (#37) as those are planned to be combined like peanut-butter and jelly to achieve decoupling of credential management in Concourse. The latest revision introduces support for encrypting which was the missing piece of the puzzle.
There'd still be a few direct integrations remaining: policy enforcement (OPA is currently the only implementation, but theoretically it's generic), metrics, and tracing. Using Prototypes for these scenarios is probably impractical since those would benefit from lower latency and high throughput.
| # Open Questions | ||
|
|
||
| How do we manage changes to the interface ? | ||
| How does this work impact standardizing on [opentelemetry](https://github.com/concourse/concourse/pull/5104) for metrics ? |
There was a problem hiding this comment.
I think this boils down to the general question: what will be the API within Concourse for a certain integration that is powerfull enough so it can support multiple external implementations as plugins?
Let's focus on this observability angle: is OpenTelemetry the API to use inside of Concourse or is a Concourse specific API needed delegating to an OpenTelemetry specific implementation? There are arguments to say on both approaches.
Looking at the OpenTelemetry website, I read:
OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.
The client side delegates to a collector for further processing. This is your API & plugin. So for observability, I would just pick OpenTelemetry as the API to integrate in the Concourse codebase and everybody will be able to plug it to their own observability platform, whether OpenSource or commercial.
Putting yet another API in front of OpenTelemetry would be a waste of cycles and nobody would win here IMO.
|
Looking at this rfc, I'm not sure the scope is clear... May I suggest the focus be placed on the plugin system for Concourse integrations and let each plugin (for metrics, secrets, etc.) have there own rfc? |
|
After checking with @xtreme-sameer-vohra I think we can just close this. We have a path to having extensible credential managers with #27 which proposes that they be implemented with prototypes (#37). Metrics providers are still baked in but we're keeping an eye on OpenTelemetry which could help address that problem with a more modular approach. In any case a separate fresh RFC is probably the best way to go from here. |
5 participants
See proposal.md