[rfc] Separate Chains controller system and userspace identities.

[wlynch]

Writing down some thoughts that have been rattling in my head for a bit. Please leave any feedback + comments!

/cc @lcarva @chitrangpatel for explicit feedback, but feedback welcome from everyone.

tl;dr: I think we should separate Chains into 2 separate service account identities: trusted build provenance attestations and untrusted userspace signing.

Background

Chains today will sign 2 types of things:

• Build (i.e. Run) Provenance that it generates itself
• OCI images specified in Run output by user pipelines.

Today, we run Chains in a single reconciler with a single service account and single config - this means that the same key/identity that is used to sign build provenance is also used to sign OCI refs that users output from their pipelines.

This can make it hard to reason about the trust boundary of things Chains signs - especially if clients choose to do unexpected things like have Chains sign an "OCI image" that is actually another provenance document, or something that wasn't actually output by the Pipeline.

Proposal

Split the trust boundary into 2 parts separated by service account:

• tekton-chains (existing service account): service account that outputs trusted system-generated provenance.
• tekton-chains-userspace: service account that outputs "untrusted" (really depends on how much you trust the underlying pipelines) userspace derived signatures.

By separating by service account, we can guarantee distinct identities even with use of keyless signing. The consequence of this means that the controller will need to be split into 2 deployments/configs/etc. (one for each service account), though this also opens up a bunch of possibilities for different Chains deployments such as having a cluster-wide build provenance config (tekton-chains) and configurable namespace/tenant configs (tekton-chains-userspace).

Alternatives

• Remove OCI signing from chains and recommend everyone to sign in their Runs - I think there's value in having the auto-sign behavior, even if it doesn't give you as much metadata as signing within the build itself. I'd like to avoid this.
• Keep things as they are - This is going to be a problem for deployments where the Chains operators don't trust the user pipelines (this came up in a recent Slack thread). I think we need some answer.
• Same controller/service account, different configs / keys - This won't work for keyless signing, since they'd have the same identity.

[lcarva]

Thanks for filing this, @wlynch!

Overall, I do like the idea of doing something about the current state. We should arrive at a point where we can say anything signed by the Chains controller key/identity is trusted (i.e. the SLSA Provenance because it is generated by Chains).

The auto-signing of images is very convenient, but it is also very easy to misuse it without even realizing it. Users may get a false sense of security which, arguably, is worse than not having a signature at all.

Deploying the Chains controller, or some version of it, in a given namespace is an interesting approach. This would allow a user to leverage Chains to auto-sign their own images. This has a similar trust level to having the user sign their own images via a Run, I think. Basically, if the user has access to the signing key (long-lived or keyless), then they should be empowered to sign things themselves. (You build it, you sign it).

Maybe an approach we can take is to split the Chains controller, one part of it deals with generating and signing SLSA Provenance, while another deals with image signatures. If we make each configurable enough, we can enable a lot of interesting deployment configurations. I believe this is probably what you have in mind. I just want to clarify that I mean the split should likely happen at the controller level.

We should be mindful of the performance impact. We could unknowingly double the footprint of Chains. This may be a problem for users dealing with tens of thousands Runs.

[wlynch]

Maybe an approach we can take is to split the Chains controller, one part of it deals with generating and signing SLSA Provenance, while another deals with image signatures. If we make each configurable enough, we can enable a lot of interesting deployment configurations. I believe this is probably what you have in mind. I just want to clarify that I mean the split should likely happen at the controller level.

💯 Exactly!

There's a few ways we can configure this - we don't necessarily need per-namespace controllers (though this is also an option!) - we could have the centralized trusted control-plane provenance controller with a centralized userspace controller that knows how to reach in and get per-namespace configuration.

We should be mindful of the performance impact. We could unknowingly double the footprint of Chains. This may be a problem for users dealing with tens of thousands Runs.

Yeah this is a downside, but I think unavoidable to some degree since we need to have the service account isolation.
The good news though is by making the components separate, you could chose not to run the userspace controller. 🤷

[chitrangpatel]

Thanks @wlynch! I think splitting this up makes a lot of sense. I like the fact that cluster operators can choose not to run the userspace controller. 👍 from me.