#117 was evocative of some lessons that @vaikas-google and I have learned in the service-catalog project and I wanted to write them up because I think they are relevant to the question of cluster- or ns- scoped resources for Bus and possibly other resources in the eventing project.
In service-catalog, there is a thing called a 'service broker', which provides a set of 'services' (a capability that can be provisioned/deprovisioned) - which have 'plans' (tier of a service). Initially, we built a cluster-scoped set of resources to model these concepts in the service-catalog. This was a simple model to adopt and allowed us to avoid the problem of potentially having large numbers of namespaces that needed the same resources. It also allowed us to punt on the question of referencing services and plans across namespaces (cross-namespace references are currently a grey area in k8s).
We built this functionality first because it was easiest and convenient. However, we found that this was insufficient for how people wanted to use the catalog. What our users really wanted in the service-catalog project is a rich set of RBAC-driven ACL-filtering capabilities to share services/plans so that users had a view of these resources based on what they were permissioned to see. Unfortunately, there is not an tractable general path to ACL-filtering APIs in kubernetes, and there are some fundamental hurdles:
- k8s authorizers may be entirely external to the cluster and change out of band relative to k8s
- Not all authorizers can report a list of the subjects that are authorized for a resource
The experience that we could deliver (since we had to be backward compatible with the original cluster-scoped resources that we had released) was to have both cluster-scoped and ns-scoped versions of the broker, service class, and plan resources.
- The cluster scoped resources are things everyone in a cluster should be able to see/use
- The ns-scoped resources are used to put highly-privileged things into individual namespaces that should have access to them
- Developers can use ns-scoped resources to build new brokers and test them in the privacy of their own namespace
Note that in this model, access control is meaningful only at the level of a namespace - you cannot acheive further subdivision than the entire collection of a resource in a namespace. There are also two very similar sets of resources that have to be kept in sync - the cost is non-zero for doing two sets of resources.
More recently, in the cluster-registry, we had a similar tension between the cluster and namespace scope. The tension in this case is one of running the cluster registry as an aggregated API vs running as a standalone component. What we ultimately decided to do is:
- Make the
Cluster resource (that literally represents a k8s cluster) ns-scoped
- Have a well-known resource to hold the resources that should be available to everyone
I'm not sure if such a setup would be possible for us to use, but thought I would mention it for completeness. I think what this would imply for our API is that all references to the Bus type (as the example that started this braindump) would be implicitly namespace scoped and implicitly to the namespace of the referencing object unless a namespace was explicitly supplied. There are things to think through about how authorization checks would be performed and what failed checks would mean for references across namespaces. The benefit here would be more cohesive API - no jumping between cluster- and ns-scoped variants of the same resource, less API surface to learn, etc. The cost is that this is relatively unexplored territory, and being among the first to do something can be confusing and painful.
I realize that I'm late to the party here and certainly not demanding any work change course because of this, but I thought I would share my experiences here because they seem relevant. This is a tricky area and one where the UX that k8s can provide currently doesn't exactly match what users seem to want.
#117 was evocative of some lessons that @vaikas-google and I have learned in the service-catalog project and I wanted to write them up because I think they are relevant to the question of cluster- or ns- scoped resources for Bus and possibly other resources in the eventing project.
In service-catalog, there is a thing called a 'service broker', which provides a set of 'services' (a capability that can be provisioned/deprovisioned) - which have 'plans' (tier of a service). Initially, we built a cluster-scoped set of resources to model these concepts in the service-catalog. This was a simple model to adopt and allowed us to avoid the problem of potentially having large numbers of namespaces that needed the same resources. It also allowed us to punt on the question of referencing services and plans across namespaces (cross-namespace references are currently a grey area in k8s).
We built this functionality first because it was easiest and convenient. However, we found that this was insufficient for how people wanted to use the catalog. What our users really wanted in the service-catalog project is a rich set of RBAC-driven ACL-filtering capabilities to share services/plans so that users had a view of these resources based on what they were permissioned to see. Unfortunately, there is not an tractable general path to ACL-filtering APIs in kubernetes, and there are some fundamental hurdles:
The experience that we could deliver (since we had to be backward compatible with the original cluster-scoped resources that we had released) was to have both cluster-scoped and ns-scoped versions of the broker, service class, and plan resources.
Note that in this model, access control is meaningful only at the level of a namespace - you cannot acheive further subdivision than the entire collection of a resource in a namespace. There are also two very similar sets of resources that have to be kept in sync - the cost is non-zero for doing two sets of resources.
More recently, in the cluster-registry, we had a similar tension between the cluster and namespace scope. The tension in this case is one of running the cluster registry as an aggregated API vs running as a standalone component. What we ultimately decided to do is:
Clusterresource (that literally represents a k8s cluster) ns-scopedI'm not sure if such a setup would be possible for us to use, but thought I would mention it for completeness. I think what this would imply for our API is that all references to the Bus type (as the example that started this braindump) would be implicitly namespace scoped and implicitly to the namespace of the referencing object unless a namespace was explicitly supplied. There are things to think through about how authorization checks would be performed and what failed checks would mean for references across namespaces. The benefit here would be more cohesive API - no jumping between cluster- and ns-scoped variants of the same resource, less API surface to learn, etc. The cost is that this is relatively unexplored territory, and being among the first to do something can be confusing and painful.
I realize that I'm late to the party here and certainly not demanding any work change course because of this, but I thought I would share my experiences here because they seem relevant. This is a tricky area and one where the UX that k8s can provide currently doesn't exactly match what users seem to want.