decision: Segmenting API Gateway features into PolicyAttachment CRDs

[arkodg]

Description:

Adding options for how API Gateway features such as ratelimiting can be expressed as PolicyAttachment CRDs. Please dont focus on the names, just added placeholders.

1. one generic policy CRD with features as top level fields

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Policy
 .....
 spec:
   authentication:
   .....
   ratelimiting:
   ....

2. one generic policy CRD with networking (shapes traffic) and security (authenticates & authorizes traffic) as top fields

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Policy
 .....
 spec:
   security:
     authentication:
     ....
   networking:
     ratelimiting:
     .....

3. 2 Policy CRDs - one for networking and the other for security

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: SecurityPolicy
 ....
 ....
 spec:
   authentication:

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: NetworkingPolicy
 ....
 spec:
   ratelimiting:
   ....

4. 1 CRD per API Gateway feature

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Authentication
 ....
 spec:
   jwt:
   ....

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: Ratelimiting
 ....
 spec:
   ....

[arkodg]

@Xunzhuo @danehans @youngnick @skriss @LukeShu @AliceProxy can you please vote on this issue

[Xunzhuo]

I prefer 3 or 4, they have more independence and loose coupling between features.

[arkodg]

I vote for 2. or 3. and here's why

• 1 or 2 CRDs are easier to maintain for the project than multiple
• The user can express their intent in 1 instantiated resource (useful when team is small) or multiple ( useful when multiple teams author intent)

[Alice-Lilith]

I think 3 would be preferable over 1 and 2 although I prefer 4 the most. Allows for clear separation of resources depending on what the user is trying to accomplish. Prioritize keeping things self contained. While you could group two features into a single group like networking / security I feel that when a user has the goal of setting up a ratelimt they shouldn't need to worry about what other resources exist and then check to see if they have ratelimits configured on them and need to be edited vs. extended versus creating a new resource.

IE: I'm a cluster admin and I know that I want to setup authentication, I shouldn't have to use kubectl for some other resource and then check which of those resources configure authentication vs. a different security feature. I should just be able to quickly list all the resources in the cluster by the feature I'm trying to configure and get a rough idea from the info via kubectl get that I dont need to dig into any thing more in depth to figure out what my cluster config is.

Makes it easier for us to iterate over each individual feature separately. Not sure that the further grouping of a bunch of different things together is really benefiting the user but I'd like to hear any counter arguments.

IE: kubectl get ratelimits let's me roughly see the ratelimiting config for the cluster and then I can dive into a specific one if needed based on the previous info.

Vs. kubectl get policy I can see there are a bunch of policies configured for the cluster but I have to go into more depth with each of them to figure out what all is being configured.

Just to make my vote more clear:
4 > 3 > 2 > 1 (4 being the most desirable and 1 being the least)

[sunjayBhatia]

Also, see the Policy Boilerplate section which uses MUST language around the default and override fields for Policies. The hierarchical nature and defaulting/override behavior is a core concept of the Policy resource. Using more focused resources IMO leads to a lot less complexity for implementers and users when taking this into account.

[arkodg]

Policy Boilerplate

I think the complexity of merging individual policies is the same for 3 and 4

3 just gives the flexibility of merging personas into one (a small independant team) or allowing
individual personas to author and instantiate many policy resources

apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: SecurityPolicy
 ....
 ....
 spec:
   authentication:
   .....
   cors:
   .....
   default:
     ......  
     authentication:
     .....
     cors:
     ....

[arkodg]

I think 3 would be preferable over 1 and 2 although I prefer 4 the most. Allows for clear separation of resources depending on what the user is trying to accomplish. Prioritize keeping things self contained. While you could group two features into a single group like networking / security I feel that when a user has the goal of setting up a ratelimt they shouldn't need to worry about what other resources exist and then check to see if they have ratelimits configured on them and need to be edited vs. extended versus creating a new resource.

IE: I'm a cluster admin and I know that I want to setup authentication, I shouldn't have to use kubectl for some other resource and then check which of those resources configure authentication vs. a different security feature. I should just be able to quickly list all the resources in the cluster by the feature I'm trying to configure and get a rough idea from the info via kubectl get that I dont need to dig into any thing more in depth to figure out what my cluster config is.

Makes it easier for us to iterate over each individual feature separately. Not sure that the further grouping of a bunch of different things together is really benefiting the user but I'd like to hear any counter arguments.

IE: kubectl get ratelimits let's me roughly see the ratelimiting config for the cluster and then I can dive into a specific one if needed based on the previous info.

Vs. kubectl get policy I can see there are a bunch of policies configured for the cluster but I have to go into more depth with each of them to figure out what all is being configured.

thanks for raising the argument around retrieving current intent.

• from a production point of view, the intent will version controlled / git and checked before being edited
• from a live state perspective, I can understand the case when the user only wants to retrieve the ratelimit block having a command like kubectl get ratelimit is easier than kubectl get trafficpolicy, but there are also cases (and I think this is more of a common case then edge case) where this same user owns and cares about multiple other traffic/networking properties such as loadbalancing and prefers all features being clubbed into the same resource.
  • there's also the case where the user doesnt remember the names of all the CRDs, and wants to remember the bare minimum to achieve their goal, so they might not be able to / or want to remember the exact names of each CRD such as ratelimiting, loadbalancing, waf, cors

[danehans]

xref #675 as @sunjayBhatia has spent considerable time researching this topic for Contour. As I stated here, I suggest EG focus on extensions that can be implemented as implementation-specific filters. EG can add Policy Attachment support at a future date to support overrides and defaulting.

[danehans]

+1 on 4

[lizan]

I would also +1 on 4 (non-maintainer), that gives each feature independence and they can evolve in their own pace as well.

[zinuga]

+1 for 2 and 3 as they align with well with networking and security persona

[danehans]

2 and 3 as they align with well with networking and security persona

Note that 4 meets this requirement while supporting more granular RBAC.

[arkodg]

xref #675 as @sunjayBhatia has spent considerable time researching this topic for Contour. As I stated here, I suggest EG focus on extensions that can be implemented as implementation-specific filters. EG can add Policy Attachment support at a future date to support overrides and defaulting.

• for features like auth it might need to be enforced, so policy fits better for that case imho
• the deprecation for policy and extension filter will look similar once the features have been graduated into GAPI, so no difference from that perspective