Allow users to configure xDS resources

[arkodg]

Users might want to directly configure xDS Resources in case Envoy Gateway's API doesn't support their use case.
They might want to add/merge/remove/replace xDS resources that are pushed by Envoy Gateway into Envoy Proxy.
Projects such as Istio provide APIs such as EnvoyFilter to fulfill these use cases.

Outlining work items

• Define and Design API feat: EnvoyPatchPolicy API #1410
• Implement API in the Xds Translator Implement JSON Patch in Xds Translator #1606
• Implement support in egctl for offline verification feat: egctl x translate support for EnvoyPatchPolicy #1682
• Add status to Policy to show runtime status feat: EnvoyPatchPolicy Status #1691
• E2E e2e & misc fixes for EnvoyPatchPolicy #1738
• Docs docs: Add user docs for EnvoyPatchPolicy #1733

[arkodg]

Here's one way to do it

• Lets try and implement Global Ratelimiting directly using xDS

• Lets define a new CRD called EnvoyPatch which can be used to represent the xDS resource to be patched.

// Defines a generic resource patch
message Patch {
  // Define the envoy configuration change as a JSONPatch
  // defined in https://datatracker.ietf.org/doc/html/rfc6902.
  // $hide_from_docs
  message JsonPatch {
    // Operations that can be performed for the patch.
    enum Operation {
      INVALID = 0;
      // If the path specifies an array index, the value
      // is inserted at the specified index.
      // If it specifies an object member that does not
      // already exist, a new member is added.
      // If the path specifies a member that does exist,
      // the value replaces the existing member value.
      ADD = 1;
      // Removes the value from the location specified in
      // the path.
      REMOVE = 2;
      // Replaces the value in the path location with the new value.
      REPLACE = 3;
    }
    // Patch operation
    Operation op = 1;
    // JSON Pointer for identifying the value based on
    // https://datatracker.ietf.org/doc/html/rfc6901.
    string path = 2;
    //  Resource patch.
    google.protobuf.Struct value = 3;
  }
  // Defines a partial envoy configuration message that can be patched
  // into the corresponding resource.
  message ProtoPatch {
    // Operations that can be performed for the patch.
    enum Operation {
      INVALID = 0;
      // Patch using proto merge semantics.
      MERGE = 1;
      // Add/Append the patch as a resource to its corresponding resource list.
      ADD = 2;
    }
    Operation op = 1;
    // Resource patch.
    google.protobuf.Struct value = 2;
  }
  oneof patch_type {
    // Patch using proto semantics.
    ProtoPatch proto_patch = 1;
  }
}

message  EnvoyPatch {
  oneof resource_type {
      // Apply envoy listener patches to the listener resource
     Patch listener = 1;
     // Apply envoy listener patches to the http_connection_manager resource
     Patch http_connection_manager = 2;
     // Apply envoy listener patches to the virtual_host resource
     Patch virtual_host = 3;
     // Apply envoy route patches to the route resource
     Patch route = 4;
     // Apply envoy cluster patches to the cluster resource
     Patch cluster = 5;
    
  } 
}

• Lets use Custom Filters in HTTPRoute to patch VirtualHost & Cluster resources. The major assumption here is that there is a 1:1 relationship/translation between HTTPRoute & Route as well as HTTPRoute & Cluster. This ensure consistent merging outcomes.

• One side effect here is that the Envoypatch tied to the HTTPConnectionManager or Listener will impact other HTTPRoutes attached to the same Gateway (that contains the listener fields) .

• User Intent using Gateway API

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: eg-gw
spec:
  gatewayClassName: eg-gc
  listeners:
    - name: example
      protocol: HTTPS
      port: 443
      hostname: example.com
      tls:
        certificateRefs:
          - kind: Secret
            group: ""
            name: example-cert
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1alpha2
metadata:
  name: example
spec:
  parentRefs:
    - name: eg-gw
  hostnames:
    - "example.com"
  rules:
    - filters:
        - type: ExtensionRef
           extensionRef:
            - group: config.eg.io
               kind: EnvoyPatch
               name: rlConnMgrPatch      
        - type: ExtensionRef
           extensionRef:
            - group: config.eg.io
               kind: EnvoyPatch
               name: rlVirtualHostPatch   
        - type: ExtensionRef
           extensionRef:
            - group: config.eg.io
               kind: EnvoyPatch
               name: rlClusterPatch           
      backendRefs:
        - name: example
           port: 8080
---
apiVersion: config.eag.io/v1alpha1
kind: EnvoyPatch
metadata:
  name: rlClusterPatch
spec:
  cluster:
    protoPatch:
      op: ADD
      value:
      name: rate_limit_cluster
      type: STRICT_DNS
      connect_timeout: 10s
       lb_policy: ROUND_ROBIN
        http2_protocol_options: {}
        load_assignment:
          cluster_name: rate_limit_cluster
          endpoints:
           - lb_endpoints:
              - endpoint:
                   address:
                     socket_address:
                       address: ratelimit.svc.cluster.local
                       port_value: 8081
---
apiVersion: config.eag.io/v1alpha1
kind: EnvoyPatch
metadata:
  name: rlVirtualHostPatch
spec:                       
  virtualHost:
    protoPatch:
      op: MERGE
        value:
           rate_limits:
             - actions:
                - remote_address: {}                       
---
apiVersion: config.eag.io/v1alpha1
kind: EnvoyPatch
metadata:
  name: rlConnMgrPatch
spec:                       
   httpConnectionManager:
       protoPatch:
         op: MERGE
         value:
           http_filters:
           - name: "envoy.filters.http.ratelimit"
             typed_config:
               "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit"
               domain: "eag-ratelimit"
               failure_mode_deny: true
               timeout: 1s
               rate_limit_service:
                 grpc_service:
                   envoy_grpc:
                     cluster_name: rate_limit_cluster
                 transport_api_version: V3

[danehans]

A key point from the above EnvoyFilter reference:

This feature must be used with care, as incorrect configurations could potentially destabilize the entire mesh.

I believe @LukeShu shared concerns about exposing xDS config through the API.

I can see the need to support this but IMHO it's a long-term goal since the project has a ton of work to implement the initial system design and support the core/extended Gateway APIs.

[arkodg]

sure, this might not be part of the first release but since this is highlighted in the goals, im kickstarting the discussions, since it might require upstream API changes.

[youngnick]

Yes, one of the reasons I've been very reluctant to do anything like this for Contour in the past is because the HTTPConnectionManager filter chains have to be constructed carefully to get the outcomes we want; I feel like it's a bit too easy to create a config that has a bad interaction with a designed behavior.

[mattklein123]

IMO I would avoid any kind of patch semantics at all. I think it's too confusing and error prone. The way I would model this is:

1. You are either using the official API (no exceptions)
2. You are in xDS mode and you are on your own, and we will offer a conversion that will spit out (1) -> (2) to get you started.

IMO this is the best path forward.

[arkodg]

cool, sg, should we start off with the same approach - all official EG or all BYO, for Bootstrap as well ?

[danehans]

cool, sg, should we start off with the same approach - all official EG or all BYO, for Bootstrap as well ?

+1 for consistency

[youngnick]

I definitely agree with @mattklein123 - part of the point of the upstream API is to have extension points to allow users to model concepts that we may use xDS to implement - there are a number of extension points and the Policy APIs for this exact reason.