Decide on the Deployment Model

[arkodg]

Outlining various deployment models that allow the project to achieve its goals

CP = Control Plane
DP = Data Plane

1. Combine CP and Envoy (DP) into a single deployment

• Similar to what is done in Emissary today
• Every CP watches for a specific GatewayClass, as well as the Gateway and HTTPRoute resources linked to it
• Pros
  • Simple to deploy & operate
  • Makes internal & user cert management simpler (Unix domain sockets b/w Envoy and xDS Server)
  • Simpler upgrades
• Cons
  • More load on k8s API Server
  • Resource consumption increases - CPs are duplicating work
  • Aggregation of stats/status across all identical controllers moves into the CLI, external layer

2. Provision a unique CP and Envoy (DP) per GatewayClass

• Similar to what is done in Contour today
• The main provisioner watches for a specific GatewayClass, and creates a CP per GatewayClass
• Each CP watches for Gateway and HTTPRoutes and configures the xDS Server tied to it
• Pros
  • Simpler CP
  • Envoy scales with App load but CP doesnt
  • API server load independent from App load
  • Isolates user certs by isolating xDS Servers
• Cons
  • Non trivial Authn + Authz between CP and Envoy
  • More containers/deployments to manage & operate
  • Another communication channel over the network between the main provisioner and every CP to aggregate stats/status
  • Non trivial upgrades

3. Common CP with a unique Envoy (DP) per GatewayClass

• Similar to what is done in Istio today

• The xDS Server is shared across all Envoys

• The DP component also contains an agent (similar to Istio-agent) that performs SDS local to the Envoy

• Pros

  • Most efficient model
  • Easier to aggregate stats
  • Easier to collect cluster state

• Cons

  • Non trivial upgrades
  • Non trivial Authn + Authz between CP and Envoy
  • Implement & Maintain a DP Sidecar Agent / SDS Server

Relates to #40 & #41

[alexgervais]

A couple of questions...

• In option 3, does the CP watch for a single GatewayClass (and all of its Gateway), or does it watch for all GatewayClass?
• Adding a provisioner to option 3, how would it be different to option 2?

And some thoughts...

• Aiming for simplicity, option 1 seems the easiest to secure, operate, "get started", and has the lowest friction for common operations like upgrades (prevents version drifts between CP and DP)... although it seems like an antipattern in most Envoy Proxy deployments. The issue is scalability and fine-tuning.
• We might simplify option 2 by including the provisioner inside the CP (to manage Envoy Proxy fleet) as opposed to having a CP-provisioner. This would better align with decisions to run a single component. Engineering Decisions to Prepare Implementing Design Doc #60, Initial Config Management Design #87

[arkodg]

A couple of questions...

• In option 3, does the CP watch for a single GatewayClass (and all of its Gateway), or does it watch for all GatewayClass?

The assumption made above was that the CP would watch for all GatewayClasses. It could watch for a single Gatewayclass which would simplify the design for 3, eliminating the need for an explicit SDS Server / Agent per DP Instance (which was suggested to isolate creds).

• Adding a provisioner to option 3, how would it be different to option 2?

Naming is tricky, in the above diagram, the provisioner represents a parent CP (watching for GatewayClass resources) that provisions a child CP per GatewayClass (that watches & manages sub resources - Gateway etc ) . 1, 2 & 3 would require a DP Provisoner.

[danehans]

+1 for option #2. IMHO it strikes a balance between options 1 and 3 (simplicity and flexibility).

[skriss]

Personally I'd lean towards 2 or 3, in order to be able to scale the control plane and data plane independently. I like that with option 2, you have separate failure domains for separate Gateways, so e.g. an issue with an internal Gateway/CP doesn't necessarily affect an external-facing Gateway, but agree it's more workloads to manage.

A couple clarifications on Contour's model, just for reference:

• the provisioner watches all GatewayClasses with a given controller name (projectcontour.io/gateway-controller)
• a separate CP and Envoy fleet is provisioned for each Gateway (not each GatewayClass).
• each CP programs its corresponding Envoy fleet based on the Gateway it's for, plus the associated HTTPRoutes

[lizan]

Yeah agreed on 2 or 3. IMO they might not be different that much at the end, so we can start with 2 and potentially support 3 model in the future.

[arkodg]

I think 2 & 3 are great long term solutions that allow the project to handle higher scale as well as consolidate multiple config classes /GatewayClass.
However I'd prefer if the project started with 1

• This would allow the initial phase of the project to concentrate on UX, ease of use and features
• This would add less operational burden on early users adopting this project who might need to upgrade often since the project will likely end up releasing more often in the beginning.
• 1 will be easier to test & run adding more velocity to the project

[youngnick]

I think that 1 is not a good idea. Although it seems like it's easier to manage, it adds big problems later once you're in prod, in that it's much harder to do upgrades without dropping traffic than if you can upgrade the control plane and data plane independently. This is why Contour moved from a single-Pod model like 1 to the split model it has today.

I am very against a multitenanted control plane model because whatever engineering effort we save in having less things running, we will have to pay in careful security engineering to ensure that it's impossible for one data plane tenant to access another data plane tenant's data. This is particularly important in the ingress case, because the secret data often includes keypairs for whole site serving TLS certificates, or even worse, keypairs for wildcard serving certs. It is absolutely a solvable problem, but by keeping the differrent parts separate, we separate the security blast radius as well as the operational one.

In terms of authenticating between the control plane and Envoy, Contour has solved this problem in the simple case, using an on-disk SDS for hot-reload of certificates, an install process that generates one-year validity certs for both Contour and Envoy, and updates them on each upgrade. The update is seamless because the secrets change "on disk", Envoy sees the SDS change, and then control plane connection is rebuilt in less than 30 seconds. Notably, because it uses Kubernetes Secrets as intended and can handle rotations whenever you're ready, it plugs in easily to Vault or other secret-management tools - the Secret admin just has to ensure that the Secrets are updated for both Envoy and Contour around the same time.

I would also argue that 2's upgrades, although they seem harder, are actually operationally better because they give control back to the user over when various parts are upgraded. The downside is that the control plane needs to wait some period (Contour uses at least 1 release) before it can start using features added in a new Envoy version.

Although Contour's current model installs an CP and DP per Gateway at the moment, that is not required by the architecture or the Gateway API . The Gateway API has explicit support for merging Gateway config, in order to facilitate resource-sharing and reduce the operating cost of running the CP.

With respect to how many GatewayClasses to watch, we don't do a good job of making this clear in the Gateway API docs, but I've always considered a GatewayClass to be equivalent to a controller plus its config (that's why there's a paramRef in the GatewayClass spec). That is, it's a controller, plus an associated set of config. In the style of implementation we're doing where the controller runs in-cluster, having a GatewayClass == a control plane makes both designing the config for the control plane, and the control plane itself, more straightforward.

I guess I'd summarize my thoughts here by saying that:

• I prefer option 2
• It's basically the standard Operator pattern, that provisions CPs and DPs as required to satisfy Gateways, dynamically.
• This is basically what we had in mind when we built the upstream API, so it fits well with the Gateway API constructs.
• Operational complexity is present but expected in the Operator pattern. People don't expect to be managing the CP or DP directly in this pattern, they expect the Operator to handle it for you, including upgrades. The good news is that we can codify the provisioner to be able to handle upgrades and other operations (like secret rotation), and give it our best engineering towards making the operations happen with minimal user intervention.
• In terms of having to build things, we should have to build very little, because we can lift a lot of what Contour is doing already.

I'd definitely like to hear from @LukeShu about option 1, since that's what Emissary does today, and I remember him having some concerns about it, but I can't remember if they were distinct from what I mentioned above.

[wilsonwu]

+1 on option 2, as same as Contour, it is OK for extension in future, because it is already compatible with option 1. And option 2+3 both fine and for more modes.

[arkodg]

I think that 1 is not a good idea. Although it seems like it's easier to manage, it adds big problems later once you're in prod, in that it's much harder to do upgrades without dropping traffic than if you can upgrade the control plane and data plane independently. This is why Contour moved from a single-Pod model like 1 to the split model it has today.

@youngnick could you please elaborate on this. AFAIK with 1, you would upgrade the DP as often as you would upgrade the CP. In 2 or 3, you could upgrade the CP more often than the DP (since DP upgrade would be opt in). In both cases the design for hitless upgrades based on L3 LB health checks, draining procedure and outlier detection for envoy upstream, would be the same.

[danehans]

xref #103 regarding a decision to split EG k8s and non-k8s.