Decision: Shipping artifact structure

[youngnick]

As part of discussing #103, the community discussed how we should ship Envoy Gateway, in order to meet the "run on Kubernetes or not" part of the design. Actual decision records are bolded italicized.

We agreed that the code paths for these two options should be largely separate because they are so different.

We discussed whether that meant we should produce separate binaries or subcommands of a single binary, and we agreed that we will ship subcommands of a single binary for the different operating modes.

Then we discussed how we will actually ship this single binary with subcommands, and decided that we will ship both container and standalone binary for the single binary.

These decisions are not final until this issue has received maintainer votes in the form of comments or reacts, and community feedback one way or the other.

[arkodg]

+1 for shipping a single binary, a container image packaged with the single binary, and representing the entry points for kube and local as subcommands within that same binary

[danehans]

We discussed whether that meant we should produce separate binaries or subcommands of a single binary, and we agreed that we will ship subcommands of a single binary for the different operating modes.

+1 hold until #105 (comment) is answered.

Then we discussed how we will actually ship this single binary with subcommands, and decided that we will ship both container and standalone binary for the single binary.

+1 hold until #105 (comment) is answered.

[danehans]

xref #106 (comment) for capturing decisions in the design doc.

[alexgervais]

We agreed that the code paths for these two options should be largely separate because they are so different.

+1

We discussed whether that meant we should produce separate binaries or subcommands of a single binary, and we agreed that we will ship subcommands of a single binary for the different operating modes.

+1

Then we discussed how we will actually ship this single binary with subcommands, and decided that we will ship both container and standalone binary for the single binary.

+1

[danehans]

@youngnick, et al. can you help me better understand the benefits of using separate subcommands to support different operating environments instead of using a CLI flag/config field? This is how I envision running EG:

Run EG on Kube:

Using the providers flag:

$ kubectl apply -f ./my/eg/deployment.yaml
$ cat ./my/eg/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eg
  namespace: eg-system
spec:
...
      containers:
      - command: ["envoy-gateway"]
        args:
        - start
        - --providers=Kubernetes

Using the providers field of the config file:

$ kubectl apply -f ./my/eg/deployment.yaml
$ cat ./my/eg/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eg
  namespace: eg-system
spec:
...
      containers:
      - args:
        - start
        - --config-path=/my/eg/config.yaml
        command: ["envoy-gateway"]
        volumeMounts:
          - name: eg-config
            mountPath: /my/eg
            readOnly: true
...
      volumes:
        - name: eg-config
          configMap:
            name: eg
            items:
            - key: config.yaml
              path: config.yaml

$ cat ./my/eg/config.yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BootstrapConfig
spec:
  providers:
  - kubernetes: {}

The EG start subcommand will run the k8s controllers and any other Kube-specific functionality since providers=Kubernetes.

Run EG on Docker:

$ cat compose.yaml
version: '3'

services:
  eg:
    image: envoyproxy/gateway:v0.2.0
    command: start --providers=Docker
    ports:
      # EG metrics port
      - "8080:8080"
    volumes:
      # So that EG can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

Now run EG:

docker-compose up -d eg

When EG runs with providers=Docker it will configure the Provisioner to use the "Docker" provisioning backend. The Docker Provisioner can create the gatewayclass/gateway/httproute resources in memory using default settings (if not provided by the user) and use Docker Engine APIs to create the CP/DP containers, Docker networking, etc. from these sources.

This seems to align with #107. It appears that other API gateway projects are able to run in different environments without requiring separate subcommands so I question why it's needed for EG.

More on Providers

EG runtime configuration can be achieved through "Providers", aka ConfigSources from the design doc. The providers are infrastructure components that EG calls to find relevant information about establishing and maintaining its runtime configuration. For example:

• When providers=Kubernetes, EG will configure the Provisioner to use the "Kubernetes" backend. The Kubernetes Provisioner will CRUD the CP/DP Deployment, Service, ClusterRole, etc.
• When providers=Docker, EG will not start the Kubernetes controllers (and any other k8s-specific functionality). EG will configure the Provisioner to use the "Docker" backend. The Docker Provisioner will call Docker Engine APIs to CRUD the CP/DP containers, set up Docker networking, etc.
• When providers=File, EG will watch GatewayClass/Gateway/HTTPRoute resources in a provided directory path to create the CP/DP. EG will configure the Provisioner to use the "Internal" backend. To run the CP/DP processes, the Internal Provisioner will call internal APIs, e.g. CreateCP() and CreateDP() to run the CP/DP processes.
• EG will validate the list of providers to ensure only supported provider combinations are provided. For example, providers=Kubernetes,Docker will cause EG to not start while providers=File,HTTP will succeed and EG will merge the runtime configs from both providers.

[youngnick]

I think that what we will need to build is a little more nuanced than that, see #107 (comment) for a little more detail there.

To answer the question, I'm worried about code separation and complexity of testing. Subcommands enforce a strong separation all they way up to the main.go of the parent command, so it will be harder to have abstractions or code leak across.

Additionally, I'm worried about how complex the testing regime will be for all of the provider options you describe above. At the very least, we'll need to confirm that the provider/mode selection code works with a set of tests of main.go and other high-level things, which I've found tricky to do in the past.

All of these are not super strong arguments, just convenience. But I think that there's definitely something to talk about with respect to what constitutes a platform or provider.

[danehans]

But I think that there's definitely something to talk about with respect to what constitutes a platform or provider.

@youngnick thanks for the feedback. I've tried to better define provider in #95.

[danehans]

#111 fixes this issue. #95 provides additional supporting details of the config API.

We agreed that the code paths for these two options should be largely separate because they are so different.

We agreed that EG will have a single entrypoint. An operator can be considered in the future for managing EG. As #111 states, EG (control plane) will manage the Envoy proxy fleet (data plane).

We discussed whether that meant we should produce separate binaries or subcommands of a single binary, and we agreed that we will ship subcommands of a single binary for the different operating modes.

This N/A since EG will have a single command entrypoint, i.e. start.

Then we discussed how we will actually ship this single binary with subcommands, and decided that we will ship both container and standalone binary for the single binary.

EG release artifacts will consist of a Docker image and a bin.