Support Sidecar Processes

[josephlewis42]

In what area(s)?

/area API

/area autoscale
/area build
/area monitoring
/area networking
/area test-and-release

/kind proposal

Describe the feature

It would be useful to allow developers to run additional sidecar processes next to their application code. Examples:

• Abstraction layers like dapr or sidecars based on Go CDK
• Cloud SQL proxy create cloudsql sidecar container #4659
• BeyondCorp proxies to other services
• Injected logging/monitoring systems

[knative-prow-robot]

@josephlewis42: The label(s) kind/proposal cannot be applied. These labels are supported: ``

Details

In response to this:

In what area(s)?

/area API

/area autoscale
/area build
/area monitoring
/area networking
/area test-and-release

/kind proposal

Describe the feature

It would be useful to allow developers to run additional sidecar processes next to their application code. Examples:

• Abstraction layers like dapr or sidecars based on Go CDK
• Cloud SQL proxy create cloudsql sidecar container #4659
• BeyondCorp proxies to other services
• Injected logging/monitoring systems

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[poy]

For another piece of data, Cloud Foundry supports sidecars: http://v3-apidocs.cloudfoundry.org/version/release-candidate/#sidecars

[savitaashture]

I had the very same thought and started a proposal here.
It's still in progress but it captures our thoughts so far. I'll throw it on the agenda for the next API meeting

[dgerd]

Adding two earlier issues along the same lines as well as a document on the previous decision here to only support a single container with the goal of providing context.

Thanks for putting together a proposal and I think it would be great to discuss it further in a future API WG meeting.

#1794
#3384
https://docs.google.com/document/d/1p42n_WwQc1Z3FsXr5ih4NnMY3PRzY15Ro2EU-mk_EuA/edit?ts=5b9aa556

[josephlewis42]

@dgerd I think that doc is private, would you be able to make it world readable?

Thanks!

[dgerd]

@josephlewis42 Looks like it was shared with the knative-dev group. I have added read access for knative-users group as well.

You can find directions to join those groups here: https://knative.dev/community/

[ahmetb]

Is this possibly a change of decision made in #3384 (comment) as RevisionSpecTemplate now has a containers[] field?

[dgerd]

That comment was made after RevisionSpecTemplate had containers[].

The doc that @savitaashture put together explores exactly what changing that decision may look like.

[evankanderson]

I'd like to also better understand how the two containers relate, and why two containers are better than two processes in one container.

1. What happens if one container dies and not the other?
2. One container will be scaled and driven by requests. If we turned off the CPU between requests, would that cause a problem?
3. Do you need any sort of shared disk/etc between the two processes?
4. How would you expect monitoring, etc to work for the secondary container?

With respect to workarounds, depending on what your sidecar container is for, it may make sense to use mutating admission controllers to inject the container into all pods that match a certain pattern (ala Istio sidecar injection), but that only works for certain scenarios.

[dgerd]

why two containers are better than two processes in one container.

• Application Dependencies - Applications can be written in multiple languages or the same language at multiple versions without concern of conflict (i.e. Python 2.7 and Python 3)
• Separate Build Processes - Easier to reason about single repository -> single build -> single container.
• Security Boundaries - Blast radius is smaller for compromise of a process separated by a container boundary
• Resource Starvation - Guaranteed resources for the processes running in separate containers

I will leave the particular use-case answers to others, but without a use-case in mind my stab at your questions.

1. Today we have a readinessProbe that determines if the Pod is healthy and I would imagine this would stay the same. So really this question boils down to: if the sidecar cannot become ready/dies does the pod die and stop serving traffic, or can it continue. Atomic readiness seems like the easiest answer here, but not sure if it is the right answer.
2. The impact here seems the same regardless of if you have 2 processes or 2 containers. We don't do this today in Knative so we will have to consider the impact of this change regardless of if there is a second container. I could imagine that if turning off CPU between requests is important to build it becomes a cluster configurable setting to preserve backwards compatibility. In that case this problem can be controlled by the operator.
3. I am going to guess that a number of use-cases will need shared disk as it is one of the two ways to share state between containers.
4. I don't have anything to add here.

The workaround seems feasible, but still subject to the same questions you posed.

[josephlewis42]

I'll also take a stab at this:

I'd like to also better understand how the two containers relate, and why two containers are better than two processes in one container.

Along with the points @dgerd mentioned:

• Licensing constraints -- your database might be under a viral license but your main server process isn't or vice-versa.
• Separation of duties -- the database team, storage architects, platform operators, and security teams can all provide their own sidecars to unify access to those systems.
• Lifecycle management -- sidecars need distinct lifecycle management from the apps if they're managed by 1p and 3p teams. Breaking apart containers means they can be independently tested/verified for industries that require audit compliance.
• Two processes in one container means coupling the lifecycle via some other means. It's better to have visility into that at the platform level rather than have everyone invent their own mechanisms.

What happens if one container dies and not the other?

The underlying serving pod should be considered unhealthy--ideally the serving container should also become unhealthy because it should be performing health checks on its dependencies.

One container will be scaled and driven by requests. If we turned off the CPU between requests, would that cause a problem?

I was under the impression that serving scaled requests by increasing/decreasing entire pods based on request metrics, under which both processes should be scaled up/down together so each serving container has a copy of its sidecars.

Do you need any sort of shared disk/etc between the two processes?

I think this is up to the spec. Most of the sidecars I listed only need network accessibility. e.g. the serving process hits the sidecar which uses some logic to manipulate/forward the request.

How would you expect monitoring, etc to work for the secondary container?

Health checks should be done via the main service (e.g. for a MySQL forwarding sidecar, the main process should tie its health to the health of the service) or via k8s container health checks.

With respect to workarounds, depending on what your sidecar container is for, it may make sense to use mutating admission controllers to inject the container into all pods that match a certain pattern (ala Istio sidecar injection), but that only works for certain scenarios.

I don't believe this makes sense in the context of Knative Serving because--to the best of my knowledge--serving doesn't guarantee it's going to use pods underneath. Instead it would be a workaround for a specific version of serving. If additional containers were brought into the spec, then it could be used on any Knative Serving compatible platform.