Switch Activator to DaemonSet

[vagababov]

In what area(s)?

/area autoscale

Describe the feature

Currently we un Activator as a deployment with an HPA autoscaler.
For Node Local Scheduling, we need to run Activator as a DaemonSet.
The work here is consisted of several steps:

1. Find the appropriate instance requirements.
   • Currently we're requesting 300m of CPU and 400MB of memory for each instance. This presumes running just a few instances.
   • Our integration tests explicitly ask for 2 instances.
   • For benchmarks we are asking for 10-15 depending on the test, though they are strongly over-provisioned there). On my cluster I have reliably achieved desired results with 4-5 instances per 1000RPS and 150 target pods with CC=1 on 15 node cluster.
   • If we're going to switch to a DaemonSet, we probably need to lower these number somewhat.
   • We need to measure against the benchmarks to make sure the current performance levels do no degrade.
2. Provide optional template that permits running Activator as DS in v0.13.
3. Switch DaemonSet as default and set HPA min limits to 0 and deployment replicas to 0 in v0.14.
4. [possibly] Remove the existing configuration in v0.15

/assign @vagababov
/cc mattmoor @markusthoemmes

[mattmoor]

We want the DaemonSet to exist for sure for an overlapping release with the Deployment before we scale it down, otherwise we are racing the creation of new pods against collecting old pods.

When we create the DS the deployment should naturally scale down because so much of its load should be offloaded.

I think the only other nit is that step 2 should set replicas to zero on the deployment, as I doubt that doing so on the HPA scales it to zero.

[markusthoemmes]

Is the decision to make the activator our scheduling unit final? I still think that the activator might need to scale differently than „per node“ for its dataplane duties 🤔

[mattmoor]

I still think that the activator might need to scale differently than „per node“ for its dataplane duties.

Can you elaborate?

[markusthoemmes]

Sure,
With OpenWhisk we had a similar model of having the scheduling components distributed per node and each scheduled local containers. That was a mistake but we‘re not 100% in the same situation here. However, for OpenWhisk it wouldve been very beneficial to separate the scheduling components (which can be per node for local state) from the data plane components which can in theory handle multiple nodes, or if the load is super high, share the load of one node between them.

With the activator handling more and more traffic in more and more situations I wonder if we want to tie its scale to the number of nodes. That for example prohibits us moving them to some very large machines and run each on a massive CPU. That would currently be beneficial for our perfect loadbalancer for example. The less replicas the better.

I‘m not saying a DaemonSet is a nono. I‘d just like us to think about the notion of tying this data plane component‘s scale to the number of nodes and its consequences.

[vagababov]

I am actually fine (and I considered that as an option in the document (in progress)) for keeping both per-node and HPA scaled Activators. But significantly reducing the resources allocated to the free roaming activators.

[vagababov]

@mattmoor : updated step (3) and added numbering for more precise attribution.

[mattmoor]

Couple things:

• Folks can tweak what we release, and adding a NodeSelector or using a Deployment with anti-affinity for their own pods both seem like viable ways to not need a whole DaemonSet. Unless we're talking about a scenario where # activators > # nodes, which would be a use case I'd like to understand better.

• To be pedantic, we're making the scheduling choice ahead-of-time and programming the options into the load-balancer's Endpoints, we aren't empowering the activator to spin up pods anywhere but their own node. Perhaps you are referring to spinning up the pod as "scheduling", in which case I'm just splitting hairs, but I want to make sure we're talking about the same thing.

[JRBANCEL]

Are we going to use .spec.template.spec.nodeSelector?
Running Activator on every node is crazy. It is going to prevent anyone with a cluster with many nodes from using Knative.
Do we really need Activator to be a DS for Local Scheduling?

[mattmoor]

Running Activator on every node is crazy. It is going to prevent anyone with a cluster with many nodes from using Knative.

Let's not deal in FUD here. What are the concrete concerns?

Do we really need Activator to be a DS for Local Scheduling?

Sure it's a blunt instrument, we could probably write our own autoscaler and scheduler for the activator to do something a bit more curated, but again: we shouldn't just do that because of FUD.

[markusthoemmes]

So the concern (based on OpenWhisk experience) from my end is:

Conflating data-plane with control-plane. I think it's a valuable use-case to deploy the activators on a few beefy machines to a) make them quick and b) guard them from noisy neighbor effects on the worker nodes themselves, where just about any other workload can run. It shields the data-plane from an application going haywire and thus causing havok on other app's latencies.

Arguably, a lot of the OpenWhisk concerns don't apply here because

1. The activator's overhead is quite small (OpenWhisk's Scala based stack is not)
2. We don't necessarily constraint routing in the activator to just the local containers, so load can still even itself out across all of the activators.

[markusthoemmes]

Folks can tweak what we release, and adding a NodeSelector or using a Deployment with anti-affinity for their own pods both seem like viable ways to not need a whole DaemonSet.

That's right but if we build the scheduling capabilities (spawning of containers locally) into the activator, we're binding these mechanisms into the deployment of the activator. If I want to limit where they get deployed I now also limit the nodes where I can make use of NodeLocalScheduling for resiliency and potentially performance.

I guess the underlying question is: Do we want that behavior? Do we want to constraint the nodes we can schedule to by the nodes that have activators deployed? And if we want to do that: Why?