[webhook] reduce memory usage of the webhook

[dprotaso]

Expected Behavior

Webhook memory usage should remain somewhat linear with the resources that are being watched

Actual Behavior

@sravankumar777 reported high memory usage here: knative/serving#7195

I think it's because our informers are watching all secrets here: https://github.com/knative/pkg/blob/master/webhook/certificates/controller.go#L59

Steps to Reproduce the Problem

1. See Memory usage for knative webhook gradually increases over the time until memory limit is reached serving#7195

Additional Info

We should ensure we're scoping other informers appropriately as well

[mattmoor]

There is an injection context function that can be used to scope the injected informers to a single namespace. I think we'll also want the capacity to do this per-type as there are contexts where I could see us doing this for some types (Secrets, ConfigMaps) and not for others (Bindings).

[mattmoor]

It would be good to get a heap profile of what's going on in these situations too rather than just guessing 👼

[bbrowning]

Here's a screenshot of a heap profile for a case I recently ran into. If I'm reading it right, about 83% of the in_use heap space is from reading secrets from all over the cluster?

[mattmoor]

@bbrowning can you confirm that this is fixed at HEAD?

[mattmoor]

cc @markusthoemmes

[dprotaso]

PR that has the fix: #1152 which is part of 0.14 serving release

[sravankumar777]

@dprotaso & @mattmoor, looks like the issue for webhook pod memory usage is not resolved with knative-serving v0.14.0 release.

Below are the memory usage dashboard for two different environment, on which knative-serving is deployed.

$ kubectl get pod webhook-c9cfb67b7-6q8nb -nknative-serving -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'
webhook	map[cpu:200m memory:200Mi]
istio-proxy	map[cpu:2 memory:512Mi]

$ kubectl get pod webhook-c9cfb67b7-6q8nb -nknative-serving -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'
webhook	map[cpu:20m memory:20Mi]
istio-proxy	map[cpu:50m memory:180Mi]

[dprotaso]

Thanks for the update @sravankumar777

Is there anything happening in your cluster over that duration? ie. I believe @bbrowning's saw a similar issue when creating many namespaces.

[vagababov]

I verified that number of namespaces does not trigger this.
#1158  — is definitely part of 0.14 branch.
Can you check that you indeed run 0.14 or newer?
Otherwise can you share what is done to the cluster? I.e. do you create lots of services or revisions or some other objects?

[sravankumar777]

@dprotaso & @vagababov, as mentioned earlier we do definitely around 300+ namespaces.
Apart from this, we do run e2e serving test continuouly from a testing namespace to monitor working of management components.

e2e serving test:

1. Create KSVC
2. Pod Autoscaling based on 30QPS on a KSVC.
3. Delete KSVC.

This KSVC is always created with a random name on every instance.
Plan to upgrade to v0.15.0 very soon.

Thank you.

[vagababov]

I've just created 200 namespaces and noticed 0 uptick in memory. Let me try to do the same with creating a service in each.