Memory usage for knative webhook gradually increases over the time until memory limit is reached

[sravankumar777]

What version of Knative?

v0.12.1

Actual Behavior

We deployed knative-serving v0.12.1 over Kubernetes Cluster with Istio as Service Mesh.
Over the time with this version of knative-serving, we observed memory usage for webhook pod gradually increases and caused pod restart automatically.

Steps to Reproduce the Problem

1. Deploy knative-serving with v0.12.1 with Istio as service mesh.
2. Monitor memory usage for webhook pod in knative-serving namespace.

Logs:

$ kubectl get pod webhook-567d974c6b-wjfnn -nknative-serving -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'
webhook	map[cpu:200m memory:200Mi]
istio-proxy	map[cpu:2 memory:512Mi]

$ kubectl get pod webhook-567d974c6b-wjfnn -nknative-serving -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'
webhook	map[cpu:20m memory:20Mi]
istio-proxy	map[cpu:50m memory:180Mi]

As from grafana, memory usage for webhook start at around 162 MiB & over 3-4 hours duration this increased to 167MiB.
This grafana dashboard is taken for webhook pod, after the auto restart of pod.

[knative-prow-robot]

@sravankumar777: The label(s) area/webhook cannot be applied, because the repository doesn't have them

Details

In response to this:

What version of Knative?

v0.12.1

Actual Behavior

We deployed knative-serving v0.12.1 over Kubernetes Cluster with Istio as Service Mesh.
Over the time with this version of knative-serving, we observed memory usage for webhook pod gradually increases and caused pod restart automatically.

Steps to Reproduce the Problem

1. Deploy knative-serving with v0.12.1 with Istio as service mesh.
2. Monitor memory usage for webhook pod in knative-serving namespace.

Logs:

$ kubectl get pod webhook-567d974c6b-wjfnn -nknative-serving -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'
webhook	map[cpu:200m memory:200Mi]
istio-proxy	map[cpu:2 memory:512Mi]

$ kubectl get pod webhook-567d974c6b-wjfnn -nknative-serving -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'
webhook	map[cpu:20m memory:20Mi]
istio-proxy	map[cpu:50m memory:180Mi]

As from grafana, memory usage for webhook start at around 162 MiB & over 3-4 hours duration this increased to 167MiB.
This grafana dashboard is taken for webhook pod, after the auto restart of pod.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dprotaso]

Hey @sravankumar777 was there an earlier version of Knative where you didn't see usage increase like this?

[dprotaso]

For context our webhooks actually watch K8s resources such as secrets, webhook configurations and with 0.13 CRD definitions. We do this so the webhook can bootstrap it's own certs and keep the certificate bundle in the various resources up to date.

Thus it may be expected that the memory increases slightly but I believe we're not limiting the scope of certain informers - ie. secrets to just the namespace of our webhook.

I've created an issue where the webhook framework resides: knative/pkg#1152

[sravankumar777]

@dprotaso, we were not monitoring the memory usage with earlier versions.
From release v0.12.1, we started to monitor the memory usage for all pods in knative-serving.

As per my understanding & the limits configured for memory on webhook container, it is 200MiB and immediately after the webhook pod restarted it started from approximately 160MiB.
There may be chances for slight increase over the time and might cause for pod restart.

[bbrowning]

I've seen something similar that directly correlates to the number of namespaces in a cluster, but that may only be because more namespaces means more secrets? From my testing, 80 namespaces on my cluster is 85MB of webhook pod memory usage on startup. 170 namespaces on my cluster is 180MB of webhook pod memory usage on startup.

[bbrowning]

You can replicate the memory increase with additional namespaces via:

for i in {1..90}; do kubectl create ns testns$i; done

Then delete the webhook pod, it will get recreated, and observe the higher memory usage of the pod compared to before creating those blank namespaces.

[bbrowning]

Here's a snapshot of the memory usage of the webhook pod at steady-state followed by creation of 30 empty namespaces and a restart followed by creation of an additional 60 (for 90 total) empty namespaces and a restart of that pod again. You'll see that the startup memory usage grows substantially, although the steady-state memory usage doesn't differ much in the 3 cases.

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

[sravankumar777]

/remove-lifecycle stale

[matzew]

I noticed a recent bump on the settings in this PR:

https://github.com/knative/serving/pull/8048/files