Autoscaling to Zero

[holoed]

Expected Behavior

Autoscaler to Set Replica Desired Pod count to zero after no HTTP traffic for a few days.

Actual Behavior

Autoscaler Replica Desired Pod count stays above zero for days even though there is no HTTP traffic

Steps to Reproduce the Problem

1. AKS Knative 1.1.0
2. A few knative services orchestrated through knative eventing
3. Every once in a while one or few services don't scale to zero for a few days (for example during a weekend of no traffic)
4. Deleting the Activator pod in Knative Serving temporarily resolves the issue and those services that were stuck, gradually scale back to zero after the Activator pod fully terminates (takes a a few minutes to terminate old stuck Activator).

Additional Info

[nader-ziada]

Can you please give more details about how you created your cluster and your configuration so we can try to reproduce the issue?

[holoed]

Cluster contains 9 services, connected with Parallel and Sequential flows, deployed on Azure AKS.
Services are written in Python, JavaScript and Scala.

It happened again today.
The log of the Activator pod says:
"Failed probing pods" for the service that is now stuck doing nothing and is not being terminated.
Also says "error roundtripping http://.../healthz: context deadline exceeded"
caller: "net/revision_backends.go:340"

[nader-ziada]

"Failed probing pods" for the service that is now stuck doing nothing and is not being terminated.
Also says "error roundtripping http://.../healthz: context deadline exceeded"
caller: "net/revision_backends.go:340"

does the service have a custom health check? looks from the error posted here that the probe of the pod fails with timeout , if so, use a transport that will close the connection after every request here. Otherwise the cached connections will prohibit scaling to zero, due to unsuccessful probes to the Activator.

[JulesMac]

When we see this issue the - Activator log when contains the following repeated log entries (multiple times a second)

{"severity":"ERROR","timestamp":"2022-01-20T13:56:56.175362314Z","logger":"activator","caller":"net/revision_backends.go:396","message":"Failed to probe clusterIP 172.17.208.170:80","commit":"7499ae2","knative.dev/controller":"activator","knative.dev/pod":"activator-6fb68fff7b-d8mhw","knative.dev/key":"master/XXXXXXXXXXX-00001","error":"unexpected status code: want [200], got 470","stacktrace":"knative.dev/serving/pkg/activator/net.(*revisionWatcher).checkDests\n\tknative.dev/serving/pkg/activator/net/revision_backends.go:396\nknative.dev/serving/pkg/activator/net.(*revisionWatcher).run\n\tknative.dev/serving/pkg/activator/net/revision_backends.go:441"}

We think the service XXXXXXXXXX scaled to 0 then we send in some requests - it seems to be trying to scale up but some how failing to manage this - after some time (could be in the region of 10 mins) new pods are created but none of them get any traffic - the replicate set for the pod gradually gets bigger (we normally don't see this scale up beyond one pod in this environment)

kubectl get replicasets.apps
NAME DESIRED CURRENT READY AGE
XXXXXXX-00001-deployment-5857ff6b75 36 36 36 19h

[julz]

Based on the above error the activator is trying to probe the cluster IP for the service on the /healthz endpoint and receiving a 470 status code. That is.. surprising to say the least since I don't think anything in knative should ever send that status code (it's not even a real HTTP status code as far as I know 🤔). Do you have anything interesting network-wise in the setup, e.g. some sort of service mesh configured? If so could you share details? What networking plugin are you using?

[julz]

It would be interesting, when you next see the log message, to try manually curling 172.17.208.170:80 (or whatever the message logs) from within the cluster. Does it return a 470 status code? Does it return a body that gives any extra clues?

[evankanderson]

Is there a service mesh of some sort (or other transparent HTTP proxy) in the configuration?

[JulesMac]

We are using an AKS cluster in azure following these instructions - using the kourier networking layer (there are a few minor config mods to integrate with azure ingress etc) but as far as we can see this all seems to be 'OK'. Not sure is there are any AKS/Azure specific installation setup recommendation (we've not found that seem to be up to date).

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[unzld]

/reopen
@holoed did you manage to solve this issue?