Stop scaling up if pods are crashlooping

[josephburnett]

Expected Behavior

If pods are crashlooping, they will never successfully process requests and scaling up will not help. Autoscaling should cease scale up if pods are crashlooping.

Actual Behavior

No such circuit breaker exists.

Additional Info

Re: #2117 (comment)

[hohaichi]

@josephburnett Do you have a scenario/repro? Is this a crashlooping sceanrio: a pod crash when it receives X concurrent requests, and keep coming back, receiving X concurrent requests, crashing again? If so, I think it should continue to scale up so that each pod will receive less than X concurrent requests, right?

If the scenario is a pod crashing while booting up, then I don't see why it could be scaled up. It is a bug not scaling down in this case (#3456), and I'm fixing it.

[vagababov]

/milestone Ice Box

[markusthoemmes]

Is this effectively caught via our deployment check and our activation timeout?

[jonjohnsonjr]

The activation timeout covers the [0, minScale] range, but for (minScale, infinity) I don't believe we will stop trying to scale. I can't of a situation where some (but not all) pods are crashlooping AND the right thing to do is to stop scaling. Detecting that only some pods are crashlooping would also be kind of expensive.

I believe we will surface a ProgressDeadlineExceeded status in this case, which might be the only reasonable thing to do...

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale