Return 503 on overload

[josephburnett]

Expected Behavior

When Elafros becomes overloaded with too many requests, it should immediately return a 503 Service Unavailable response.

Actual Behavior

There is no limit on the request allowed into the user code.

Steps to Reproduce the Problem

1. Send a lot of requests at once.
2. Notice latency blows up. Requests start timing out.

Additional Info

The first place to implement this is in the queue-proxy on the revision pods. We must

1. enforce a maximum number of requests to be allow into the user code
2. create a queue for any requests beyond that amount
3. limit the queue to a maximum length and return 503's for any requests received beyond that limit

[josephburnett]

We will also want this behavior to be enforced on the activator. Maybe with different, more lenient max queue length parameters.

[mdemirhan]

This should be implemented as a policy using Istio OutlierDetection options rather than queue proxy (if possible). See https://istio.io/docs/reference/config/istio.networking.v1alpha3.html#OutlierDetection

[josephburnett]

We need the signal about incoming request rate to feed to the autoscaler. Until we drive all autoscaling from the Envoy proxy, I think we need to let these requests through to the queue-proxy.

But this is a great pointer for that piece of the puzzle. Thanks!

[mdemirhan]

My only request is to utilize Istio wherever possible and not duplicate work in queue proxy. Not only it has performance implications, but also it will make it harder to retire queue proxy. But like you said, we will need to understand how to drive autoscaler from Istio/Envoy first.

[akyyy]

These could be helpful. https://docs.giantswarm.io/guides/advanced-ingress-configuration/
ingress.kubernetes.io/limit-connections: number of concurrent connections allowed from a single IP address.
ingress.kubernetes.io/limit-rps: number of connections that may be accepted from a given IP each second

[jchesterpivotal]

Agreed on using upstream capabilities wherever possible.

Two questions:

• The current description is "When Elafros becomes overloaded with too many requests". Does that mean Elafros-the-runtime, or system behaviour in support of apps being managed by Elafros? Eg. does this story talk about how Elafros reacts when it gets application backpressure, or when its own components are overwhelmed, or both?
• Does this exclude non-HTTPS protocols? The suggestions above (Istio, ingress controller) are currently scoped to HTTPS only.

edit: for the benefit of those reading this later, "Elafros" was a pre-release name for Knative.

[grantr]

The suggestions above (Istio, ingress controller) are currently scoped to HTTPS only.

Technically Istio ingress also supports HTTP and HTTP/2 (including gRPC).

[jchesterpivotal]

Touché

[evankanderson]

/kind spec