health check for AIServiceBackend

[kerthcet]

Description:

Describe the desired behavior, what scenario it enables and how it
would be used.

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: qwen2--5-coder
  namespace: default
spec:
  timeouts:
    request: 3m
  schema:
    name: OpenAI
  backendRef:
    name: qwen2--5-coder-lb
    kind: Service
    port: 8080

Here's one service backend we defined in the cluster and refer to the model: qwen2.5-coder, however, even the service doesn't exist or the service is not ready, when we query the model list, we can still see the model from v1/models/, results look like:

{
  "data": [
    {
      "id": "qwen2-0.5b",
      "created": 1745481246,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "qwen2.5-coder",
      "created": 1745481246,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    }
  ],
  "object": "list"
}

Then in our chatbot, we can still query the model while it's always failed. My question is can we remove the model from the model list unless the service is ready?

Thanks.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[mathetake]

I think this is a reasonable request but it would be hard to implement in reality. wdyt? @nacx

[nacx]

I agree it would be great to have this, but it is going to be complex to implement.

For use cases using cloud models, availability is usually not going to be an issue, unless a model is, for example, retired. In that case, it would look more like a signal to update a no-longer-valid configuration than an error that should be gracefully handled.

In my opinion, the use case for health/availability is more relevant when using on-prem models, but this is a feature that should be well-defined: how do we want to define the health monitors and the thresholds for those? Is this something clients should be able to handle themselves by leveraging the metrics produced by the ai-extproc? Health checking is a complex problem, and whitelisting/evicting models can be tricky.

I think the best way to move this issue forward is to start a proposal/design doc to define the different problem statements and proposed solutions so that we can all have a better understanding of the ramifications and complexity/scope. This would allow us to have better-informed discussions and actionable things to be done. @kerthcet would you be willing to take this?

[yuzisun]

@nacx Cloud can have outages and model endpoints sometimes have degraded performance which makes health monitoring essential for production SLA applications. I think health monitoring should be a built-in feature in envoy ai gateway to detect failures proactively and fallback to other regions or other providers depending on the configuration. In terms of implementation, we can't rely on the ai-exproc metrics as the request varies a lot for LLM workload, we can send a small request every few seconds to the supported chat completion endpoints by turning on streaming to capture the TTFT/TPS metrics, these metrics can be used to surface the health status and make the failover decisions.

[kerthcet]

For use cases using cloud models, availability is usually not going to be an issue, unless a model is, for example, retired

Not yet, service health detection is always essential, you just can't guarantee the service is always reliable, also as @yuzisun mentioned above.

but this is a feature that should be well-defined

Agree.

I think the best way to move this issue forward is to start a proposal/design doc to define the different problem statements and proposed solutions

What we're proposing right now is we'll have a controller to orchestrate all the stuff, generally looks like:

• once inference service is created, we'll create a corresponding AIServiceBackend
• once inference service has ready endpoints, we'll inject the router rule to the AIGatewayRoute
• once inference service has no-ready endpoints or deleted, we'll remove the rule from the AIGatewayRoute

Decoupled from the ai gateway orchestration, one potential problem is the configurations management, like the weights. Would like to hear advices from you guys. Also cc @pacoxu who leads the poc implementation.

[mathetake]

@kerthcet yeah so i guess in general having a foundational code for passive/active health checking for each AIServiceBackend in our controller code could be useful and we can iterate on that to utilize it for not only the drop/update of a failing backend rule from AIGatewayRoute but also for fallback, etc later. This could be a core part of the project and I would like to see that integrated inside our code rather than forcing users like you to implement your own controller.

I think for the configuration could be a part of AIServiceBackend that might look like

diff --git a/api/v1alpha1/api.go b/api/v1alpha1/api.go
index d79b268..e4d33f5 100644
--- a/api/v1alpha1/api.go
+++ b/api/v1alpha1/api.go
@@ -374,10 +374,52 @@ type AIServiceBackendSpec struct {
        // +optional
        Timeouts *gwapiv1.HTTPRouteTimeouts `json:"timeouts,omitempty"`
 
+       // ActiveHealthCheck defines the active health check that can be configured for this backend.
+       //
+       // The controller will perform the health checks and update the status of the AIServiceBackend.
+       // The failing backend will be dropped from the routing table
+       ActiveHealthCheck *ActiveHealthCheck `json:"activeHealthCheck,omitempty"`
+
        // TODO: maybe add backend-level LLMRequestCost configuration that overrides the AIGatewayRoute-level LLMRequestCost.
        //      That may be useful for the backend that has a different cost calculation logic.
 }
 
+type (
+       // ActiveHealthCheckType specifies the type of active health check. Currently, only "Default" is supported.
+       ActiveHealthCheckType string
+       // ActiveHealthCheck defines the active health check configuration.
+       ActiveHealthCheck struct {
+               // Type specifies the type of active health check. Currently, only "Default" is supported.
+               Type ActiveHealthCheckType `json:"type"`
+               // Default is the default active health check configuration.
+               Default *ActiveHealthCheckDefault `json:"default,omitempty"`
+       }
+       ActiveHealthCheckDefault struct {
+               Request struct {
+                       Method  string `json:"method"`
+                       Path    string `json:"path"`
+                       Body    string `json:"body"`
+                       Timeout int    `json:"timeout"`
+               }
+               // Thresholds defines the thresholds for the active health check.
+               //
+               // This will be similar to outlier detection (Envoy's active health check) API, but will come with
+               // LLM specific concepts like "tps" and "consecutive5xx" etc., as well as this works at the backend level.
+               // https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/outlier_detection.proto#envoy-v3-api-msg-config-cluster-v3-outlierdetection
+               Thresholds struct {
+                       TPS            int `json:"tps"`
+                       Consecutive5xx int `json:"consecutive5xx"`
+                       // ....
+               }
+       }
+)
+
+const (
+       // ActiveHealthCheckTypeDefault is the default active health check type, which is making an HTTP request to the backend.
+       ActiveHealthCheckTypeDefault ActiveHealthCheckType = "Default"
+       ActiveHealthCheckTypeCustom  ActiveHealthCheckType = "Custom"
+)
+

wdyt? cc @yuzisun @wengyao04 @arkodg

[kerthcet]

This could be a core part of the project and I would like to see that integrated inside our code rather than forcing users like you to implement your own controller.

Thanks @mathetake , that would be great to support it inside the ai gateway.

Regarding to the API design, why should we distinguish the Default and Custom, from API perspective, they doesn't make any difference.

Also I just thought of one potential problem here, as we know, the Pod will configure the health probe, and we define another probe here, which means (1x ~ 2x) more queries, more instances, less serious, is that acceptable? Although the health probe would be very lightweight, like ping-pong or just generating one token.

[yuzisun]

For use cases using cloud models, availability is usually not going to be an issue, unless a model is, for example, retired

Not yet, service health detection is always essential, you just can't guarantee the service is always reliable, also as @yuzisun mentioned above.

but this is a feature that should be well-defined

Agree.

I think the best way to move this issue forward is to start a proposal/design doc to define the different problem statements and proposed solutions

What we're proposing right now is we'll have a controller to orchestrate all the stuff, generally looks like:

• once inference service is created, we'll create a corresponding AIServiceBackend
• once inference service has ready endpoints, we'll inject the router rule to the AIGatewayRoute
• once inference service has no-ready endpoints or deleted, we'll remove the rule from the AIGatewayRoute

Decoupled from the ai gateway orchestration, one potential problem is the configurations management, like the weights. Would like to hear advices from you guys. Also cc @pacoxu who leads the poc implementation.

@kerthcet We are already working on this implementation from kserve side, see kserve/kserve#4319.
cc @sivanantha321

[mathetake]

Regarding to the API design, why should we distinguish the Default and Custom, from API perspective, they doesn't make any difference.

yeah sorry nvm, i was under the impression that we want custom "callout" related stuff but i don't think we need at the first iteration. you can ignore Type ActiveHealthCheckType json:"type"`` etc.

Also I just thought of one potential problem here, as we know, the Pod will configure the health probe, and we define another probe here, which means (1x ~ 2x) more queries, more instances, less serious, is that acceptable? Although the health probe would be very lightweight, like ping-pong or just generating one token.

I was thinking to perform health probe from the controller instead of extproc, and the controller will update the extproc configuration according to the active health check result. does that make sense?

[kerthcet]

@kerthcet We are already working on this implementation from kserve side, see kserve/kserve#4319.

Great to see on-going practice here, I'll keep following your implementation on kserve side.

I was thinking to perform health probe from the controller instead of extproc, and the controller will update the extproc configuration according to the active health check result. does that make sense?

Yeah, this is a feasible solution, I'm just worry about the performance if we have lots of service to sync with. And will non-blocking health probe lag the controller orchestration. Hope I understood your meanings correctly.

[mathetake]

yeah you understand my comment correctly. I share your concern but I generally prefer these health probes to be performed in the controller (one pod when leader election enabled) vs the extproc (can be many pods). Even if the number of services is large, both still need to talk to every service to perform health check so i guess the situation might be almost the same. I assume the health check failure is rare (otherwise i am like why it's used in production yeah), so I think the reconciliation won't happen very often in any case i guess. not strong opinion though.

thoughts? @yuzisun

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.