TokenRateLimitPolicy counts 404 responses against quota, blocking legitimate requests

[noyitz]

Summary

TokenRateLimitPolicy applies rate limiting before Gateway API route matching occurs, causing 404 responses to consume user quotas. This results in legitimate requests being blocked when users make mistakes in URLs or follow incorrect documentation.

Environment

• Kuadrant Version: (deployed via Helm charts)
• OpenShift Version: 4.19.9+
• Gateway API Version: OpenShift native implementation
• Cluster: OpenShift on AWS
• Gateway Controller: openshift.io/gateway-controller/v1

Steps to Reproduce

1. Setup: Deploy Kuadrant with TokenRateLimitPolicy on Gateway (rate limit: 5 requests)
2. Get authentication token from MaaS API
3. Make 5 requests to wrong URL (missing endpoint path):

   for i in {1..5}; do
     curl -H "Authorization: Bearer $TOKEN" \
          -H "Content-Type: application/json" \
          -d '{"model": "test", "prompt": "Hello"}' \
          "http://maas.example.com/llm/model-name" # Missing /v1/chat/completions
   done

4. Make 6th request to wrong URL:
   curl -H "Authorization: Bearer $TOKEN"
   "http://maas.example.com/llm/model-name"
5. Test correct URL:
   curl -H "Authorization: Bearer $TOKEN"
   "http://maas.example.com/llm/model-name/v1/chat/completions"

Expected Behavior

• Steps 1-3: Return 404 Not Found without consuming rate limit quota
• Step 4: Return 404 Not Found without consuming rate limit quota
• Step 5: Return 200 OK with successful response

Actual Behavior

• Steps 1-3: Return 404 Not Found and consume quota (5/5 requests used)
• Step 4: Return 429 Rate Limit Exceeded (quota exhausted)
• Step 5: Return 429 Rate Limit Exceeded (legitimate request blocked)

Root Cause

TokenRateLimitPolicy is applied at the Gateway level before route matching:

Request → TokenRateLimitPolicy (quota consumed) → Gateway Routing → 404

Should be:

Request → Gateway Routing → Route Match → TokenRateLimitPolicy → Backend
Request → Gateway Routing → No Match → 404 (no quota consumption)

Impact

• Poor User Experience: Typos in URLs exhaust user quotas
• Legitimate Requests Blocked: Correct requests fail after URL mistakes
• Billing/Quota Issues: Users charged for failed requests

Configuration

Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: maas-default-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: https
port: 443
protocol: HTTPS
hostname: maas.apps.cluster.example.com

TokenRateLimitPolicy

apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
name: gateway-token-rate-limits
namespace: openshift-ingress
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: maas-default-gateway
limits:
per-user:
rates:
- limit: 5
duration: 10s

Reproduction Environment

This was discovered and reproduced in the https://github.com/opendatahub-io/maas-billing on customer environment:

• Gateway: maas.apps.cluster-tvp85.tvp85.sandbox1981.opentlc.com
• Model endpoint: /llm/facebook-opt-125m-simulated/v1/chat/completions

Logs

Example sequence showing the issue:
Request 1 to /llm/model-name: 404 (quota: 1/5)
Request 2 to /llm/model-name: 404 (quota: 2/5)
Request 3 to /llm/model-name: 404 (quota: 3/5)
Request 4 to /llm/model-name: 404 (quota: 4/5)
Request 5 to /llm/model-name: 404 (quota: 5/5)
Request 6 to /llm/model-name: 429 Rate Limit Exceeded
Request 7 to /llm/model-name/v1/chat/completions: 429 Rate Limit Exceeded

Suggested Fix

Rate limiting policies should only be evaluated after successful route matching, or provide configuration to exclude certain HTTP status codes (like 404) from quota consumption.

Possible approaches:

1. Apply TokenRateLimitPolicy at HTTPRoute level instead of Gateway level
2. Add configuration to exclude specific status codes from rate limiting
3. Change policy evaluation order to happen after routing

Additional Context

• Related to Gateway API policy attachment and request processing order
• Affects any application using Kuadrant TokenRateLimitPolicy at Gateway level
• Issue discovered during MaaS Platform validation guide testing

[jasonmadigan]

a when predicate on the TRLP would probably help.

 when:
  - predicate: 'request.path == "/v1/chat/completions"'

Is there a RateLimitPolicy on the Gateway too? given the route 404s (and presumably doesn't return usage.total_tokens), token counts wouldn't be increasing

[eguzki]

Rate limiting, whether token-based or request-based, follows the routing rules defined in the HTTPRoute objects. I don’t have access to your HTTPRoute, so I can’t fully reproduce your issue. It would be very helpful if you could share your HTTPRoute so we can investigate further.

In the meantime, I tested with the following HTTPRoute (showing only the relevant parts):

spec:
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: "/v1/chat/completions"
      backendRefs:
        - name: trlp-tutorial-llm-sim
          port: 80

Then I tried a non-existent path, for example /other. As expected, the rate limiting module did not trigger, and I received a 404 response.

For your information, the Wasm plugin’s “routing configuration” is derived from the HTTPRoute embedded in its configuration. This means that rate limiting and authentication services are only activated for valid paths. I hope this helps.

Could you share the WasmPlugin object (the one holding the Wasm configuration)?

[noyitz]

those are the policies we were using:
https://github.com/opendatahub-io/maas-billing/blob/main/deployment/base/policies/
both rate limit and token limit are over there.