bug(proxy): inference.local buffers entire streaming response, inflating TTFB

[johntmyers]

Summary

The sandbox proxy's inference.local interception path fully buffers upstream streaming responses (SSE) before sending any bytes back to the client. This converts a streaming response (fast TTFB, incremental tokens) into a buffered response (slow TTFB, instant completion), causing downstream clients with TTFB timeouts to abort before the proxy finishes buffering.

Actual Behavior

1. Client sends POST /v1/chat/completions with "stream": true to inference.local
2. Proxy forwards to upstream (e.g. NVIDIA NIM endpoint), which begins streaming SSE events (TTFB ~200ms)
3. response.bytes().await in proxy_to_backend() blocks for the full generation time (10-60s)
4. Proxy sends nothing back to the client during this entire period
5. Client's TTFB/first-event timeout fires and aborts the request
6. Proxy eventually finishes buffering and tries to write to a closed connection

Expected Behavior

The proxy should forward response headers and SSE chunks to the client incrementally as they arrive from the upstream, preserving the streaming semantics and sub-second TTFB.

Root Cause

Three layers conspire to prevent streaming:

1. navigator-router/src/backend.rs:112-115 — response.bytes().await collects the entire upstream response body into memory before returning
2. ProxyResponse struct (backend.rs:9-13) — stores body as a single bytes::Bytes blob with no streaming abstraction
3. format_http_response() (l7/inference.rs:244-246) — replaces upstream Transfer-Encoding: chunked with Content-Length based on buffered body size, then writes everything in a single write_all call

Affected Path

Only the HTTPS/CONNECT inference interception path (inference.local) is affected. The plain HTTP forward proxy path uses copy_bidirectional and streams correctly.

Call chain: handle_inference_interception() → route_inference_request() → proxy_with_candidates() → proxy_to_backend() → response.bytes().await

Fix Direction

• Add a streaming variant of proxy_to_backend() that returns headers + impl Stream<Item = Bytes> instead of a fully-buffered ProxyResponse
• Have route_inference_request() write response headers immediately, then forward body chunks incrementally to the TLS client
• For streaming responses, emit Transfer-Encoding: chunked instead of Content-Length

[johntmyers]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High — clear path, all APIs exist

Summary

The inference proxy path buffers the entire upstream response via response.bytes().await before sending anything to the client. The fix adds a streaming variant of the proxy backend that returns response headers immediately and streams body chunks incrementally using HTTP chunked transfer encoding. Non-streaming (buffered) responses continue to work as before.

Scope

• crates/navigator-router/src/backend.rs: Add StreamingProxyResponse type and proxy_to_backend_streaming() that returns headers + reqwest::Response (for incremental .chunk() reads) instead of buffering the entire body
• crates/navigator-router/src/lib.rs: Add proxy_with_candidates_streaming() method on Router, export new types
• crates/navigator-sandbox/src/l7/inference.rs: Add format_http_response_header() (status line + headers for chunked TE), format_chunk(), and format_chunk_terminator() helpers
• crates/navigator-sandbox/src/proxy.rs: Update route_inference_request() to use the streaming path — write headers first, then stream body chunks via chunked TE

Implementation Steps

1. Add StreamingProxyResponse and proxy_to_backend_streaming() in backend.rs — extract shared request-building into a helper to avoid duplication with the existing buffered path
2. Add proxy_with_candidates_streaming() on Router in lib.rs, export new types
3. Add chunked TE formatting helpers in inference.rs (format_http_response_header, format_chunk, format_chunk_terminator)
4. Update route_inference_request() in proxy.rs to use streaming: write headers immediately, loop on response.chunk(), write each chunk in chunked TE framing, terminate
5. Add unit tests for new helpers and integration tests

Test Plan

• Unit tests: Chunked TE formatting helpers in inference.rs, build_backend_request extraction in backend.rs
• Integration tests: proxy_to_backend_streaming() against wiremock (already a dev-dependency)
• E2E: The examples/local-inference/inference.py script serves as the acceptance test (TTFB << total time when streaming)

Risks & Open Questions

• reqwest::Response is not Debug — StreamingProxyResponse cannot derive Debug (minor; use manual impl or skip)
• The existing proxy_to_backend() (buffered) stays for mock routes which are inherently non-streaming

Documentation Impact

• None expected — architecture docs don't cover internal proxy plumbing at this level

---

Revision 1 — initial plan

[johntmyers]

🏗️ build-from-issue-agent

Implementation Complete

PR: #261

What was built

Added a streaming proxy path for inference.local that writes response headers immediately and forwards body chunks incrementally using HTTP chunked transfer encoding, replacing the previous response.bytes().await buffering that inflated TTFB.

Tests

• Unit: 7 tests added (chunked TE formatting helpers)
• Integration: 10 existing tests pass (shared request-building helper)
• E2E: examples/local-inference/inference.py with TTFB instrumentation

Docs updated

• examples/local-inference/README.md — updated with NVIDIA provider quick start
• No architecture doc changes needed

The issue will auto-close when the PR is merged.