fix(api): record completions even when client disconnects (#58) * fix(api): record completions even when client disconnects (#21) This fix ensures that LLM completions are properly recorded to the database even when the client disconnects mid-stream or times out. Backend changes: - Add 'aborted' status to CompletionsStatusEnum - Create StreamingContext class to manage streaming state with abort detection - Register abort event listener on request.signal to save completion on disconnect - Update processStreamingResponse to use StreamingContext - Update processNonStreamingResponse to await database writes and detect aborts - Apply changes to all three endpoints: completions, messages, responses Frontend changes: - Add 'aborted' status handling in ts-pattern exhaustive matching - Add amber indicator badges for aborted requests in requests and embeddings tables - Add i18n translations for 'Aborted' status (en-US: "Aborted", zh-CN: "已中止") Database migration: - Add 'aborted' value to completions_status enum Co-Authored-By: Claude Opus 4.5 * fix(api): address PR review feedback for abort handling Addresses review comments from CodeRabbit and Gemini: 1. StreamingContext improvements: - Pass "Client disconnected" reason to saveCompletion for better logging - Handle already-aborted signals (AbortSignal spec: addEventListener won't trigger for already-aborted signals) - Fix TPM consumption to use Math.max(0, ...) for token values, ensuring partial usage is charged even when output is 0 2. Non-streaming abort handling: - Save completion as "aborted" when processing throws after client disconnect (previously left as "pending") 3. Streaming error logging: - Add ctx.isAborted() check before logger.error to reduce noise when client disconnects Co-Authored-By: Claude Opus 4.5 * fix(api): use type guard for error message in abort handling Fix no-base-to-string lint error by using instanceof Error check instead of String(error ?? "...") pattern. Co-Authored-By: Claude Opus 4.5 * fix(api): improve error handling robustness in abort catch blocks Address Gemini critical review: 1. Remove flawed `completion.status === "pending"` check that could prevent logging when status was already updated 2. Use .catch() instead of await for addCompletions to handle database failures gracefully without blocking error response Co-Authored-By: Claude Opus 4.5 * fix(api): add explicit Content-Type headers for streaming and non-streaming responses Addresses reviewer feedback: - Set Content-Type: text/event-stream for SSE streaming responses - Set Content-Type: application/json for non-streaming JSON responses When using async generators in Elysia, the content-type cannot be auto-detected, so we need to explicitly set the appropriate headers based on the request type. Co-Authored-By: Claude Opus 4.5 * refactor(api): use return for non-streaming responses in async generators Per Elysia documentation, async generators should: - Use `return` for non-streaming responses (Elysia converts to normal JSON response) - Use `yield` only for streaming responses This ensures: - Non-streaming responses get proper `application/json` content-type - Streaming responses get proper streaming behavior - Early error responses are returned as normal JSON, not streamed References: - https://elysiajs.com/essential/handler - https://github.com/elysiajs/elysia/issues/742 Co-Authored-By: Claude Opus 4.5 * fix(api): use await with try-catch for addCompletions in error handlers Changed from fire-and-forget .catch() pattern to await with try-catch for addCompletions calls in non-streaming error handlers. With the refactoring to use `return` instead of `yield`, the fire-and-forget pattern is problematic because: 1. The function returns immediately after return statement 2. The pending promise may not complete before the function exits Using await ensures the completion record is reliably saved before returning the response or exiting the handler. Co-Authored-By: Claude Opus 4.5 * fix(api): continue collecting chunks after client disconnect Previously, when a client disconnected mid-stream: 1. The abort handler immediately saved partial data 2. The for-await loop broke, discarding remaining chunks Now the behavior is: 1. No immediate save on abort (removed abort handler) 2. Continue reading all chunks from upstream 3. Only skip yielding to disconnected client 4. Save full response with "aborted" status when stream ends This ensures complete data is captured for observability even when clients disconnect before the stream finishes. Co-Authored-By: Claude Opus 4.5 * feat: add logging when client disconnects during streaming Log an info-level message when we first detect that a client has disconnected during streaming. This helps track client disconnect events while still continuing to collect all upstream data for accurate token metering. Co-Authored-By: Claude Opus 4.5 * fix: prevent duplicate completion saves and add tool call handling Address CodeRabbit review feedback: - Add alreadySaved check in catch blocks to prevent duplicate database writes if processNonStreamingResponse already saved the completion before consumeTokens throws an error - Add tool call handling (content_block_start/stop) to messages.ts and responses.ts streaming to match completions.ts behavior Co-Authored-By: Claude Opus 4.5 --------- Co-authored-by: Claude Opus 4.5

[Bug] Completion API does NOT handle abort properly #21

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions