Skip to content

feat(usage): classify finish_reason values and flag error reasons in has_error#3071

Open
kilo-code-bot[bot] wants to merge 1 commit intomainfrom
feat/finish-reason-error-classification
Open

feat(usage): classify finish_reason values and flag error reasons in has_error#3071
kilo-code-bot[bot] wants to merge 1 commit intomainfrom
feat/finish-reason-error-classification

Conversation

@kilo-code-bot
Copy link
Copy Markdown
Contributor

@kilo-code-bot kilo-code-bot Bot commented May 6, 2026

Summary

  • Adds a shared finishReason module (apps/web/src/lib/ai-gateway/finishReason.ts) with a zod enum (FinishReasonSchema) covering every finish_reason / stop_reason / Responses API status value we currently see across OpenAI/OpenRouter chat completions, Anthropic Messages, OpenAI Responses, and Vercel AI SDK style outputs.
  • Splits those values into ERROR_FINISH_REASONS (truncation/refusal/upstream failure/interrupted) and NON_ERROR_FINISH_REASONS (normal completion + catch-alls), and exposes isErrorFinishReason().
  • Wires isErrorFinishReason(finish_reason) into the hasError calculation of all three microdollar usage parsers — the OpenRouter chat completions stream + string parser (processUsage.ts), the Anthropic Messages stream + string parser (processUsage.messages.ts), and the OpenAI Responses stream parser (processUsage.responses.ts). The Responses API string parser keeps its existing stricter status !== 'completed' rule untouched.
  • Adds a unit test for the classification helper and zod enum.

Net effect: a 200 OK response that ends with stop_reason: "refusal", finish_reason: "length" / "content_filter" / "error", etc. is now recorded with has_error: true in microdollar_usage, instead of being silently logged as a successful row. Unrecognised provider strings remain non-errors so we don't generate spurious failures when a provider invents a new value.

Verification

  • Manually traced each parser path to confirm the signal flows from the SSE event / response body into coreProps.hasError for both error and non-error finish_reason values.
  • Reviewed existing test fixtures (finish_reason: 'stop', approved snapshots with 'end_turn' / 'completed') — all classified as non-error so existing assertions are unaffected.

Visual Changes

N/A

Reviewer Notes

  • I deliberately classified length and max_tokens as errors (truncation surfaces something product/customers usually want to see). If you'd rather keep truncation as a non-error, move those two entries between the two arrays in finishReason.ts and the test will still pass after a one-line update.
  • unknown and other are treated as non-errors so that brand-new upstream values don't immediately spike error rates; pair this with existing Sentry warnings if we want visibility on novel values.
  • The Responses API string parser already had status !== 'completed' as its error rule, which is strictly broader than isErrorFinishReason. I left it alone to avoid weakening that path.
  • Skipped pnpm typecheck / pnpm test / pnpm format per the request not to run long-running tasks before pushing — please re-run CI if you want a clean signal.

…has_error

Introduce a shared zod enum + helper (isErrorFinishReason) for the set of
finish_reason / stop_reason / status values we observe across OpenAI chat
completions, OpenRouter, Anthropic Messages API, OpenAI Responses API, and
Vercel AI SDK style responses.

Reasons that indicate truncation, refusal, content filtering, upstream
failure, or an interrupted in_progress stream now flip has_error to true
in all three usage parsers (chat completions, messages, responses string
path). Normal completion reasons (stop, end_turn, tool_use, completed,
stop_sequence, tool_calls/tool-calls) and unclassified catch-alls
(unknown, other, null) keep has_error driven only by status code and
abort signals as before.
@kilo-code-bot
Copy link
Copy Markdown
Contributor Author

kilo-code-bot Bot commented May 6, 2026

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
apps/web/src/lib/ai-gateway/finishReason.ts 41 Streaming Responses cancelled statuses are not classified as errors, so a 200 stream ending with status: "cancelled" can be recorded as has_error: false unless the reader abort path also fires.
Other Observations (not in diff)

Issues found in unchanged code that cannot receive inline comments:

File Line Issue
Files Reviewed (5 files)
  • apps/web/src/lib/ai-gateway/finishReason.ts - 1 issue
  • apps/web/src/lib/ai-gateway/finishReason.test.ts - 0 issues
  • apps/web/src/lib/ai-gateway/processUsage.ts - 0 issues
  • apps/web/src/lib/ai-gateway/processUsage.messages.ts - 0 issues
  • apps/web/src/lib/ai-gateway/processUsage.responses.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by gpt-5.5-20260423 · 384,645 tokens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant