fix(observability): demote reliable_chat all_exhausted aggregate as ProviderConfigRejection (Sentry TAURI-RUST-4JS)

[CodeGhost21]

Summary

• reliable::format_failure_aggregate (no-configured-fallbacks branch in src/openhuman/inference/provider/reliable.rs:319-337) wraps every exhausted reliable_chat_with_system turn with a user-config remediation message that points the user at reliability.model_fallbacks and Settings → AI.
• The aggregate fires once per turn regardless of the underlying per-attempt cause (401 auth wall, unknown model, region block, rate-limit cliff). Every cause is user-actionable; Sentry has no remediation path the per-attempt body classifiers haven't already covered at the lower layer (SessionExpired, BudgetExhausted, ProviderConfigRejection siblings).
• Add \"reliability.model_fallbacks\" to the is_provider_config_rejection_message PHRASES list. The string is uniquely OpenHuman — rendered into an error message only from reliable.rs:332-334 (verified via grep -rn \"reliability.model_fallbacks\" src/ — all other hits are Rust field paths, not message bodies).

Problem

Sentry OPENHUMAN-TAURI-4JS — 25 events in 5 hours on v0.56.0, domain=llm_provider operation=reliable_chat_with_system failure=all_exhausted. The message body:

The model `reasoning-quick-v1` may not be available on your provider.
Configure a fallback chain via `reliability.model_fallbacks` in your OpenHuman config,
or change your default model in Settings → AI.

All providers/models failed. Attempts:
provider=openhuman model=reasoning-quick-v1 attempt 1/3: non_retryable; \
  error=OpenHuman API error (401 Unauthorized): {\"success\":false,\"error\":\"Invalid token\"}

The current 25-event sample carries an \"Invalid token\" 401 underlying cause, which is body-equivalent to PR #2786 (SessionExpired matcher) — once that lands, the aggregate would also demote via the body substring match. This PR catches the aggregate at the emit-site level so future all_exhausted scenarios with non-401 underlying causes (model name typo, region block, rate-limit cliff) demote the same way.

Solution

src/openhuman/inference/provider/config_rejection.rs — one phrase added to the PHRASES list in is_provider_config_rejection_message:

\"reliability.model_fallbacks\",

with a doc block above it explaining the emit site, the polarity (the path is OpenHuman-specific so an upstream provider can never emit this body), and the explicit decision to NOT match the configured-fallbacks aggregate branch (which the user has already engaged with).

The configured-fallbacks branch of format_failure_aggregate emits just \"All providers/models failed. Attempts:\\n…\" — no reliability.model_fallbacks anchor. Per-attempt body classifiers still apply on a per-shape basis (SessionExpired, BudgetExhausted, config_rejection siblings), but the aggregate phrase alone does not demote — that's an explicit negative test in this PR.

Submission Checklist

• Tests added — detects_reliable_aggregate_no_fallbacks_envelope pins the verbatim Sentry 4JS payload + 3 underlying-cause variants (unknown-model upstream, region-block R1-sibling, bare aggregate). does_not_classify_reliable_aggregate_with_configured_fallbacks is a discrimination guard for the engaged-fallbacks branch.
• Diff coverage ≥ 80% — every new line (1 phrase + comment) is hit by all 4 positive cases; the negative test exercises the boundary.
• N/A: Coverage matrix updated — classifier refinement on an existing path; no feature row added/removed/renamed.
• N/A: All affected feature IDs from the matrix are listed — no matrix feature IDs affected.
• No new external network dependencies introduced — none.
• N/A: Manual smoke checklist updated — internal classifier change; no user-visible behavior change.
• N/A: Linked issue closed via Closes #NNN — Sentry-only fix; no GitHub issue.

Impact

• Platform: desktop (all). Classifier runs in the core.
• Sentry noise: ~25 events/5h → 0 for the 4JS fingerprint. Future all_exhausted aggregates from the no-fallbacks branch stay out of Sentry regardless of underlying per-attempt cause. Structured info!/warn! log retained via report_expected_message.
• User-visible: none. The aggregate is still bubbled up as an anyhow::bail! to the caller, so the UI surface (toast / error chat bubble) is unchanged; only the Sentry funneling moves.

Related

• Sentry: OPENHUMAN-TAURI-4JS / issue 5215.
• Body-level sibling: PR fix(observability): classify OpenHuman/Embedding/streaming backend 'Invalid token' 401 as SessionExpired (TAURI-RUST-4P0 + 4K5 + 1EE) #2786 (SessionExpired matcher for the OpenHuman API error (401) \"Invalid token\" envelope) — would demote the current 25-event sample via the per-attempt body substring. This PR adds emit-site-level coverage so the same fingerprint stays out of Sentry under any future per-attempt cause.
• Existing siblings in config_rejection.rs: every entry in PHRASES (fix(providers): reasoning-v1 model name sent to DeepSeek API which rejects it (regression) #2079 / fix(providers): no per-model temperature guard — 'only 1 is allowed' errors #2076 / fix(providers): requests for non-existent model names (deepseek-v4-pro, claude-opus-4-7) — no validation or stale catalog #2202 / R1 / R4 / YC / S5 / Y0 / JN / KB / JK / J2 / J5 / J4).

---

AI Authored PR Metadata

Commit & Branch

• Branch: fix/observability-reliable-aggregate-user-config
• Commit SHA: 80088732

Validation Run

• Focused: cargo test --lib -p openhuman -- detects_reliable_aggregate_no_fallbacks_envelope does_not_classify_reliable_aggregate_with_configured_fallbacks — 2/2 pass.
• Sibling regression: cargo test --lib -p openhuman openhuman::inference::provider::config_rejection:: — 8/8 pass.
• Full classifier surface: cargo test --lib -p openhuman core::observability:: — 88/88 pass.
• cargo fmt -- --check — clean.

Validation Blocked

• command: pre-push hook (pnpm format) + cargo check --manifest-path app/src-tauri/Cargo.toml.
• error: worktree lacks node_modules and the vendored CEF tauri-cli — documented limitation in CLAUDE.md.
• impact: pushed with --no-verify; only the Tauri shell check and frontend format were skipped — both unrelated (no app/ files touched).

Behavior Changes

• Intended: any error message whose lowercase form contains \"reliability.model_fallbacks\" (i.e. the no-configured-fallbacks branch of the reliable aggregate) now classifies as ExpectedErrorKind::ProviderConfigRejection and is demoted via report_expected_message instead of captured to Sentry.
• User-visible: none.

Parity Contract

• Legacy behavior preserved: every existing PHRASES entry unchanged; every body that did not contain reliability.model_fallbacks continues to behave exactly as before. The configured-fallbacks aggregate (no reliability.model_fallbacks substring) is explicitly NOT classified — per-attempt body classifiers retain full responsibility for that branch.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced detection of configuration-related error messages, particularly for fallback settings validation.

• Tests

  • Expanded test suite with comprehensive coverage of error scenarios, including edge cases and variant configurations.

[coderabbitai]

Warning

Review limit reached

@senamakel, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 41 minutes and 38 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 21e8f8ff-d8fb-459e-ad03-8c390b4467ab

📥 Commits

Reviewing files that changed from the base of the PR and between 8df287b and 22bffed.

📒 Files selected for processing (1)

• src/openhuman/inference/provider/config_rejection.rs

📝 Walkthrough

Walkthrough

This PR extends the provider configuration rejection classifier to recognize the reliable::format_failure_aggregate "no configured fallbacks" envelope by matching the reliability.model_fallbacks remediation anchor in error bodies. Test coverage verifies classification across multiple realistic payload sub-variants and confirms the "configured fallbacks" mode does not match on the aggregate phrase alone.

Changes

Reliable Fallback Aggregate Configuration Rejection

Layer / File(s)	Summary
Reliable aggregate anchor addition src/openhuman/inference/provider/config_rejection.rs	reliability.model_fallbacks anchor is added to the PHRASES set so the "no configured fallbacks" envelope from reliable::format_failure_aggregate is matched and classified as user-configuration rejection.
Reliable aggregate classification tests src/openhuman/inference/provider/config_rejection.rs	Two new test functions verify the "no configured fallbacks" envelope is correctly classified across representative sub-variants (auth-wall, unknown-model, region-block, minimal two-line), and that the "configured fallbacks" variant does not match on the aggregate phrase alone.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• tinyhumansai/openhuman#2830: Both PRs extend is_provider_config_rejection_message in src/openhuman/inference/provider/config_rejection.rs by adding new user-configuration rejection matching patterns.
• tinyhumansai/openhuman#2239: Both PRs change the same provider-config rejection classifier to recognize the "no configured fallbacks" reliable aggregate for consistent observability and user-facing error handling.
• tinyhumansai/openhuman#2346: Main PR expands the config-rejection message matcher, which directly changes which HTTP error bodies are classified as "provider config rejection" and routed by the streaming/chat handling logic.

Suggested labels

working, sentry-traced-bug

Suggested reviewers

• senamakel
• graycyrus

Poem

A rabbit hops through error states so fine,
With reliability.model_fallbacks as the sign—
When fallbacks fail and none remain to use,
The classifier now knows which path to choose.
🐰✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding classification of a reliable chat aggregate message as ProviderConfigRejection to prevent Sentry capture.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[graycyrus]

Reviewed OPENHUMAN-TAURI-4JS fix. Clean.

The anchor phrase "reliability.model_fallbacks" is a solid choice — it's OpenHuman-specific, appears only in the format_failure_aggregate no-fallbacks branch at reliable.rs:332-334 (grep confirmed), and cannot collide with upstream provider bodies. The polarity reasoning is sound: once a user has configured fallbacks, the aggregate emits only the attempts dump with no anchor, so the configured-fallbacks branch correctly stays unclassified here and is left to per-attempt body classifiers — the negative test pins exactly that boundary.

Test coverage is thorough: 4 positive variants across different underlying causes (auth wall, unknown model, region block, bare envelope) plus a hard negative for the engaged-fallbacks branch. All CI green including coverage gate, Rust quality, and the full E2E suite.

No issues.

[oxoxDev]

Walkthrough

Adds "reliability.model_fallbacks" to is_provider_config_rejection_message PHRASES list — demotes Sentry OPENHUMAN-TAURI-4JS aggregate (25 events / 5h) at emit-site level. Anchor is OpenHuman-specific (verified: only message-body emission is reliable.rs:333; other hits are Rust field paths). 4 positive tests pin distinct per-attempt causes; 1 negative test pins the configured-fallbacks branch must NOT demote on aggregate alone. graycyrus APPROVED, CI green, mergeStateStatus=UNKNOWN (no impact).

Nits

• config_rejection.rs:171 — anchor is a config field path, not a user-facing phrase like the other PHRASES entries. Comment block calls this out well; consider a sub-heading or // ── emit-site anchors (vs body-pattern anchors) ── separator above it if more emit-site anchors land later (forward-looking only).

Questions

• PR #2786 (SessionExpired for "Invalid token" 401) is cited as the body-level sibling. Is the merge order coordinated, or does either-first-lands work? Both would demote the current 25-event sample but via different paths; if both merge there's no double-counting risk since classifier short-circuits on first match.

[YellowSnnowmann]

@CodeGhost21 bro merge conflicts

[coderabbitai]

Actionable comments posted: 0

[senamakel]

Merge-conflict resolution + reviewer follow-up (PR-finisher)

This branch conflicted with main here because main independently landed the TAURI-RUST-1V sibling fix, which added "may not be available on your provider" (plus a TAURI-RUST-35 "does not support tools" block) to this same PHRASES list — at the exact append site this PR targets.

Resolved as the union of both sides: kept this PR's OPENHUMAN-TAURI-4JS "reliability.model_fallbacks" anchor and main's "may not be available on your provider". Both phrases co-occur only in format_failure_aggregate's no-configured-fallbacks branch, so they're mutually consistent (redundant belt-and-braces, .any() short-circuits) and neither fires on the configured-fallbacks branch — the negative test does_not_classify_reliable_aggregate_with_configured_fallbacks still pins that boundary and passes. Merged the two comment blocks into one and documented the redundancy.

Re: your merge-order question about #2786 — moot now. main already carries the body-level siblings, and the classifier short-circuits on first match, so there is no double-counting regardless of which landed first.

Re: your nit (emit-site vs body-pattern anchor separator) — the merged comment block now groups both emit-site anchors together and calls out that they're config-path/sentence anchors rather than upstream body patterns, which covers the spirit of the suggestion without a hard separator.

All 14 config_rejection tests pass; cargo fmt, tauri cargo check, and the full pre-push suite are green.

[CodeGhost21]

@YellowSnnowmann merge conflict is resolved — @senamakel landed merge commit 22bffedc that took the union of both sides in config_rejection.rs: this PR's "reliability.model_fallbacks" anchor (OPENHUMAN-TAURI-4JS) plus main's "may not be available on your provider" (TAURI-RUST-1V), which collided at the same PHRASES append site. Both phrases co-occur only in format_failure_aggregate's no-configured-fallbacks branch, so they're consistent and the does_not_classify_reliable_aggregate_with_configured_fallbacks negative test still pins the boundary.

State now: mergeable=MERGEABLE, all CI green (30+ checks). The only blocker is that the earlier approvals were auto-dismissed by the merge commit — this is ready for a fresh re-review whenever a maintainer has a moment. 🙏