fix(observability): skip Sentry for transport-level + transient-upstream errors (TAURI-32 / 5Z / 2G)

[CodeGhost21]

Summary

Consolidates three transport-level Sentry-noise fixes (previously split across #1594 + #1601) into a single PR. All three share the same classifier (ExpectedErrorKind::NetworkUnreachable / TransientUpstreamHttp in src/core/observability.rs) and the same call-site pattern (swap report_error → report_error_or_expected so the classifier picks up the error chain). One review, one merge — no duplicated classifier commits across branches.

Covers

• OPENHUMAN-TAURI-32 — web_channel.run_chat_task (reqwest transport errors: DNS / TCP / TLS / cert / ISP-block shapes, observed from RU user where api.tinyhumans.ai was unreachable). Adds the NetworkUnreachable variant + is_network_unreachable_message matching 8 transport shapes.
• OPENHUMAN-TAURI-5Z — agent.run_single aggregate path. Adds the TransientUpstreamHttp variant + is_transient_upstream_http_message pinned to the "<provider> API error (<status>" wire format with TRANSIENT_PROVIDER_HTTP_STATUSES (408/429/502/503/504), so it catches per-turn 5xx/429/timeout that has escaped the provider-scoped before_send filter under a different domain tag.
• OPENHUMAN-TAURI-2G — IntegrationClient::post and IntegrationClient::get (17 events from a SG user, all tls handshake eof). Same classifier picks them up.

Problem

Transport-level reqwest failures fire before any HTTP status is observed. There is no status, no trace, no payload — nothing Sentry can group, facet, or action on. The reliable-provider layer already retries with backoff/fallback; when those exhaust, downstream call sites (web_channel, agent.run_single, integrations.{get,post}) re-emit report_error on top of the aggregate event. One user with a flaky connection (VPN drop, captive portal, ISP MITM, transient handshake reset) produces a sustained stream of identical "events" that are pure environmental noise.

The same noise pattern hits the agent layer when transient upstream HTTP statuses (408/429/502/503/504) escape the provider-scoped before_send filter under a domain=agent tag — handled here by the second classifier variant.

Solution

src/core/observability.rs:

• Add ExpectedErrorKind::NetworkUnreachable and is_network_unreachable_message matching 8 transport-level shapes (error sending request for url, dns error, connection refused, connection reset, network is unreachable, no route to host, tls handshake, certificate verify failed).
• Add ExpectedErrorKind::TransientUpstreamHttp and is_transient_upstream_http_message pinned to the "<provider> API error (<status>" wire format from providers::ops::api_error for TRANSIENT_PROVIDER_HTTP_STATUSES (408/429/502/503/504) — anchored to the "api error (" prefix so a free-form mention of "504" elsewhere isn't silenced.
• report_expected_message handles both new variants with tracing::warn!, so sentry-tracing emits a breadcrumb instead of an error event.

src/openhuman/channels/providers/web.rs:

• Switch the run_chat_task failure site from report_error → report_error_or_expected so it routes through the classifier.
• The user-facing chat_error event is unchanged — still goes through classify_inference_error, still emits the sanitized generic message via the existing socket bridge.

src/openhuman/agent/harness/session/runtime.rs:

• Switch Agent::run_single from report_error → report_error_or_expected. Result::Err return and DomainEvent::AgentError publish are unchanged — only the Sentry event is suppressed.

src/openhuman/integrations/client.rs:

• Switch IntegrationClient::post and IntegrationClient::get transport-failure sites from report_error → report_error_or_expected. Non-2xx and envelope-error paths are unchanged — those are actionable backend failures and should continue to surface.

Status-bearing failures (404 / 500 / etc.) outside the curated transient set are untouched by the new classifier and still surface via their existing paths.

Submission Checklist

• Tests added or updated — three new unit tests in core/observability.rs: classifies_network_unreachable_errors (8 transport-level patterns including the literal OPENHUMAN-TAURI-32 / TAURI-2G message bodies), classifies_transient_upstream_http_errors (the 5 transient codes against the canonical "API error (<status>" wire format), and two regression guards: does_not_classify_unrelated_provider_errors_as_network and does_not_classify_actionable_provider_errors_as_transient_upstream (locks 404 / 500 outside both classifiers). Existing start_chat_emits_sanitized_chat_error_on_inference_failure already drives the new code path end-to-end with the exact "error sending request for url (…)" payload and stays green.
• Diff coverage ≥ 80% — N/A locally; will run via CI. All new lines in observability.rs are covered by the new tests; the one-line call-site swaps in web.rs, runtime.rs, and client.rs are exercised by existing forced-error tests in those modules.
• Coverage matrix updated — N/A: behaviour-only change (Sentry classification, no new user-visible feature row).
• All affected feature IDs listed under ## Related — see below.
• No new external network dependencies introduced.
• Manual smoke checklist updated — N/A: change is on the error-reporting side, not a release-cut surface.
• Linked issue — see ## Related.

Impact

• Runtime: desktop only (web channel, agent harness, and integrations client all run in the core sidecar).
• Security: no change. Error text was already redacted via sanitize_api_error upstream; this PR only changes the log level / Sentry routing, not the message content.
• Performance: negligible — one extra to_ascii_lowercase + contains lookup per error site that calls report_error_or_expected.
• Migration / compatibility: none. Existing callers of report_error are unaffected.

Related

• Closes OPENHUMAN-TAURI-32
• Closes OPENHUMAN-TAURI-5Z
• Closes OPENHUMAN-TAURI-2G
• Supersedes fix(observability): skip Sentry for net errors (OPENHUMAN-TAURI-32) #1594 (the network-unreachable classifier + agent runtime switch, now folded in here)
• Prior art: fix(observability): skip Sentry for param-validation errors (OPENHUMAN-TAURI-20) #1575 (param-validation), fix(observability): skip Sentry report for provider "Budget exceeded" errors #1530 (budget-exceeded), fix(observability): drop transient upstream HTTP from Sentry (429/408/502/503/504) #1529 (transient upstream HTTP at provider layer)

---

AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issues

• OPENHUMAN-TAURI-32 — https://tinyhumans.sentry.io/issues/OPENHUMAN-TAURI-32
• OPENHUMAN-TAURI-5Z — https://tinyhumans.sentry.io/issues/OPENHUMAN-TAURI-5Z
• OPENHUMAN-TAURI-2G — https://tinyhumans.sentry.io/issues/OPENHUMAN-TAURI-2G

Commit & Branch

• Branch: fix/integrations-transport-sentry-2g (based on origin/main)
• Commits: 25b1a998 (NetworkUnreachable + web channel) · c41f8416 (integrations client) · 202a82e3 (TransientUpstreamHttp + agent runtime — cherry-pick from fix(observability): skip Sentry for net errors (OPENHUMAN-TAURI-32) #1594)

Validation Run

• pnpm --filter openhuman-app format:check — N/A: Rust-only change.
• pnpm typecheck — N/A: Rust-only change.
• Focused tests: cargo test --manifest-path Cargo.toml --lib core::observability:: (15/15 pass on the combined branch).
• Rust fmt/check: cargo check --manifest-path Cargo.toml (clean, only pre-existing dead-code warnings).
• Tauri fmt/check: N/A — no app/src-tauri/ changes.

Validation Blocked

• command: pnpm pre-push (lint:commands-tokens / react-hooks/set-state-in-effect)
• error: regex/lint hits on src/components/commands/ Tailwind tokens and pre-existing React-hooks warnings in BootCheckGate.tsx, RotatingTetrahedronCanvas.tsx, CommandProvider.tsx, TriggerToggles.tsx, MemoryNavigator.tsx — unrelated to this PR's Rust-only diff.
• impact: Pushed with --no-verify per CLAUDE.md guidance for hook failures unrelated to the changes.

Known CI Failures (pre-existing on main)

• Rust Core Tests + Quality and Rust Core Coverage (cargo-llvm-cov) will be red on this PR with the same 38 test failures that also fail on main (e.g. main run 25785337660 on commit de33b129). Failing tests are concentrated in openhuman::config::ops::*, openhuman::credentials::cli::*, openhuman::local_ai::schemas::*, openhuman::subconscious::executor::*, openhuman::threads::ops::*, openhuman::update::ops::*, plus core::jsonrpc::tests::thread_not_found_rpc_error_does_not_report_to_sentry — all infrastructure flakiness from shared global state (TEST_ENV_LOCK, sentry hub, tracing subscriber). All 38 pass locally in isolation; specifically core::jsonrpc::tests::thread_not_found_rpc_error_does_not_report_to_sentry passes and proves "unrelated RPC errors still reach Sentry" — i.e. the combined classifier does not over-match.

Behavior Changes

• Intended behavior change: transport-level connection failures and transient upstream HTTP failures (408/429/502/503/504) from web_channel.run_chat_task, agent.run_single, and IntegrationClient::{get,post} no longer create Sentry error events; they emit a warn-level breadcrumb instead.
• User-visible effect: none. The on-screen chat_error message and integration error surfaces are unchanged.

Parity Contract

• Legacy behavior preserved: report_error continues to emit error events at all other call sites; only report_error_or_expected gates on the classifier, and only the new NetworkUnreachable / TransientUpstreamHttp shapes are added — existing LocalAiDisabled / ApiKeyMissing behavior is unchanged.
• Guard/fallback/dispatch parity checks: does_not_classify_unrelated_provider_errors_as_network and does_not_classify_actionable_provider_errors_as_transient_upstream lock 404 / 500 outside the new classifiers.

Duplicate / Superseded PR Handling

• Duplicate PR(s): fix(observability): skip Sentry for net errors (OPENHUMAN-TAURI-32) #1594 (fix/observability-network-unreachable-sentry) — same classifier + web channel + agent runtime changes.
• Canonical PR: this one.
• Resolution: closing fix(observability): skip Sentry for net errors (OPENHUMAN-TAURI-32) #1594 as superseded; its agent-runtime commit (a21dd58a) is cherry-picked into this branch as 202a82e3.

Summary by CodeRabbit

• Bug Fixes

  • Improved detection of transport-level network failures (DNS/TCP/TLS/connectivity) so they’re classified as expected and logged as warnings instead of triggering error events.
  • Updated chat, integration client, and agent runtime reporting to use the new expected-vs-unexpected handling, reducing noisy alerts.

• Tests

  • Added unit tests covering multiple network-failure message variants and verifying provider errors remain correctly classified.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 31364b1b-56ce-429f-b6a7-ec94a4cb7e84

📥 Commits

Reviewing files that changed from the base of the PR and between 202a82e and a4919cb.

📒 Files selected for processing (1)

• src/core/observability.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• src/core/observability.rs

---

📝 Walkthrough

Walkthrough

This PR classifies transport-level “can’t reach server” failures as NetworkUnreachable and logs them as warn-level expected events. It updates three call sites (web chat, integrations client POST/GET, agent runtime) to use the expected-error reporting path and adds unit tests for classification behavior.

Changes

Network-unreachable observability handling

Layer / File(s)	Summary
NetworkUnreachable variant and detection logic src/core/observability.rs	ExpectedErrorKind gains NetworkUnreachable. expected_error_kind adds detection for transport/DNS/TCP/TLS/certificate/connectivity failure markers via is_network_unreachable_message and transient-upstream checks. Unit tests validate positive matches and non-classification of status-bearing provider errors.
Expected message reporting for NetworkUnreachable src/core/observability.rs	report_expected_message handles NetworkUnreachable by emitting a standardized tracing::warn! breadcrumb (“skipped expected network-unreachable error”) instead of reporting an error event.
Web chat error reporting integration src/openhuman/channels/providers/web.rs	start_chat task error handler switches from report_error to report_error_or_expected, routing transport failures through the new classification and documenting the warn-level breadcrumb behavior.
Integration client error reporting updates src/openhuman/integrations/client.rs	IntegrationClient::post and IntegrationClient::get .map_err paths now call report_error_or_expected (instead of report_error), preserving enriched error chains but classifying common reqwest transport/connect/TLS failures as expected.
Agent runtime error reporting update src/openhuman/agent/harness/session/runtime.rs	Agent::run_single Err branch now calls report_error_or_expected so exhausted retries / transient upstream failures are demoted to expected warn-level breadcrumbs while preserving existing AgentError publish and error return.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• Transient upstream failures reach Sentry as errors, drowning real bugs #1608: Similar observability changes to classify transport/transient upstream failures and update call sites to demote expected transport errors.

Possibly related PRs

• tinyhumansai/openhuman#1264: Modifies src/core/observability.rs and callsites to change error reporting foundations; related to this PR's classification extensions.
• tinyhumansai/openhuman#1529: Suppresses Sentry reports for transient upstream HTTP failures; conceptually overlaps with transient-upstream handling here.
• tinyhumansai/openhuman#1512: Adjusts expected classification for provider HTTP statuses (e.g., 429); related to transient-status classification.

Suggested reviewers

• senamakel

Poem

🐰 I sniffed the wires and gave a twitch—
A DNS hiccup, a TLS glitch.
Not every fall needs a siren's song,
Now we log a whisper when networks go wrong.
Hopping on, the observability hop! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: introducing observability fixes to skip Sentry reporting for transport-level and transient upstream HTTP errors while preserving error handling.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[graycyrus]

PR Review — fix(observability): skip Sentry for transport-level + transient-upstream errors

Walkthrough

Adds two new ExpectedErrorKind variants (NetworkUnreachable, TransientUpstreamHttp) to src/core/observability.rs and switches four call sites from report_error → report_error_or_expected. Transport-level connection failures and exhausted-retry transient HTTP responses are demoted to tracing::warn! breadcrumbs instead of Sentry error events. User-visible behavior (chat_error, AgentError, integration error returns) is entirely preserved.

Changes

File	Summary
src/core/observability.rs	New NetworkUnreachable (8 transport substrings) + TransientUpstreamHttp (TRANSIENT_PROVIDER_HTTP_STATUSES × "api error (" prefix) variants; report_expected_message match arms; 4 new unit tests
src/openhuman/agent/harness/session/runtime.rs	Agent::run_single → report_error_or_expected
src/openhuman/channels/providers/web.rs	run_chat_task → report_error_or_expected
src/openhuman/integrations/client.rs	IntegrationClient::{post,get} → report_error_or_expected

---

Findings

[major] extra tags silently dropped on expected path (observability.rs:94-96)

When an error IS classified as expected, report_error_or_expected calls report_expected_message(kind, &message, domain, operation) — the extra slice is not forwarded. Caller-supplied tags (session_id, error_kind, path, thread_id, request_id) are silently dropped from the warn breadcrumb.

This is pre-existing (same for LocalAiDisabled/ApiKeyMissing), but the two new variants are exactly where those tags matter most for ops debugging. In client.rs, path is the only way to know which integration endpoint was affected — and it's currently thrown away.

Suggestion: Forward extra to report_expected_message and include the tags as structured fields in the tracing::warn! calls.

[major] format! allocation inside .any() in is_transient_upstream_http_message (observability.rs:104-108)

Allocates a new String per status code per call. Error path so no perf regression, but unnecessary and diverges from the &str-only style of is_network_unreachable_message:

// suggestion: pre-compute as compile-time constants
fn is_transient_upstream_http_message(lower: &str) -> bool {
    const PATTERNS: &[&str] = &[
        "api error (408",
        "api error (429",
        "api error (502",
        "api error (503",
        "api error (504",
    ];
    PATTERNS.iter().any(|p| lower.contains(p))
}

[minor] "connection reset" substring match may be too broad (observability.rs:80)

A non-transient 500 whose response body contains "connection reset" (e.g. nginx upstream reset) would be silenced as NetworkUnreachable. Low-probability but worth a targeted regression guard test:

#[test]
fn does_not_classify_provider_error_with_connection_reset_body_as_network() {
    assert_eq!(
        expected_error_kind(
            "Provider API error (500): upstream connection reset while reading response"
        ),
        None
    );
}

If this test fails, tighten the match to exclude strings that contain the "api error (" provider prefix.

Nitpicks

• report_expected_message says "skipped transient upstream HTTP error" but other arms say "skipped expected …" — inconsistent grep-ability.
• is_transient_upstream_http_message closing ))) is visually dense — confirm cargo fmt is happy.
• runtime.rs:519 passes ("error_kind", sanitized_message.as_str()) as an extra tag, but for classified errors this tag is dropped (see finding 1), making the call-site signature misleading.

Verified / Looks Good

• TransientUpstreamHttp correctly anchored to "api error (" prefix — bare "504" mentions don't match ✓
• NetworkUnreachable operates on reqwest error chain, not HTTP body content in client.rs ✓
• DomainEvent::AgentError publish and Err return unconditionally preserved in runtime.rs ✓
• classify_inference_error in web.rs runs before the detailed string → user-visible chat_error unaffected ✓
• TRANSIENT_PROVIDER_HTTP_STATUSES reused (single source of truth) ✓
• Negative guard tests lock 404/500 outside both classifiers ✓
• Existing E2E test start_chat_emits_sanitized_chat_error_on_inference_failure exercises the new code path ✓

Overall: clean, well-motivated PR with solid test coverage. The two major items are real but straightforward fixes.