feat(health): Auto log collection mode with per-endpoint SSE->Periodic fallback

[mkoci]

Summary

Fixes #1005.

After the merge of #711, collectors.logs.mode is a single global choice. When an SSE-mode endpoint returns HealthError::SseNotAvailable, Collector::start_streaming logs the error and retries forever with exponential backoff (max 30s) instead of falling back to periodic polling. The LogCollectionMode doc string claimed periodic was an automatic fallback, which was a good idea and also a lie.

This PR introduces LogCollectionMode::Auto as the new default. It spawns SSE per-endpoint and transparently (tracing:warn!) downgrades to periodic polling when SSE is unsupported or repeatedly failing.

How it works

• A per-endpoint failure budget classifies connect failures:
  • SseNotAvailable - BMC doesn't support SSE subscriptions - fallback to periodic
  • All other connection errors (network, auth, 5xx, TLS) are counted in a rolling window (default 5 failures in 5 min)
• Hitting either threshold for any BMC records the endpoint_key in a process-wide LogDowngradeRegistry to downgrade the SSE task
• In the discovery loop, prune_finished_logs() drops the exited slot before the respawn pass, and the Auto arm in spawn_collectors_for_endpoint checks registry.is_downgraded(&key) to pick the periodic LogsCollector instead of following normal reconnect logic
• Downgrading is in-memory. In order to regain SSE (perhaps after a BMC firmware update) the Health process would need to be restarted - slightly controversial? Might lead to cases where many BMCs are failing for other reasons so they are downgraded to periodic.
• Observability is a one-shot tracing::warn! on each downgrade transition keyed on endpoint_key + reason; existing _stream_reconnections_total / _stream_errors_total already surface per-endpoint activity pre-downgrade

Config surface

[collectors.logs]
mode = "auto" # new default

[collectors.logs.periodic]
logs_collection_interval = "5m"
state_refresh_interval = "30m"
logs_state_file = "/tmp/logs_collector_{machine_id}.json"

[collectors.logs.auto]
sse_not_available_threshold = 1
connect_failure_window = "5m"
connect_failure_threshold = 5

mode = "auto" requires a [collectors.logs.periodic] section (the downgrade target). [collectors.logs.auto] is optional with sensible defaults. Existing mode = "sse" (no periodic) and mode = "periodic" configs keep working unchanged.

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed - ONGOING
• No testing required (docs, internal refactor, etc.)

Historical context

@Copilot actually did something useful with a comment and suggested we create #1005 as a follow-up on PR #711 (the SSE streaming log support PR).

[github-actions]

🔐 TruffleHog Secret Scan

✅ No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

_{🕐 Last updated: 2026-04-21 17:55:04 UTC | Commit: 2998de4}

[Copilot]

Pull request overview

Adds a new per-endpoint “auto” log collection mode so the health service can start with SSE streaming and transparently downgrade individual endpoints to periodic polling when SSE is unsupported or repeatedly fails, matching the documented behavior and fixing the “retry forever” issue.

Changes:

• Introduces LogCollectionMode::Auto (new default) plus AutoModeConfig downgrade thresholds and updated config/docs/validation.
• Adds an in-memory LogDowngradeRegistry and an auto-mode SSE task (spawn_sse_log_auto) with a per-endpoint failure budget.
• Updates the discovery loop to prune finished log collectors so downgraded endpoints can be respawned as periodic collectors.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/health/src/discovery/spawn.rs	Adds Auto mode spawning logic (SSE first, periodic after downgrade) and unit tests.
crates/health/src/discovery/iteration.rs	Prunes finished log collectors before respawn pass to enable SSE→periodic replacement.
crates/health/src/discovery/context.rs	Adds LogDowngradeRegistry to discovery context and implements prune_finished_logs().
crates/health/src/config.rs	Adds Auto mode + AutoModeConfig, updates docs, defaults, and validation/tests.
crates/health/src/collectors/runtime.rs	Exposes internal helpers and adds Collector::is_finished() + Collector::spawn_task() for auto-mode tasks.
crates/health/src/collectors/mod.rs	Re-exports downgrade-related types and wires in spawn_sse_log_auto.
crates/health/src/collectors/logs/mod.rs	Adds auto + downgrade modules and exports registry/reason types.
crates/health/src/collectors/logs/downgrade.rs	New in-memory downgrade registry with idempotent marking + tests.
crates/health/src/collectors/logs/auto.rs	New auto-mode SSE task with failure budgeting + tests.
crates/health/example/config.example.toml	Updates example config to mode = "auto" with required periodic + optional auto knobs.

Comments suppressed due to low confidence (1)

crates/health/src/config.rs:538

• LogsCollectorConfig::validate enforces that periodic exists for auto, but it doesn't validate the AutoModeConfig knob values themselves. As written, sse_not_available_threshold = 0 or connect_failure_threshold = 0 will cause immediate downgrade on first failure (since the counter will always be >= 0), and connect_failure_window = "0s" will effectively prevent accumulation (window resets every record). Consider adding validation (when mode = Auto and auto is set) that thresholds are > 0 and connect_failure_window is non-zero, returning a clear error message for invalid values.

impl LogsCollectorConfig {
    pub fn validate(&self) -> Result<(), String> {
        match self.mode {
            LogCollectionMode::Auto if self.periodic.is_none() => Err(
                "[collectors.logs.periodic] is required when mode = \"auto\" (used as the \
                 downgrade target when SSE is unavailable)"
                    .to_string(),
            ),
            LogCollectionMode::Periodic if self.periodic.is_none() => {
                Err("[collectors.logs.periodic] is required when mode = \"periodic\"".to_string())
            }
            LogCollectionMode::Sse if self.periodic.is_some() => {
                Err("[collectors.logs.periodic] should not be set when mode = \"sse\"".to_string())
            }
            _ => Ok(()),
        }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.