Feature: Tunnel Health Check and Fast Recovery

[crazydi4mond]

Problem

Currently, the tunnel lacks robust health monitoring and automatic recovery mechanisms. When the tunnel fails (network issues, resolver unavailability, silent packet loss), detection is slow and recovery requires manual restart.

This becomes especially critical in highly restricted network environments where the tunnel may be established through an intermediary server running in non-interactive or headless mode. In such scenarios, recovering from a failure requires restarting the tunnel from the server side—but access to that server may be limited, intermittent, or available only during narrow time windows. Without self-healing capabilities, a silent tunnel failure can render the connection unusable until the next opportunity for manual intervention.

Current State

• QUIC keep-alive is enabled (400ms) but never verified
• Connection close/reset is detected via callbacks, but triggers program exit
• No automatic reconnection logic
• No per-resolver health tracking
• Recursive mode has no timeout for unresponsive resolvers
• Path quality metrics are collected but not used for failure detection

Proposed Solution

Client-Side (Primary)

1. Active Health Probing

   • Periodic lightweight probes independent of data transfer
   • Configurable interval (e.g., --health-check-interval)
   • Detect silent failures within seconds

2. Automatic Reconnection

   • Exponential backoff on connection failure
   • Configurable retry budget and max delay
   • Preserve TCP listeners during reconnection attempts

3. Per-Resolver Health Tracking

   • Track success/failure rate per resolver
   • Automatic failover to healthy resolvers
   • Circuit breaker pattern for repeatedly failing resolvers

4. Path Quality Thresholds

   • Use existing RTT/loss metrics for degradation detection
   • Switch paths when quality drops below threshold

Server-Side (Optional)

• Session idle timeout tracking (already partially exists in UDP fallback)
• Metrics/logging for client health events

---

We'd love to hear your feedback on these proposed solutions. We're happy to contribute to the implementation—just wanted to align on the overall strategy and approach before diving into the code.

[Mygod]

Confirming that this bug can be reproduced.

[Mygod]

What I can currently reliably reproduce is that if one underlying stream is blocked, it would actually block the entire QUIC stream from progressing and thus stall the connection. A fix for this is in progress but it may not necessarily be the same bug reported by @crazydi4mond.

[saeidshirazi]

@Mygod, could you please prepare a document explaining how to fully set this up for end users in Iran? The entire network has been cut off, and we need an immediate solution to bypass censorship.
Perhaps integrating this with https://github.com/MHSanaei/3x-ui would be one of the best solutions.

[crazydi4mond]

@saeidshirazi we are working on those solutions. Please check this organization in a few days:
https://github.com/net2share

[Mygod]

Updated issue status below. Disclaimer: Just a note to myself. Take everything below with a huge grain of salt.

Slipstream Tunnel Health: Four Gaps (Status + Rationale)

1) Keep-alive is enabled but not verified

Status: Present keep-alive, no active verification.

Evidence: --keep-alive-interval default 400ms in crates/slipstream-client/src/main.rs; applied via picoquic_enable_keep_alive in crates/slipstream-client/src/runtime.rs. No code checks for keep-alive success or missing acks.

Impact: Silent tunnel failures don’t trigger reconnect; the session can appear alive indefinitely.

Worth fixing? Yes (high impact). This is the core silent-failure issue.

Suggested direction: Add an application-level health probe with timeout and a reconnect trigger.

---

2) No per-resolver health tracking / circuit breaker

Status: Missing.

Evidence: ResolverState tracks debug counters and probe attempts but no success/failure scoring or disable/enable logic (crates/slipstream-client/src/dns/resolver.rs, crates/slipstream-client/src/dns/response.rs).

Impact: If one resolver goes dark or degrades, the client can keep using it indefinitely, delaying recovery.

Worth fixing? Yes (high impact). Enables automatic failover and faster recovery in hostile networks.

Suggested direction: Track rolling success/failure; temporarily disable failing resolver paths; re-enable after backoff.

---

3) Recursive mode has no timeout for unresponsive resolvers

Status: Missing.

Evidence: expire_inflight_polls only applies to authoritative mode (AUTHORITATIVE_POLL_TIMEOUT_US in crates/slipstream-client/src/dns/poll.rs). Recursive mode has no equivalent timeout.

Impact: Recursive resolvers can become permanently stuck, with no trigger to retry or fail over.

Worth fixing? Yes (likely bug). Minimal change, clear correctness gain.

Suggested direction: Add timeout tracking for recursive requests and integrate with resolver health.

---

4) Path quality metrics collected but not used for failure detection

Status: Metrics exist, unused for decisions.

Evidence: fetch_path_quality used for pacing/debug logging only (crates/slipstream-client/src/runtime.rs, crates/slipstream-client/src/runtime/path.rs).

Impact: Degraded paths are not penalized, so the tunnel may stay on poor or failing paths.

Worth fixing? Yes, but later. Useful but riskier (false positives, noisy metrics).

Suggested direction: Use as a secondary signal after probes/health tracking; apply conservative thresholds and hysteresis.

---

Summary Recommendation

Prioritize (1) health probes + timeout and (2) resolver health/circuit breaker, then (3) recursive timeout. Add (4) path-quality thresholds after core detection is in place to avoid aggressive failovers.

[Mygod]

I can still reproduce some connection stalling. Investigating.

Update: I was running the old code. Never mind.

[EndPositive]

I'm not sure why per-resolver health is an issue to be handled in slipstream code? Isn't the whole point of using QUIC that QUIC itself will handle path connectivity issues? In my earlier research, QUIC will automatically remove that path, and even resend frames on other paths. Also if one resolver drops, that wouldn't drop the whole QUIC connection, but simply that resolver's path.

I also don't understand why we would need to manually verify the keep alive time? Wouldn't that be a QUIC issue too? If there aren't any responses to those, QUIC would just drop that path.

In any case, my point is that QUIC should handle these cases, and if it doesn't, then that's an issue worth investigating in picoquic or our specific use of that library. Perhaps we can share qlog files from both client and server to actually verify the described behavior in these kind of issues. I extensively tested slipstream on emulated networks usjng tc, so I'm quite confident in its reliability.

[Mygod]

Hi @EndPositive. :)

The four (remaining) issues I wrote above are really just hypotheses at this point. (I was being lazy and copying AI's output as a note for myself later.) As you can see, the "evidence" given is just some code that these models read and believed. It might very well be the case you described that QUIC actually does handle these issues under the hood.

With that said, I did identify a real issue (#45). It appears that QUIC multiplexing does not handle heterogeneous streams very well, and this seems to be by specs and not a bug of a specific library. Basically, if one stream stalls and eats up all the max_data budget, then that stream would actually block the entire connection and no new streams can be established.

[EndPositive]

So if I understand correctly, you're describing an issue where a single stream within a connection can't progress.

Let's think about the server side: I feel the only way for that to happen is for the upstream TCP connection to be stalled. Since QUIC is multiplexed on a connection, it will always continue receiving data as long as there's a healthy path. The main issue here is that if the TCP connection stalls, we have no way to tell QUIC that we're not ready to receive more data. In other words, we have no way to inform QUIC to reject further packets and decrease the window. QUIC will always just give us more data as long as the client is sending data. See this TODO here: https://github.com/EndPositive/slipstream/blob/a24450bfc6db0964945125a9f9efc394e48fb319/src/slipstream_server.c#L557

An alternative underlying issue could be that a single stream is having frame gaps. In that case, QUIC will buffer the data and wait for the gap to be filled. I'm not sure what if there are any limits here in buffer size, but one thing is sure: if the buffer keeps growing, that indicates the client is alive and sending data and is capable of filling the gap eventually.

I'm not exactly sure what the max_data param you mention is, but if it is the same as QUIC's MAX_STREAM_DATA and MAX_DATA frames, then I'm super confused. Those frames indicate total data limits, and not buffer limits? Not sure how that relates to stalled streams

[EndPositive]

From my experience last year, it's extremely difficult to make any LLM reason properly about QUIC and even worse picoquic. Picoquic's codebase is massive and changing quite regularly..

In general the concept of slipstream was to fully leverage QUIC, and only implement DNS encoding, rather than building some extra reliability features. I still believe that should be possible considering how feature-full QUIC is.