Add retry logic for httpx RemoteProtocolError: Server disconnected without sending a response

[NickBorgers]

Problem

The SPAN Panel integration frequently fails with the error:

Server disconnected without sending a response.

This occurs even when the SPAN panels are fully reachable and responding normally to direct HTTP requests.

Evidence

Integration logs show repeated failures:

ERROR custom_components.span_panel.span_panel_api: Failed to get all panel data: Unexpected error: Server disconnected without sending a response.
WARNING custom_components.span_panel.span_panel: Panel update failed (Unexpected error: Server disconnected without sending a response.); marking offline

But the panels respond fine to direct requests:

$ ping -c 2 10.212.101.111
2 packets transmitted, 2 packets received, 0% packet loss

$ curl -sk -o /dev/null -w '%{http_code}' https://10.212.101.111/api/v1/panel
412  # Expected without auth token

Root Cause

This is a well-documented httpx connection pooling issue:

1. httpx maintains a connection pool for HTTP keep-alive
2. The SPAN panel closes idle connections after its keep-alive timeout expires
3. httpx attempts to reuse a connection that was already closed server-side
4. Result: RemoteProtocolError: Server disconnected without sending a response

This is not a network issue - it's a client-side connection pool race condition.

Impact

When this error occurs repeatedly:

• The integration marks panels as offline despite them being reachable
• Aggressive retries (15-second polling) accumulate failed requests
• This can cascade to affect Home Assistant's event loop responsiveness
• In severe cases, HA's web UI becomes unresponsive

Suggested Fix

Add retry logic specifically for RemoteProtocolError. Other Home Assistant integrations have implemented this:

• Philips TV integration: "We retry once, could be a reused session that was closed."
• Axis camera integration: Added retry on this specific error

Example approach:

from httpcore import RemoteProtocolError

async def fetch_with_retry(self, url, max_retries=1):
    for attempt in range(max_retries + 1):
        try:
            return await self._client.get(url)
        except RemoteProtocolError:
            if attempt == max_retries:
                raise
            # Connection was closed server-side, retry with fresh connection
            await self._client.aclose()
            self._client = self._create_client()

Alternatively, configure httpx to disable connection pooling or reduce keep-alive timeout:

limits = httpx.Limits(max_keepalive_connections=0)
client = httpx.AsyncClient(limits=limits)

Environment

• Home Assistant version: 2025.12.4
• SPAN Panel integration version: latest from HACS
• SPAN Panel firmware: current
• Two SPAN panels configured with 15-second scan interval

References

• httpx discussion: Intermittent RemoteProtocolError
• Home Assistant Denon AVR issue
• Home Assistant Philips TV issue
• BentoML issue describing this as "well-known httpx issue"

[NickBorgers]

Additional Confirming Evidence

I'm experiencing this issue on Home Assistant 2025.12.4. Here's what I found in my logs:

Exact Error Match

ERROR [custom_components.span_panel.span_panel_api] Failed to get all panel data: 
  Unexpected error: Server disconnected without sending a response.
WARNING [custom_components.span_panel.span_panel] Panel update failed (...); marking offline
WARNING [custom_components.span_panel.coordinator] Span Panel API error: ...

Frequency

966 occurrences of "Server disconnected without sending a response" in ~2000 log lines - extremely high failure rate.

Cascade Effect

The logs show the exact cascade pattern described in this issue:

1. Rapid-fire failures - errors every ~20-30 seconds
2. System-wide impact - Supervisor timeouts occurred:

   Timeout on /info request
   Timeout on /core/info request
   Timeout on /supervisor/info request
   

3. Other integrations affected - Hue bridge: 10 Attempts to (re)connect to bridge failed
4. HA restart triggered - s6-rc services restarted shortly after the cascade

Connection Pool Race Condition Evidence

I also observed an asyncio race condition error directly after a SPAN failure:

AttributeError: 'NoneType' object has no attribute 'connection_lost'

During handling of the above exception, another exception occurred:

AttributeError: 'NoneType' object has no attribute 'close'

This appears symptomatic of the httpx connection pooling bug - the socket was already closed server-side when the client tried to clean it up.

This is causing significant instability in my Home Assistant instance.

[NickBorgers]

Additional Evidence: Deterministic Failure, Not Race Condition

I investigated this issue on my Home Assistant instance and found evidence that clarifies the failure mode.

Environment

• 2 SPAN panels (firmware spanos2/r202546/03)
• Home Assistant with span_panel integration
• scan_interval: 15 seconds (default)

Key Finding: Timing Mismatch Creates Guaranteed Failures

The SPAN panels advertise:

Connection: keep-alive
Keep-Alive: timeout=5

But the integration polls every 15 seconds. This means:

1. Request at t=0 → connection added to pool
2. Panel closes connection at t=5 (server-side)
3. Next poll at t=15 → httpx reuses stale connection → guaranteed failure

This isn't a "race condition" with probabilistic failures - it's a deterministic failure because the poll interval (15s) always exceeds the keep-alive timeout (5s).

Evidence from Logs

100% failure rate over 5+ hours:

995 consecutive SPAN errors, ZERO successes

Error pattern shows pairs ~10-12 seconds apart (2 panels failing on each poll cycle):

2025-12-29 18:49:25.897 WARNING ... Panel update failed ... marking offline
2025-12-29 18:49:35.246 ERROR ... Server disconnected without sending a response
2025-12-29 18:50:08.222 ERROR ... Server disconnected without sending a response
2025-12-29 18:50:19.366 ERROR ... Server disconnected without sending a response
[continues for 5+ hours]

Panels ARE Reachable

Direct requests from the same Home Assistant host work perfectly:

# Ping: 0% packet loss, ~2ms latency
$ ping -c 3 10.212.101.111
3 packets transmitted, 3 packets received, 0% packet loss

# Unauthenticated endpoint works
$ wget -q -O- http://10.212.101.111/api/v1/status
{"software":{"firmwareVersion":"spanos2/r202546/03",...}

# Authenticated endpoint works
$ wget --header='Authorization: Bearer <token>' http://10.212.101.111/api/v1/panel
{"mainRelayState":"CLOSED","instantGridPowerW":830.80,...}

Implications for Fix

The proposed fixes in this issue (retry logic or disabling connection pooling) should work. However, understanding this as a deterministic rather than probabilistic failure suggests:

1. Retry once should be sufficient (don't need exponential backoff)
2. Disabling keep-alive (max_keepalive_connections=0) would be the most reliable fix
3. Alternative workaround: Users could set scan_interval < 5 to keep connections alive (though this increases panel load)

The 5-second keep-alive timeout appears to be a SPAN firmware choice, so client-side handling is the appropriate fix.

---

🤖 Generated with Claude Code

[cayossarian]

Hi,

The httpx library should be seeing the FIN/RST from the client, not blindly using the closed connection. I suspect in most cases that the httpx library does see this reply from the server and thus why we don't see other reports of this problem. Your fix in the PR you sent might "fix" the problem but I suspect that you are masking some other local issue.

Having said that it does make sense to catch the exception but do it in the client library, span-panel-api., not in the span integration.

Something like this (around line 609 in client.py after the other exceptions):

        except httpx.RemoteProtocolError:
            # Server closed connection (stale keep-alive) - all pooled connections likely dead
            # Destroy client to force fresh connection pool on retry
            if self._client is not None:
                with suppress(Exception):
                    await self._client.__aexit__(None, None, None)
                self._client = None
            if attempt < max_attempts - 1:
                continue  # Immediate retry - no delay needed
            raise

Take a look and if you agree create a PR there and I will merge it.

[cayossarian]

v1.30b1 has a relevant fix, I'll let it run for a couple days and if no comments, release it noting your contribution. Thanks for pointing that out!

[NickBorgers]

Fantastic, thank you @cayossarian
I just installed the beta version and will share if the connectivity log messages go away and if it seems to prevent my Home Assistant from locking up.

[NickBorgers]

v1.30b1 is working.

Before (v1.29)	After (v1.30b1)
Errors every 30-60 seconds	Zero errors in ~5 minutes
Last error: 20:45:11	Restart: 21:03

I had been seeing very consistent failure logs and am no longer, I'm pretty sure that means my Home Assistant won't lock up anymore. Thanks much!