Skip to content

Conversation

@earakely-scale
Copy link

I ran the MCP atlas eval like so:

uv run python mcp_completion_script.py \
  --model "openai/gpt-4o" \
  --input_huggingface "bhertz/mcp_dummy" \  # TODO switch to dataset on ScaleAI
  --output "mcp_eval_4o_results.csv"

and experienced many MCP read timeouts https://pastebin.com/z6MH1K1E .

The current behavior does a 10s delay prior to retry causing possible thundering herd behavior. This can be partially mitigated using exponential backoff + jitter. I also increased the read timeout from 30s to 120s since I still saw some MCP tools consistently timeout (e.g. OSM); this is likely due to these tools simply having a larger response time.

After this change I found that nearly all read timeouts were mitigated; less than a handful occurring afterwards. I still think there is room to improve by having a granular tool-dependent timeout (e.g. some tools take longer than others and thus should have larger timeouts).

@geobio geobio closed this Dec 5, 2025
@geobio geobio deleted the earakely_retry_patch branch December 18, 2025 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants