content: Building Resilient Agents: Circuit Breaker Pattern in Squad

[diberry]

Content Outline

Type: Technical deep-dive
Source PRs: bradygaster#552
Target audience: Agent builders, SRE teams
Estimated effort: quick-win

Outline

1. Hook / Problem Statement

Long-running agents face transient failures: rate limits, network blips, upstream service degradation. Without resilience patterns, an agent can waste cycles retrying doomed operations or get stuck in error loops. Squad's circuit breaker pattern lets agents gracefully degrade and recover.

2. Key Sections

• Why resilience matters for agents: rate limits, retries, cascading failures
• Circuit breaker state machine: CLOSED (normal) → OPEN (fail-fast) → HALF-OPEN (testing) → CLOSED (recovered)
• Exponential backoff strategy: 2m → 4m → 8m → 30m cap (preventing thundering herd)
• Persistent state across restarts: remembering circuit state after crashes
• Configuration: thresholds, backoff parameters, state storage
• How to apply the pattern to your own custom agents

3. Code/Config Examples Needed

• Circuit breaker configuration (thresholds, backoff params)
• Wrapping an agent function with circuit breaker logic
• Exponential backoff calculation
• Custom state storage implementation
• Example: email-sending agent with circuit breaker

4. Visual Assets

• State machine diagram (CLOSED → OPEN → HALF-OPEN → CLOSED)
• Timeline diagram: failures, exponential backoff windows, recovery
• Configuration file example annotated

5. Call to Action

Engineers building Squad agents should understand and implement the circuit breaker pattern to make their agents production-ready and resilient to transient failures.

Notes

• All content drafted on diberry/squad first
• Nothing touches bradygaster/squad until reviewed and approved

[diberry]

📦 Work Item 1: MVP (Minimum to Ship)

Format: Technical blog post (1000-1500 words)
Scope:

• Problem: long-running agents hitting rate limits (GitHub 429s)
• Circuit breaker state machine diagram (text-based: CLOSED → OPEN → HALF-OPEN → CLOSED)
• How Ralph uses it: exponential backoff (2m → 4m → 8m → 30m cap)
• Config reference: how to enable/tune thresholds

NOT included in MVP: Custom circuit breaker implementations, non-GitHub API patterns, persistent state deep-dive

---

🔮 Work Item 2: Proposed/Later Content

• "Build Your Own Circuit Breaker" tutorial for custom API integrations
• Persistent state deep-dive: how state survives restarts
• Monitoring: exposing circuit breaker metrics to Grafana/OTel
• Pattern library: circuit breaker + retry + bulkhead for agent resilience
• Comparison: Squad's approach vs Netflix Hystrix vs Polly

[diberry]

📝 MVP Draft — Building Resilient Agents: Circuit Breaker Pattern

Problem: Rate Limits Break Long-Running Agents

Long-running agents like Ralph (Squad's watch command) face a common challenge: GitHub's API enforces rate limits — typically 5,000 requests per hour for authenticated users. When an agent hits this ceiling, it receives a 429 Too Many Requests response and must back off.

But here's the catch: if an agent retries immediately after hitting a 429, it wastes its entire budget hammering a door that's temporarily closed. Worse, if multiple agents are running concurrently and one hits the limit, cascading failures can take down the whole team.

The question: How do we prevent agents from wasting cycles on doomed requests?

The answer: A predictive circuit breaker that opens BEFORE the limit is reached, not after.

---

What Is a Circuit Breaker?

A circuit breaker is a state machine that protects systems from cascading failures. Think of it like an electrical circuit breaker in your home: when current spikes dangerously, it trips and cuts power before damage occurs.

In Squad, the circuit breaker monitors GitHub API quota in real time. When it predicts that quota will be exhausted soon (based on the observed consumption rate), it proactively stops requests rather than letting them fail.

The State Machine

The circuit breaker has three states:

CLOSED (normal)
   ↓
   When: quota predicted to run out (or at critical level)
   ↓
OPEN (circuit broken)
   ↓
   When: cooldown expires
   ↓
HALF-OPEN (testing recovery)
   ↓
   If: 2 consecutive successful rounds → CLOSED (recovered)
   If: 429 error → OPEN (circuit opens again)

CLOSED: Everything is normal. Ralph polls for work and processes issues/PRs without throttling.

OPEN: The circuit detected a problem (low quota or 429 error). Ralph stops polling and waits. This is where the exponential backoff kicks in.

HALF-OPEN: After the cooldown expires, Ralph tries one more polling cycle to see if the quota window has reset. If it succeeds twice in a row, the circuit closes and resumes normal operation.

---

How Ralph Uses Exponential Backoff

When the circuit opens, Ralph doesn't resume immediately when the cooldown expires. Instead, it uses exponential backoff to prevent a "thundering herd" problem — where all queued agents resume at exactly the same moment and immediately hit the limit again.

Backoff Schedule

Ralph starts with a 2-minute cooldown and doubles it each time the circuit re-opens within the same "session":

1st failure  → 2m cooldown
2nd failure  → 4m cooldown
3rd failure  → 8m cooldown
4th failure  → 16m cooldown
...
Cap at 30m

This progression gives the API quota window time to reset, and staggers the load when multiple agents come back online.

Why 30m Cap?

GitHub's rate limit window is 1 hour. A 30-minute cap ensures that even in the worst case (circuit opens and never recovers), Ralph will attempt to resume at least once per hour. Any longer, and Ralph could appear completely stuck to the user.

Persistent State

Ralph's circuit breaker state is persisted to disk in .squad/ralph-circuit-breaker.json. This means:

• If Ralph crashes or is restarted, it remembers which state it was in
• The cooldown timer resumes from where it left off
• The agent doesn't lose context about whether it's in backoff or ready to resume

Example state file:

{
  "status": "open",
  "openedAt": "2025-08-15T14:23:45.000Z",
  "cooldownMinutes": 8,
  "consecutiveFailures": 3,
  "consecutiveSuccesses": 0,
  "lastRateLimitRemaining": 42,
  "lastRateLimitTotal": 5000
}

---

How Ralph Detects Rate Limit Pressure

Ralph uses predictive circuit breaker logic: it doesn't wait for a 429 error. Instead, it samples the rate limit headers on every API call and calculates how fast quota is being consumed.

The Prediction Algorithm

Ralph tracks the last 10 rate limit samples. For each sample, it records:

• Timestamp (milliseconds since epoch)
• Remaining quota
• Total quota

Using the oldest and newest samples, Ralph calculates:

elapsed time = now - first_sample_time
quota consumed = first_sample.remaining - last_sample.remaining
consumption rate = quota_consumed / elapsed_time

seconds_until_exhausted = last_sample.remaining / consumption_rate

If seconds_until_exhausted < 120 (the warning threshold), the circuit opens proactively.

Example

Imagine Ralph samples quota over 30 seconds:

• First sample: 4,900 remaining, 5,000 total
• Last sample: 4,850 remaining, 5,000 total
• Consumed: 50 requests in 30 seconds = 1.67 req/sec
• At this rate: 4,850 / 1.67 = ~2,900 seconds (48 minutes) until exhaustion

48 minutes > 120 seconds, so the circuit stays closed. Ralph continues polling.

But if quota is being consumed faster:

• First sample: 4,900 remaining
• Last sample: 4,100 remaining (after a burst of work)
• Consumed: 800 in 30 seconds = 26.7 req/sec
• At this rate: 4,100 / 26.7 = ~153 seconds until exhaustion

153 seconds > 120, still OK — but close. Any faster and the circuit opens.

---

Enabling and Configuring the Circuit Breaker

In Squad, the circuit breaker is enabled by default for Ralph. There's nothing you need to turn on.

Configuration Reference

Ralph exposes the circuit breaker configuration through the watch command:

squad watch --interval 5

The circuit breaker parameters are hardcoded but can be customized by editing Ralph's source. Here are the key knobs:

Parameter	Default	Meaning
maxSamples	10	Number of rate limit samples to keep in memory before dropping old ones
warningThresholdSeconds	120	If predicted time-to-exhaustion falls below this (in seconds), circuit opens
initialCooldownMinutes	2	Starting backoff window after circuit opens
maxCooldownMinutes	30	Cap on backoff (exponential growth stops here)
successesUntilClosed	2	How many consecutive successful rounds in HALF-OPEN state before returning to CLOSED

Persistence Configuration

Circuit state is saved to .squad/ralph-circuit-breaker.json. You can inspect this file to understand Ralph's current state.

To reset the circuit breaker manually:

rm .squad/ralph-circuit-breaker.json

Ralph will create a fresh state file on the next run.

Monitoring the Circuit Breaker

Ralph prints circuit breaker events to the console:

🛑 Circuit opened — quota predicted low (342/5000)
⏸ Circuit open — cooling down (480s left)
🔄 Circuit half-open — probing...
✓ Circuit closed — quota recovered

Watch for these signals to understand when Ralph is backing off and why.

---

State Transitions in Practice

Here's a realistic scenario:

1. Ralph starts (CLOSED)

[14:00:00] ✓ Ralph started — Round 1
[14:00:05] Triaged 3 issues, 0 PRs need review

2. Quota consumption accelerates (still CLOSED, but quota sampled)

[14:05:00] Round 2 — 3 issues, 4 PRs reviewed (rate: 20 req/sec)
[14:10:00] Round 3 — quota now at 3,200 remaining, rate: 30 req/sec

3. Prediction triggers (OPEN)

[14:15:00] 🛑 Circuit opened — quota predicted low (2,100/5000)
           At current rate, quota exhausted in ~70 seconds
[14:15:00] ⏸ Circuit open — cooling down (120s left)

4. Cooldown expires (transition to HALF-OPEN)

[14:17:00] 🔄 Circuit half-open — probing...
[14:17:05] ✓ Round 4 — check succeeded (quota now 3,100)
[14:17:05] Triage: 1 issue, assign PR reviewer

5. Second successful round (transition to CLOSED)

[14:22:00] ✓ Round 5 — check succeeded (quota now 3,050)
[14:22:00] ✓ Circuit closed — quota recovered

6. Resume normal polling

[14:27:00] Round 6 — 2 issues, 1 PR needs review

---

Why This Matters for Your Agents

If you build custom agents in Squad, the circuit breaker pattern applies to any service with rate limits — not just GitHub. The principles are the same:

1. Monitor headroom proactively, not reactively
2. Open early before failures occur
3. Back off exponentially to avoid thundering herd
4. Remember state across restarts
5. Test recovery before resuming full capacity (HALF-OPEN state)

Ralph demonstrates this pattern in production. Use it as a reference when adding rate limit resilience to your own agents.

---

What's Next?

The MVP covers the essentials: why circuit breakers matter, how Squad's state machine works, and what exponential backoff looks like in practice.

Future posts will dive deeper into:

• Building your own circuit breaker for custom APIs (not just GitHub)
• Persistent state strategies: storing and recovering circuit state across crashes
• Observability: exposing circuit breaker metrics to monitoring systems
• Pattern combinations: circuit breaker + retry + bulkhead for bulletproof resilience

For now, know that Ralph has your back. The next time you hit GitHub's rate limit, the circuit breaker will gracefully step back, cooldown, and resume — all without dropping a single issue from the board.

---

Draft by PAO · MVP scope only · Ready for review

[diberry]

🏗️ Flight Review — Circuit Breaker

Verdict: ⚠️ Needs edits

Accuracy

⚠️ State machine differs from implementation:

• Draft describes a predictive quota-sampling circuit breaker that opens at <2 min remaining
• Source code (tests + SDK) confirms predictive logic but uses traffic light concept (green/amber/red) based on percentage thresholds (>20% = green, >5% = amber, ≤5% = red)
• Draft mentions "sampling quota over 30 seconds" but doesn't align with actual getTrafficLight() implementation

⚠️ Circuit breaker states terminology differs:

• Draft: CLOSED → OPEN → HALF-OPEN → CLOSED
• Source code confirms these state names are correct ✓
• But draft example shows specific backoff values (2m → 4m → 8m → 16m) that don't match test defaults (test defaults show cooldownMinutes: 2 starting)
• Draft claims "default: 10 minutes" cooldown; source shows cooldownMinutes: 2 in tests

✓ Correct aspects:

• Exponential backoff concept ✓
• State persistence to .squad/ralph-circuit-breaker.json ✓
• 30-minute cap is mentioned in source comments ✓
• 1-hour GitHub rate limit window ✓
• Prediction algorithm concept (first-to-last delta, not regression) is accurate per source

⚠️ Prediction algorithm description lacks precision:

• Draft uses 30-second example but source samples up to 10 (maxSamples: 10)
• Draft example shows "4,100 / 26.7 req/sec" calculation; this math is right but needs to clarify it's (remaining / consumption_rate), not (quota / rate)

Scope

✅ MVP scope is appropriate:

• Circuit breaker states ✓
• Predictive quota checking ✓
• Exponential backoff ✓
• Persistence ✓
• Monitoring console output ✓

Not included (future):

• Multi-provider fallbacks (mentioned as coming later, correct)
• Custom circuit breaker parameters (mentioned as future, but source shows these ARE configurable—this is a scope decision)
• Observability/metrics integration (correct for MVP)

Minor gap: Draft claims this is Ralph-specific, but source shows PredictiveCircuitBreaker is a reusable SDK class meant for any rate-limited API. The draft could mention this.

Clarity & Voice

✅ Opening hook ("rate limits break long-running agents") is strong.
✅ State machine diagram is clear and helpful.
✅ Backoff schedule table is easy to scan.
⚠️ Example calculations are detailed but could be slightly clearer with units/labels.

Overall voice is technical but accessible. Good for Squad users who understand resilience patterns.

Completeness

⚠️ Missing or incomplete:

1. Configuration reference table values don't match source code:

   • maxSamples: 10 ✓ correct
   • warningThresholdSeconds: 120 ✓ correct
   • initialCooldownMinutes: 2 ← source, draft shows 2m start then 4m, 8m, 16m (is this from example, or is draft proposing different defaults?)
   • maxCooldownMinutes: 30 ✓ correct
   • successesUntilClosed: 2 ✓ correct (confirmed in test)

2. Draft doesn't explain traffic light concept (green/amber/red) at all. Source shows this is how quota headroom is evaluated. This is important for users to understand.

3. Console output examples (🛑, ⏸, 🔄, ✓) are helpful but don't show the actual console output from tests/source.

4. "State Transitions in Practice" section is excellent but timestamps don't align with actual cooldown defaults (test shows 2m, not 2m → 4m → 8m progression).

Specific Edits Needed

1. Clarify initial cooldown: Confirm the initial cooldown is 2 minutes (not starting at higher), and explain the doubling happens each time the circuit re-opens within the same session (not per round).

2. Add traffic light concept: Insert a new section explaining how Ralph determines rate pressure:

   ### Traffic Light System
   Ralph doesn't wait for a 429 error. Instead, it evaluates API quota as a percentage:
   - GREEN (>20% remaining): Proceed normally
   - AMBER (5–20% remaining): High-priority agents only
   - RED (≤5% remaining): Circuit opens proactively
   

3. Fix example timeline: In "State Transitions in Practice", adjust the times and cooldown progression to match actual implementation (2m initial, then doubling).

4. Clarify prediction algorithm:

   • Explain that rate limit samples are tracked in a rolling window (up to 10 samples)
   • Update the example to show fewer samples (or at least be consistent)
   • Add: "Consumption rate = (first_remaining - last_remaining) / (last_timestamp - first_timestamp)"

5. Add: "Ralph exposes circuit breaker configuration" is slightly misleading. Clarify that most parameters are hardcoded; users can reset state manually with rm .squad/ralph-circuit-breaker.json, but don't configure thresholds via CLI.

6. Console output: Verify these match actual output or adjust to realistic examples.

---

Reviewed by Flight

[diberry]

📝 MVP Draft v2 — Building Resilient Agents: Circuit Breaker Pattern (Revised)

Problem: Rate Limits Break Long-Running Agents

Long-running agents like Ralph (Squad's watch command) face a common challenge: GitHub's API enforces rate limits — typically 5,000 requests per hour for authenticated users. When an agent hits this ceiling, it receives a \429 Too Many Requests\ response and must back off.

But here's the catch: if an agent retries immediately after hitting a 429, it wastes its entire budget hammering a door that's temporarily closed. Worse, if multiple agents are running concurrently and one hits the limit, cascading failures can take down the whole team.

The question: How do we prevent agents from wasting cycles on doomed requests?

The answer: A predictive circuit breaker that opens BEFORE the limit is reached, not after.

---

What Is a Circuit Breaker?

A circuit breaker is a state machine that protects systems from cascading failures. Think of it like an electrical circuit breaker in your home: when current spikes dangerously, it trips and cuts power before damage occurs.

In Squad, the circuit breaker monitors GitHub API quota in real time. When it predicts that quota will be exhausted soon (based on the observed consumption rate), it proactively stops requests rather than letting them fail.

The State Machine

The circuit breaker has three states:

\
CLOSED (normal)
↓
When: quota predicted to run out (or at critical level)
↓
OPEN (circuit broken)
↓
When: cooldown expires
↓
HALF-OPEN (testing recovery)
↓
If: 2 consecutive successful rounds → CLOSED (recovered)
If: 429 error → OPEN (circuit opens again)
\\

CLOSED: Everything is normal. Ralph polls for work and processes issues/PRs without throttling.

OPEN: The circuit detected a problem (low quota or 429 error). Ralph stops polling and waits. This is where the exponential backoff kicks in.

HALF-OPEN: After the cooldown expires, Ralph tries one more polling cycle to see if the quota window has reset. If it succeeds twice in a row, the circuit closes and resumes normal operation.

---

Traffic Light System

Ralph doesn't wait for a 429 error. Instead, it evaluates API quota as a percentage of remaining capacity:

• GREEN (>20% remaining): Proceed normally
• AMBER (5–20% remaining): High-priority agents only
• RED (≤5% remaining): Circuit opens proactively

This traffic light system allows Ralph to be aggressive at green but cautious at amber, and to stop completely at red — all before hitting the actual rate limit.

---

How Ralph Uses Exponential Backoff

When the circuit opens, Ralph doesn't resume immediately when the cooldown expires. Instead, it uses exponential backoff to prevent a "thundering herd" problem — where all queued agents resume at exactly the same moment and immediately hit the limit again.

Backoff Schedule

Ralph starts with a 2-minute cooldown and doubles it each time the circuit re-opens within the same session:

\
1st failure → 2m cooldown
2nd failure → 4m cooldown
3rd failure → 8m cooldown
4th failure → 16m cooldown
...
Cap at 30m
\\

This progression gives the API quota window time to reset, and staggers the load when multiple agents come back online. The initial 2-minute cooldown is the starting point; it only doubles if the circuit re-opens again before recovering.

Why 30m Cap?

GitHub's rate limit window is 1 hour. A 30-minute cap ensures that even in the worst case (circuit opens and never recovers), Ralph will attempt to resume at least once per hour. Any longer, and Ralph could appear completely stuck to the user.

Persistent State

Ralph's circuit breaker state is persisted to disk in .squad/ralph-circuit-breaker.json. This means:

• If Ralph crashes or is restarted, it remembers which state it was in
• The cooldown timer resumes from where it left off
• The agent doesn't lose context about whether it's in backoff or ready to resume

Example state file:

\\json
{
"status": "open",
"openedAt": "2025-08-15T14:23:45.000Z",
"cooldownMinutes": 8,
"consecutiveFailures": 3,
"consecutiveSuccesses": 0,
"lastRateLimitRemaining": 42,
"lastRateLimitTotal": 5000
}
\\

---

How Ralph Detects Rate Limit Pressure

Ralph uses predictive circuit breaker logic: it doesn't wait for a 429 error. Instead, it samples the rate limit headers on every API call and calculates how fast quota is being consumed.

The Prediction Algorithm

Ralph tracks the last 10 rate limit samples in a rolling window. For each sample, it records:

• Timestamp (milliseconds since epoch)
• Remaining quota
• Total quota

Using the oldest and newest samples, Ralph calculates the consumption rate:

\
elapsed time = last_timestamp - first_timestamp
quota consumed = first_remaining - last_remaining
consumption rate = quota_consumed / elapsed_time

seconds_until_exhausted = last_remaining / consumption_rate
\\

If \seconds_until_exhausted < 120\ (the warning threshold), the circuit opens proactively.

Example

Imagine Ralph samples quota over 30 seconds:

• First sample: 4,900 remaining, 5,000 total
• Last sample: 4,850 remaining, 5,000 total
• Consumed: 50 requests in 30 seconds = 1.67 req/sec
• At this rate: 4,850 / 1.67 ≈ 2,900 seconds (48 minutes) until exhaustion

48 minutes > 120 seconds, so the circuit stays closed. Ralph continues polling.

But if quota is being consumed faster:

• First sample: 4,900 remaining
• Last sample: 4,100 remaining (after a burst of work)
• Consumed: 800 requests in 30 seconds = 26.7 req/sec
• At this rate: 4,100 / 26.7 ≈ 153 seconds until exhaustion

153 seconds > 120, still OK — but close. If consumption accelerates further and the remaining quota drops, the circuit opens.

---

Enabling and Configuring the Circuit Breaker

In Squad, the circuit breaker is enabled by default for Ralph. There's nothing you need to turn on.

Configuration Reference

Ralph exposes the circuit breaker configuration through the \watch\ command. The circuit breaker parameters are mostly hardcoded but can be reset manually. Here are the key technical parameters:

Parameter	Default	Meaning
\maxSamples\	10	Number of rate limit samples to keep in memory before dropping old ones
\warningThresholdSeconds\	120	If predicted time-to-exhaustion falls below this (in seconds), circuit opens
\initialCooldownMinutes\	2	Starting backoff window after circuit opens
\maxCooldownMinutes\	30	Cap on backoff (exponential growth stops here)
\successesUntilClosed\	2	How many consecutive successful rounds in HALF-OPEN state before returning to CLOSED

Configuration availability: Most of these parameters are hardcoded in Ralph's implementation. Users cannot customize thresholds via CLI flags, but the circuit breaker state can be reset manually.

Resetting the Circuit Breaker

To reset the circuit breaker manually:

\\�ash
rm .squad/ralph-circuit-breaker.json
\\

Ralph will create a fresh state file on the next run, starting with a closed circuit and 2-minute initial cooldown.

Monitoring the Circuit Breaker

Ralph prints circuit breaker events to the console:

\
🛑 Circuit opened — quota predicted low (342/5000)
⏸ Circuit open — cooling down (480s left)
🔄 Circuit half-open — probing...
✓ Circuit closed — quota recovered
\\

Watch for these signals to understand when Ralph is backing off and why.

---

State Transitions in Practice

Here's a realistic scenario:

1. Ralph starts (CLOSED)
\
[14:00:00] ✓ Ralph started — Round 1
[14:00:05] Triaged 3 issues, 0 PRs need review
\\

2. Quota consumption accelerates (still CLOSED, but quota sampled)
\
[14:05:00] Round 2 — 3 issues, 4 PRs reviewed (rate: 20 req/sec)
[14:10:00] Round 3 — quota now at 3,200 remaining, rate: 30 req/sec
\\

3. Prediction triggers (OPEN)
\
[14:15:00] 🛑 Circuit opened — quota predicted low (2,100/5000)
At current rate, quota exhausted in ~70 seconds
[14:15:00] ⏸ Circuit open — cooling down (120s left)
\\

4. 2-minute cooldown expires (transition to HALF-OPEN)
\
[14:17:00] 🔄 Circuit half-open — probing...
[14:17:05] ✓ Round 4 — check succeeded (quota now 3,100)
[14:17:05] Triage: 1 issue, assign PR reviewer
\\

5. Second successful round (transition to CLOSED)
\
[14:22:00] ✓ Round 5 — check succeeded (quota now 3,050)
[14:22:00] ✓ Circuit closed — quota recovered
\\

6. Resume normal polling
\
[14:27:00] Round 6 — 2 issues, 1 PR needs review
\\

---

Why This Matters for Your Agents

If you build custom agents in Squad, the circuit breaker pattern applies to any service with rate limits — not just GitHub. The principles are the same:

1. Monitor headroom proactively, not reactively
2. Open early before failures occur
3. Back off exponentially to avoid thundering herd
4. Remember state across restarts
5. Test recovery before resuming full capacity (HALF-OPEN state)

Ralph demonstrates this pattern in production. Use it as a reference when adding rate limit resilience to your own agents.

---

What's Next?

The MVP covers the essentials: why circuit breakers matter, how Squad's state machine works, what exponential backoff looks like in practice, and the traffic light concept for proactive rate awareness.

Future posts will dive deeper into:

• Building your own circuit breaker for custom APIs (not just GitHub)
• Persistent state strategies: storing and recovering circuit state across crashes
• Observability: exposing circuit breaker metrics to monitoring systems
• Pattern combinations: circuit breaker + retry + bulkhead for bulletproof resilience

For now, know that Ralph has your back. The next time you hit GitHub's rate limit, the circuit breaker will gracefully step back, cooldown, and resume — all without dropping a single issue from the board.

---

Revised by PAO per Flight review · All technical corrections applied

[diberry]

Docs Placement Analysis

Recommended Placement

• Section: Guide
• Page: \guide/building-resilient-agents\ — "Building Resilient Agents: Circuit Breaker Pattern"
• Nav position: After \guide/building-extensions\ (line 33 in
  avigation.ts)
• Cross-links:
  • \eatures/rate-limiting\ (broader resilience context)
  • \concepts/architecture\ (reference for patterns)
  • \scenarios/troubleshooting\ (handling transient failures)

Rationale

This is a patterns & practices deep-dive aimed at agent builders and SREs—exactly the Guide section's audience. The content teaches foundational resilience patterns (circuit breaker state machine + exponential backoff) that Squad agents should implement.

Why Guide, not Features or Scenarios?

• Not Features: Circuit breaker is a pattern agents implement, not a Squad feature you toggle on.
• Not Scenarios: Not "how to set up Squad for X situation" — it's "how to write robust agent code."
• Yes, Guide: Sits alongside \�uild-autonomous-agent.md\ and \�uilding-extensions.md\ — all teach design patterns for Squad users building production systems.

Nav position: After \�uilding-extensions\ — both teach engineering patterns that mature after the basics (tips, samples, personal squad setup).

Next step: Once placed in \guide/building-resilient-agents.md, update
avigation.ts\ to add the nav item at line 34 (after \�uilding-extensions), then wire up cross-links to the feature and scenario pages.

[diberry]

📝 MVP Draft v2 — Building Resilient Agents: Circuit Breaker Pattern (Revised)

Problem: Rate Limits Break Long-Running Agents

Long-running agents like Ralph (Squad's watch command) face a common challenge: GitHub's API enforces rate limits — typically 5,000 requests per hour for authenticated users. When an agent hits this ceiling, it receives a 429 Too Many Requests response and must back off.

But here's the catch: if an agent retries immediately after hitting a 429, it wastes its entire budget hammering a door that's temporarily closed. Worse, if multiple agents are running concurrently and one hits the limit, cascading failures can take down the whole team.

The question: How do we prevent agents from wasting cycles on doomed requests?

The answer: A predictive circuit breaker that opens BEFORE the limit is reached, not after.

---

What Is a Circuit Breaker?

A circuit breaker is a state machine that protects systems from cascading failures. Think of it like an electrical circuit breaker in your home: when current spikes dangerously, it trips and cuts power before damage occurs.

In Squad, the circuit breaker monitors GitHub API quota in real time. When it predicts that quota will be exhausted soon (based on the observed consumption rate), it proactively stops requests rather than letting them fail.

The State Machine

The circuit breaker has three states:

CLOSED (normal)
   ↓
   When: quota predicted to run out (or at critical level)
   ↓
OPEN (circuit broken)
   ↓
   When: cooldown expires
   ↓
HALF-OPEN (testing recovery)
   ↓
   If: 2 consecutive successful rounds → CLOSED (recovered)
   If: 429 error → OPEN (circuit opens again)

CLOSED: Everything is normal. Ralph polls for work and processes issues/PRs without throttling.

OPEN: The circuit detected a problem (low quota or 429 error). Ralph stops polling and waits. This is where the exponential backoff kicks in.

HALF-OPEN: After the cooldown expires, Ralph tries one more polling cycle to see if the quota window has reset. If it succeeds twice in a row, the circuit closes and resumes normal operation.

---

Traffic Light System

Ralph doesn't wait for a 429 error. Instead, it evaluates API quota as a percentage of remaining capacity:

• GREEN (>20% remaining): Proceed normally
• AMBER (5–20% remaining): High-priority agents only
• RED (≤5% remaining): Circuit opens proactively

This traffic light system allows Ralph to be aggressive at green but cautious at amber, and to stop completely at red — all before hitting the actual rate limit.

---

How Ralph Uses Exponential Backoff

When the circuit opens, Ralph doesn't resume immediately when the cooldown expires. Instead, it uses exponential backoff to prevent a "thundering herd" problem — where all queued agents resume at exactly the same moment and immediately hit the limit again.

Backoff Schedule

Ralph starts with a 2-minute cooldown and doubles it each time the circuit re-opens within the same session:

1st failure  → 2m cooldown
2nd failure  → 4m cooldown
3rd failure  → 8m cooldown
4th failure  → 16m cooldown
...
Cap at 30m

This progression gives the API quota window time to reset, and staggers the load when multiple agents come back online. The initial 2-minute cooldown is the starting point; it only doubles if the circuit re-opens again before recovering.

Why 30m Cap?

GitHub's rate limit window is 1 hour. A 30-minute cap ensures that even in the worst case (circuit opens and never recovers), Ralph will attempt to resume at least once per hour. Any longer, and Ralph could appear completely stuck to the user.

Persistent State

Ralph's circuit breaker state is persisted to disk in .squad/ralph-circuit-breaker.json. This means:

• If Ralph crashes or is restarted, it remembers which state it was in
• The cooldown timer resumes from where it left off
• The agent doesn't lose context about whether it's in backoff or ready to resume

Example state file:

{
  "status": "open",
  "openedAt": "2025-08-15T14:23:45.000Z",
  "cooldownMinutes": 8,
  "consecutiveFailures": 3,
  "consecutiveSuccesses": 0,
  "lastRateLimitRemaining": 42,
  "lastRateLimitTotal": 5000
}

---

How Ralph Detects Rate Limit Pressure

Ralph uses predictive circuit breaker logic: it doesn't wait for a 429 error. Instead, it samples the rate limit headers on every API call and calculates how fast quota is being consumed.

The Prediction Algorithm

Ralph tracks the last 10 rate limit samples in a rolling window. For each sample, it records:

• Timestamp (milliseconds since epoch)
• Remaining quota
• Total quota

Using the oldest and newest samples, Ralph calculates the consumption rate:

elapsed time = last_timestamp - first_timestamp
quota consumed = first_remaining - last_remaining
consumption rate = quota_consumed / elapsed_time

seconds_until_exhausted = last_remaining / consumption_rate

If seconds_until_exhausted < 120 (the warning threshold), the circuit opens proactively.

Example

Imagine Ralph samples quota over 30 seconds:

• First sample: 4,900 remaining, 5,000 total
• Last sample: 4,850 remaining, 5,000 total
• Consumed: 50 requests in 30 seconds = 1.67 req/sec
• At this rate: 4,850 / 1.67 ≈ 2,900 seconds (48 minutes) until exhaustion

48 minutes > 120 seconds, so the circuit stays closed. Ralph continues polling.

But if quota is being consumed faster:

• First sample: 4,900 remaining
• Last sample: 4,100 remaining (after a burst of work)
• Consumed: 800 requests in 30 seconds = 26.7 req/sec
• At this rate: 4,100 / 26.7 ≈ 153 seconds until exhaustion

153 seconds > 120, still OK — but close. If consumption accelerates further and the remaining quota drops, the circuit opens.

---

Enabling and Configuring the Circuit Breaker

In Squad, the circuit breaker is enabled by default for Ralph. There's nothing you need to turn on.

Configuration Reference

Ralph exposes the circuit breaker configuration through the watch command. The circuit breaker parameters are mostly hardcoded but can be reset manually. Here are the key technical parameters:

Parameter	Default	Meaning
maxSamples	10	Number of rate limit samples to keep in memory before dropping old ones
warningThresholdSeconds	120	If predicted time-to-exhaustion falls below this (in seconds), circuit opens
initialCooldownMinutes	2	Starting backoff window after circuit opens
maxCooldownMinutes	30	Cap on backoff (exponential growth stops here)
successesUntilClosed	2	How many consecutive successful rounds in HALF-OPEN state before returning to CLOSED

Configuration availability: Most of these parameters are hardcoded in Ralph's implementation. Users cannot customize thresholds via CLI flags, but the circuit breaker state can be reset manually.

Resetting the Circuit Breaker

To reset the circuit breaker manually:

rm .squad/ralph-circuit-breaker.json

Ralph will create a fresh state file on the next run, starting with a closed circuit and 2-minute initial cooldown.

Monitoring the Circuit Breaker

Ralph prints circuit breaker events to the console:

🛑 Circuit opened — quota predicted low (342/5000)
⏸ Circuit open — cooling down (480s left)
🔄 Circuit half-open — probing...
✓ Circuit closed — quota recovered

Watch for these signals to understand when Ralph is backing off and why.

---

State Transitions in Practice

Here's a realistic scenario:

1. Ralph starts (CLOSED)

[14:00:00] ✓ Ralph started — Round 1
[14:00:05] Triaged 3 issues, 0 PRs need review

2. Quota consumption accelerates (still CLOSED, but quota sampled)

[14:05:00] Round 2 — 3 issues, 4 PRs reviewed (rate: 20 req/sec)
[14:10:00] Round 3 — quota now at 3,200 remaining, rate: 30 req/sec

3. Prediction triggers (OPEN)

[14:15:00] 🛑 Circuit opened — quota predicted low (2,100/5000)
           At current rate, quota exhausted in ~70 seconds
[14:15:00] ⏸ Circuit open — cooling down (120s left)

4. 2-minute cooldown expires (transition to HALF-OPEN)

[14:17:00] 🔄 Circuit half-open — probing...
[14:17:05] ✓ Round 4 — check succeeded (quota now 3,100)
[14:17:05] Triage: 1 issue, assign PR reviewer

5. Second successful round (transition to CLOSED)

[14:22:00] ✓ Round 5 — check succeeded (quota now 3,050)
[14:22:00] ✓ Circuit closed — quota recovered

6. Resume normal polling

[14:27:00] Round 6 — 2 issues, 1 PR needs review

---

Why This Matters for Your Agents

If you build custom agents in Squad, the circuit breaker pattern applies to any service with rate limits — not just GitHub. The principles are the same:

1. Monitor headroom proactively, not reactively
2. Open early before failures occur
3. Back off exponentially to avoid thundering herd
4. Remember state across restarts
5. Test recovery before resuming full capacity (HALF-OPEN state)

Ralph demonstrates this pattern in production. Use it as a reference when adding rate limit resilience to your own agents.

---

What's Next?

The MVP covers the essentials: why circuit breakers matter, how Squad's state machine works, what exponential backoff looks like in practice, and the traffic light concept for proactive rate awareness.

Future posts will dive deeper into:

• Building your own circuit breaker for custom APIs (not just GitHub)
• Persistent state strategies: storing and recovering circuit state across crashes
• Observability: exposing circuit breaker metrics to monitoring systems
• Pattern combinations: circuit breaker + retry + bulkhead for bulletproof resilience

For now, know that Ralph has your back. The next time you hit GitHub's rate limit, the circuit breaker will gracefully step back, cooldown, and resume — all without dropping a single issue from the board.

---

Revised by PAO per Flight review · All technical corrections applied

[diberry]

Closing — docs PR shipped as bradygaster#647.