Add diagnostics framework — squad doctor, blame attribution, troubleshooting

[diberry]

Problem

When Squad misbehaves, it's extremely difficult to determine where the problem originates. There are at least 4 distinct failure layers, and symptoms often look identical across them:

Layer	Example Failure	What It Looks Like
Squad governance (squad.agent.md)	Bad routing rule, missing spawn template field, stale ceremony config	Agent does wrong thing or skips steps
Copilot CLI / platform	Tool unavailable, background agent silent success, context overflow	Agent appears to do nothing, partial results
LLM behavior	Model ignores instructions, hallucinates file paths, skips archival step	Agent does work but incorrectly or incompletely
User .squad/ state	Bloated decisions.md, corrupt registry.json, stale history, missing charter	Agents slow, confused, or fail to spawn

Why This Matters

Users (and the coordinator itself) currently have no systematic way to diagnose issues. When something goes wrong, the debugging process is:

1. User notices bad behavior
2. User guesses which layer caused it
3. User manually inspects files, logs, or re-runs with different prompts
4. Repeat until fixed (or give up)

This is especially painful because:

• LLM non-determinism means the same input can succeed or fail across runs
• Silent success bugs (~7-10% of background spawns) mean agents complete file writes but return no text
• Context overflow after multi-agent fan-out causes server error retry loops
• Bloated state files (decisions.md at 145KB) cause subtle degradation, not hard failures

Proposed: Diagnostic Framework

1. squad doctor CLI Command

A health check command that inspects .squad/ state and reports issues:

npx squad doctor

Checks:

• .squad/team.md exists and has ## Members section with entries
• All agents in roster have matching agents/{name}/charter.md
• All agents in roster have matching agents/{name}/history.md
• decisions.md size check (warn >20KB, error >50KB)
• decisions-archive.md exists if decisions.md has been archived before
• Each history.md size check (warn >12KB, error >30KB)
• casting/registry.json is valid JSON with all roster agents
• casting/policy.json exists and is valid
• decisions/inbox/ is empty (non-empty = Scribe didn't merge)
• config.json is valid JSON (if exists)
• No orphaned agent directories (in agents/ but not in roster)
• No missing agent directories (in roster but not in agents/)
• routing.md references only agents that exist in roster
• .gitattributes has union merge drivers for append-only files
• Total .squad/ directory size (warn >1MB, error >5MB)

Output:

🏥 Squad Doctor — Health Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Team roster: 6 members found
✅ Agent directories: all present
⚠️  decisions.md: 45KB (recommended: <20KB) — run "squad nap" to archive
✅ History files: all under 12KB
❌ decisions/inbox: 3 unmerged files — Scribe may have failed
✅ Casting state: valid
✅ .gitattributes: merge drivers configured
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Result: 1 error, 1 warning — run "squad fix" to auto-repair

2. squad doctor --layer Focused Diagnostics

Target a specific layer:

npx squad doctor --layer state     # Check .squad/ file health
npx squad doctor --layer governance # Validate squad.agent.md structure
npx squad doctor --layer platform   # Check tool availability, model access
npx squad doctor --layer all        # Full diagnostic (default)

3. squad fix Auto-Repair

For issues that have safe, deterministic fixes:

npx squad fix

Auto-fixable:

• Empty decisions/inbox/ by running Scribe merge
• Archive decisions.md entries >30 days old
• Summarize history.md files >12KB
• Create missing agent directories from roster
• Initialize missing casting/ state from existing agents
• Add missing .gitattributes merge drivers

NOT auto-fixable (report only):

• Corrupt JSON files (need manual inspection)
• Roster/routing mismatches (need human decision)
• LLM behavioral issues (need prompt tuning)

4. Diagnostic Logging in Coordinator

Add structured diagnostic breadcrumbs to the orchestration log:

## Diagnostic Context
- **decisions.md size:** 23KB (⚠️ over 20KB threshold)
- **Spawn model:** claude-sonnet-4.5 (Layer 3: task-aware)
- **Agent charter:** 2.1KB ✅
- **Agent history:** 8.4KB ✅
- **Estimated spawn context:** ~18K tokens
- **Background mode:** yes
- **MCP tools available:** github ✅, azure ❌

This gives post-mortem investigators a snapshot of the state at spawn time.

5. Blame Attribution Heuristics

When something goes wrong, the coordinator (or a diagnostic agent) can use these signals:

Symptom	Likely Layer	Diagnostic
Agent does nothing, no files written	Platform	Silent success bug. Check read_agent status.
Agent writes files but returns no text	Platform	Known ~7-10% bug. Files verify success.
Agent does wrong task or skips steps	Governance	Check charter.md, spawn prompt, routing.md
Agent hallucinates file paths	LLM	Model-specific. Try different model.
Agent is slow or runs out of context	State	Check decisions.md + history.md sizes
Scribe skips archival	LLM + State	Haiku model + large file = skipped steps
Agent reads stale decisions	State	Scribe merge didn't run; check inbox/
Ceremony doesn't trigger	Governance	Check ceremonies.md conditions
Wrong agent gets the work	Governance	Check routing.md rules
read_agent returns server error	Platform	Context overflow after fan-out

6. User-Facing Troubleshooting Guide

A doc page or skill that helps users self-diagnose:

## Quick Troubleshooting

### "My agent did nothing"
1. Run `squad doctor` — check for state issues
2. Check orchestration log for the spawn entry
3. Was it a background spawn? Check `read_agent` result
4. Try re-running with `--verbose` logging

### "Decisions keep growing"  
1. Run `squad doctor --layer state` — check decisions.md size
2. Run `squad nap` to archive old decisions
3. Check if Scribe is being spawned after work batches
4. Add a "Decisions Cleanup" ceremony for periodic maintenance

### "Agent gave wrong answer"
1. Check the agent's charter — does it cover this domain?
2. Check routing.md — was the right agent selected?
3. Check decisions.md — are there conflicting decisions?
4. Try a different model (bump to sonnet/opus for complex tasks)

Implementation Plan

1. Phase 1: squad doctor command — state health checks (most value, least risk)
2. Phase 2: Diagnostic logging — add context breadcrumbs to orchestration log entries
3. Phase 3: squad fix command — auto-repair for safe issues
4. Phase 4: Blame heuristics — teach the coordinator to self-diagnose
5. Phase 5: Troubleshooting guide — user-facing docs

Labels

squad, enhancement, squad:lead

[github-actions]

🏗️ Squad Triage — Flight (Lead)

Issue: #21 — Add diagnostics framework — squad doctor, blame attribution, troubleshooting
Assigned to: FIDO (Quality Owner)
Reason: Issue relates to testing/quality work

@copilot evaluation: 🔴 Not suitable — issue involves work outside the coding agent's capability profile.

Team roster:

• Flight (Lead) → label: squad:flight
• Procedures (Prompt Engineer) → label: squad:procedures
• EECOM (Core Dev) → label: squad:eecom
• FIDO (Quality Owner) → label: squad:fido
• PAO (DevRel) → label: squad:pao
• CAPCOM (SDK Expert) → label: squad:capcom
• CONTROL (TypeScript Engineer) → label: squad:control
• Surgeon (Release Manager) → label: squad:surgeon
• Booster (CI/CD Engineer) → label: squad:booster
• GNC (Node.js Runtime) → label: squad:gnc
• Network (Distribution) → label: squad:network
• RETRO (Security) → label: squad:retro
• INCO (CLI UX & Visual Design) → label: squad:inco
• GUIDO (VS Code Extension) → label: squad:guido
• Telemetry (Aspire & Observability) → label: squad:telemetry
• VOX (REPL & Interactive Shell) → label: squad:vox
• DSKY (TUI Engineer) → label: squad:dsky
• Sims (E2E Test Engineer) → label: squad:sims
• Handbook (SDK Usability) → label: squad:handbook
• Ralph (Work Monitor) → label: squad:ralph
• @copilot (Coding Agent) → label: squad:copilot

To reassign, remove the current squad:* label and add the correct one.

[diberry]

🏗️ Full Team Review — 5 Agents, Synthesized Findings

Reviewers: Flight (Lead), EECOM (Core Dev), FIDO (Quality Owner), Procedures (Prompt Engineer), Telemetry (Observability)

---

🔴 Critical Finding: squad doctor Already Exists

All 5 reviewers independently flagged this. packages/squad-cli/src/cli/commands/doctor.ts (16,899 bytes) already ships with 14+ health checks. The issue proposes building what's partially built. The PRD must be scoped as "extend existing doctor" not "create new command."

Existing checks: .squad/ dir, config.json, team.md + ## Members, routing.md, agents/ dir + count, casting/registry.json, decisions.md existence, rate-limit-status.json, squad.agent.md, Node ≥22.5, ESM compat, copilot-sdk patch status.

Gap (what's NOT in existing doctor): file size thresholds, roster↔directory cross-check, .gitattributes merge drivers, decisions/inbox/ non-empty check, total .squad/ size, --layer flag, worktree awareness.

🔴 Critical Finding: squad fix Should NOT Be a New Command

Flight + EECOM: squad nap already handles 4/6 proposed repair actions (history compression, log pruning, inbox merging, decision archival). Adding squad fix creates duplicate UX. Extend squad nap --repair instead.

🔴 Critical Finding: OTel Stack Completely Ignored

Telemetry: The issue proposes unstructured markdown breadcrumbs while a production-grade OTel stack exists (otel.ts, otel-metrics.ts, otel-bridge.ts, HealthMonitor, 20+ existing metrics, squad aspire command). Diagnostic breadcrumbs should be OTel span attributes, not markdown. Proposed 11 sub-issues for proper instrumentation.

🟡 Threshold Numbers Are Wrong

• Issue says history.md warn at 12KB → code uses 15KB (HISTORY_THRESHOLD = 15 * 1024)
• Issue says decisions.md at 145KB → this repo is at 345KB (7× proposed error threshold)
• The decisions-archive.md is 233KB with no size bound

🟡 Blame Heuristics = False Confidence

Flight + FIDO + Procedures + Telemetry (4/5 reviewers): The symptom→layer table has multi-layer ambiguity. "Agent does nothing" maps to Platform OR Governance OR malformed spawn prompt. Single-symptom→single-layer blame will mislead users. Reframe as "investigation checklist" not "root cause." Procedures proposes a 3-signal blame triage decision tree instead.

🟡 squad fix with LLM Summarization = Data Loss Risk

FIDO: "Summarize history.md files >12KB" spawns an LLM. Under degraded conditions (the very thing being diagnosed), this could corrupt content. Deterministic operations only in fix/repair. Needs --dry-run flag and rollback.

---

Recommended Revised Phase Plan

Phase	What	Owner	Priority
1	Extend existing squad doctor — add size checks, roster cross-check, inbox check, .gitattributes, --layer flag, worktree mode	EECOM + FIDO	P1
2	Troubleshooting guide (docs) — 4-layer model, investigation checklists (not blame tables)	Procedures + PAO	P1
3	Coordinator diagnostic breadcrumbs as OTel span attributes — captureSpawnContext(), not markdown	EECOM + Telemetry	P1
4	Extend squad nap --repair — create missing agent dirs, init missing casting state, add .gitattributes	EECOM + FIDO	P2
5	squad.decisions.* OTel metrics — size gauge, inbox depth, archive runs counter	Telemetry	P2
6	Blame triage in coordinator retry path — 3-signal decision tree, not static table	Procedures	P3

Key Additions to PRD

• Non-goals: LLM behavioral diagnosis, live MCP/model checks without session, modifying files in doctor (read-only)
• User stories: Developer debugging → one command with fix suggestions; post-mortem → spawn context in OTel; new user → setup validation
• Success metrics: doctor <500ms on 20-agent repo; >80% of fail checks output fix command; 100% spawns include context snapshot within 30 days
• Dependencies: Worktree support, StorageProvider abstraction, existing nap.ts contract
• Risks: False confidence from all-green doctor (LLM/platform layers not diagnosable statically); threshold drift between code and docs