docs(spec) Phase 3 closeout — retrospective + Phase 4 inherited follow-ups + locked decisions ✅

[jkeeley2073]

Summary

Wave 4 PR 9 — closes Phase 3. Per build-spec scope item 14 + guardrails.md § Per-phase gate. Pure spec/docs/memory closeout — no code changes.

H2 outcome

Baseline run completed 2026-05-07T16:25Z; results JSON committed at data/eval/results/wizard.20260507T162529Z.json:

Metric	Value
citation_precision_mean	0.133
citation_recall_mean	0.133
subagent_accuracy_mean	0.033
refusal_correctness_mean	0.300

AiFoundryOptions.ConfidenceThreshold stays at ADR-0017's draft 0.65 — NOT moved. Calibration deferred to Phase 4 once the upstream gaps are fixed. Decision-log entry DL 2026-05-07 — Phase 3 H2 baseline + threshold-not-moved records the rationale.

Why the metrics are floored — and why that's fine for closing Phase 3

H2 surfaced two upstream gaps that floor every metric:

1. Connected-agents dispatch is non-functional. Wizard.md instructs the Wizard to dispatch to Valuation/Rules/Repair, but FoundryAgentFactory constructs all four agents as standalone AIAgent instances with only getMachineByTitle attached — no actual sub-agent dispatch wiring exists. The Wizard either calls the function tool directly (and answers as itself) OR refuses with the agent's own OutOfScope text. Calibrating against this floor would tune for the gap, not the steady-state.
2. Eval ground-truth OPDB IDs aren't verified against deployed Cosmos. PR 8's subagent curated plausible OPDB-format IDs from machine titles, but the deployed catalog has different actual IDs. When the agent successfully calls getMachineByTitle("Godzilla") it gets the catalog's record (e.g., a Sega 1998 entry instead of the Stern 2021 entry the ground-truth expected); citation_precision / citation_recall score 0 on a structurally-correct lookup.

Both are Phase 4 first-scope items — fix them, re-run --eval, then calibrate. The H2 baseline IS the v1 regression-detection floor as ADR-0016 specified — any Phase 4 number above 0.133/0.133/0.033/0.300 is real improvement.

What ships in this PR

docs/build-spec.md

• Phase 3 status flipped to ✅ Complete (master timeline + Phase 3 header)
• Phase 3 § Retrospective populated — 10 PRs across 4 waves, test count 566 → 687, five lessons documented, H1 / H2 / H3 outcomes
• Phase 4 § Inherited Phase 3 follow-ups added — five items rolled forward as Phase 4 first scope items

docs/guardrails.md § Locked decisions

Four new locked decisions:

• Microsoft Foundry orchestration (ADR-0014)
• Per-AIAgent model selection + LRU cache + cost ceiling (ADR-0015)
• Confidence-threshold refusal mandatory (ADR-0017)
• Code-resource agent definitions (ADR-0018)

docs/decision-log.md

New entry DL 2026-05-07 — Phase 3 H2 eval baseline + ConfidenceThreshold stays at 0.65.

CLAUDE.md

• Test count 566 → 687
• 13 ADRs → 18 ADRs in documentation map
• Locked invariants 1-8 → 1-12 (4 new AI-architecture invariants → ADR pointers)
• Freshest handoff pointer → session_handoff_2026_05_07_phase3_close.md

Memory

• New: session_handoff_2026_05_07_phase3_close.md (10 PRs summary, hand-off outcomes, 5 lessons, repo state)
• MEMORY.md index points at it

Eval baseline

• data/eval/results/wizard.20260507T162529Z.json committed as the v1 reference

Phase 3 by the numbers

Metric	Phase 2 exit	Phase 3 exit	Δ
Tests	566	687	+121
ADRs	13 (0001–0013)	18 (0001–0018)	+5
PRs in phase	n/a	10	—
Idle Azure cost	~$30/mo	~$120–150/mo	+$90–120/mo (Foundry + ACR + KV + Storage + App Insights; AI Search deferred)

Audit posture

• Build green: 0 warnings, 0 errors under TreatWarningsAsErrors
• Tests green: 687/687 (no code changes; locked from PR feat(eval) Foundry EvaluationClient harness + custom citation-accuracy evaluators (Wave 3 PR 8) #92 merge)
• Identity check ✅ (94459922+jkeeley2073@users.noreply.github.com)
• Doc-only PR — eligible for the /local-review carve-out per guardrails.md § Per-PR gate
• Phase 3 § Retrospective covers all five lessons + all three hand-off outcomes per the per-phase-gate template
• Locked decisions list updated in same PR per guardrails.md § Spec maintenance
• Memory handoff written so the next session can resume cleanly

What this PR does NOT do (intentional)

• Fix any of the five Phase 3 lessons — those are Phase 4 work, scoped + prioritized in the build-spec
• Modify ADR-0017 — threshold stays at 0.65; ADR-0017 is unchanged. The decision-log entry captures the calibration-deferred rationale separately
• Bump Microsoft.Agents.AI versions — 1.4.0 GA is the current pin; future bumps land when the SDK exposes Usage / connected-agents primitives the inherited follow-ups need

After merge

Phase 3 is done. The next session should start with:

1. Phase 4 design conversation (fresh context window) — sequence the five inherited follow-ups, then add Phase 4 RAG-specific items (Cosmos Change Feed Function, PdfPig text extraction, page-aware chunking strategy, AI Search index population, semantic-ranker config)
2. H3 (Pinball Map live-API probe) at operator convenience — not a Phase 3 blocker; runnable any time via PINBALL_WIZARD_LIVE_CONTRACT_TESTS=1
3. Cost burn-rate review at end of June 2026 — first full month with Foundry idle cost; calibration moment for the $300/mo anomaly alarm

Test plan

• Build green (no code changes)
• Tests green (687/687, unchanged from PR feat(eval) Foundry EvaluationClient harness + custom citation-accuracy evaluators (Wave 3 PR 8) #92)
• Identity check (personal noreply on commit)
• All five Phase 3 lessons documented in retrospective
• Phase 4 inherited-follow-ups documented in build-spec § Phase 4
• Decision-log entry for the H2 calibration result
• Memory handoff written
• Locked decisions updated in guardrails.md
• CLAUDE.md updated with current state
• Reviewer confirms Phase 3 ✅ Complete is the right call given the H2 baseline + Phase 4 inherited follow-ups