chore(eval) Phase 4 W1-3 hardening — mfg-aware re-curation skip-on-mismatch

[jkeeley2073]

Summary

A spot-check of the W1-3 first run (PR #98) revealed a silent failure mode: the 3 Godzilla questions in the eval set are intended for Stern's 2021 Godzilla, but the deployed Cosmos catalog only contains Sega's 1998 Godzilla. The W1-3 script issued SELECT TOP 1 c.id ... STRINGEQUALS(c.title, 'Godzilla', true) and took the first hit blindly — recording the Sega record's id under each Godzilla question's expected_citation_set. The agent's correct answer about Stern 2021 would have failed eval because its citation wouldn't match the (incorrect) Sega ground truth. Same risk class exists for any title shared across manufacturers/eras.

This PR ships hardening of the recuration tool only. The next live recuration run is sequenced after the OPDB sync investigation closes (why Stern's modern catalog is currently absent from the deployed Cosmos is a separate root-cause investigation already underway). Running the live script before that closes would compound the catalog-state issue. The W1-3 first run's artifacts (data/eval/wizard.v1.jsonl and data/eval/wizard.v1.recuration.json) remain authoritative until then.

Changes

• tools/eval/wizard.v1.titles.json — add per-question expected_manufacturer column (lowercase string matching the deployed catalog's manufacturer partition value — stern, jjp, sega, etc.). All 30 rows curated from the question text + notes field. Out-of-scope rows use null to mirror their machine_title=null. _about field documents the new column and the failure-mode that motivated it.
• tools/eval/Recurate.csx — replace LookupOpdbIdByTitle (returned a tuple from SELECT TOP 1) with QueryHitsByTitle (returns all hits). Caller walks results and picks the first hit whose manufacturer matches expected_manufacturer (case-insensitive). On no-match, skip the row with new mfg_mismatch outcome — JSONL untouched. On null expected_manufacturer for an in-scope row, fall back to first-hit-wins and log a "manufacturer-unconstrained" warning. New counts (skipped_mfg_mismatch, manufacturer_unconstrained) flow into the recuration manifest's counts block; per-row outcomes carry expected_manufacturer alongside resolved_manufacturer for full audit trail.
• data/eval/README.md — append "Hardening (2026-05-08) — manufacturer-aware skip-on-mismatch" subsection documenting the failure mode + the new behavior + the dry-run verification output, with explicit tooling-only; live re-run sequenced after OPDB sync investigation closes callout.

Test Plan

• dotnet build PinballWizard.slnx — 0 warnings, 0 errors (no src/ changes; build remains green).
• dotnet script tools/eval/Recurate.csx -- --dry-run against the deployed Cosmos endpoint (pinwiz-cosmos-dev-hlpz4, the same endpoint used by the W1-3 first run; az login to the personal Earlybird sub) — captured output appended to data/eval/README.md. Result:
  • 3 Godzilla rows (ev-rules-0003, ev-valuation-0004, ev-repair-0003) now flagged mfg_mismatch (expected stern, catalog has sega) — the failure mode this PR exists to catch.
  • 5 JJP rows still resolve correctly (3× The Wizard of Oz, 2× Dialed In!) — JJP is the only manufacturer holding those titles in the deployed catalog.
  • 18 no_match rows unchanged (Stern's modern catalog still absent — separate investigation).
  • 4 out_of_scope rows unchanged (correct refusals).
  • 0 manufacturer-unconstrained (every in-scope row now has expected_manufacturer set).
• The live (non-dry-run) script was deliberately NOT run; the eval JSONL ground truth and the W1-3 recuration provenance are unchanged.

Out of Scope

• The OPDB sync investigation (why Stern's modern catalog is missing from deployed Cosmos) — separate root-cause investigation already underway.
• Re-running the live recuration to update data/eval/wizard.v1.jsonl against a populated catalog — sequenced after the OPDB sync investigation closes.
• Unit tests for Recurate.csx — the script doesn't compile as part of PinballWizard.slnx; the dry-run output documented in README is the test-of-record (it exercises the new mfg_mismatch branch on real Godzilla rows).

Checklist

• CI is green (build + test; tooling-only PR — script exercised via dry-run)
• PR title follows the Conventional Commits format
• If this is a new architectural decision, an ADR has been added under docs/adr/ — N/A (hardening of an existing tool, not a new architectural decision)
• If user-visible behavior changes, README.md and/or docs/ are updated in the same PR (data/eval/README.md appended)
• If a memory in ~/.claude/projects/c--projects-PinballWizard/memory/ is now stale, it has been updated or removed in the same PR — N/A (no memory referenced this script)
• No TODO / FIXME / commented-out code committed
• No new entries in <NoWarn> without a comment

Pre-push self-audit

Step 0 — /local-review (qualitative)

• Ran /local-review and addressed every 🔴 finding before push
• Local review outcome: 0 🔴 / 2 ⚠️ / 8 categories ✅
  • ⚠️ Test quality: dry-run-output-as-test approach is appropriate for a dotnet-script tool that doesn't compile as part of PinballWizard.slnx; the Godzilla rows actually exercise the new mfg_mismatch branch. Deferred — justification: there is no project to host xUnit tests in; the dry-run is the canonical verification path documented in the script header.
  • ⚠️ Performance: QueryHitsByTitle reads ALL hits per title rather than SELECT TOP 1. Deferred — justification: TOP 1 was the bug; reading all hits is the necessary correctness fix. RU cost is negligible across 30 questions and the dry-run completed in 3.1s (no observable regression vs. the W1-3 first run).

Step 1 — Mechanical checklist

• Every new *Options property has at least one real getter call in src/ — N/A (no *Options class added; the new expected_manufacturer side-car field is read at lines 188-190 and consumed at lines 332/344 of Recurate.csx)
• Sibling-diffed against the closest existing implementation; drift is justified or removed — diffed against the prior version of the same script; no drift.
• No bare catch { } — no new catches added.
• New ISourceScraper? — N/A
• Tests assert behavior, not just structure — the dry-run exercises the actual mfg_mismatch branch on real Godzilla rows.
• Build is zero-warning — dotnet build PinballWizard.slnx → 0 warnings, 0 errors.
• git log -1 --format='%an <%ae>' shows personal noreply, not work email — verified Jim Keeley <94459922+jkeeley2073@users.noreply.github.com>.