tracking: multi-model x multi-metric x variability gap closure roadmap

[christso]

SOURCE OF TRUTH FOR ORCHESTRATOR HANDOFF

This issue body is the canonical execution plan for this roadmap.

0) Execution Philosophy

• We keep design freedom with hard scope boundaries.
• We prefer iterative delivery and incorporate implementation learnings.
• We avoid speculative abstractions that do not serve immediate acceptance signals.

1) Architecture Boundary

Core runtime (allowed/required)

• feat(eval): suite-level total cost budget guardrail (core runtime) #369 suite-level total cost budget guardrail

Docs/examples only (non-core)

• docs/examples: add multi-model x multi-metric x variability benchmark showcase #370 multi-model benchmark showcase

External-first (tooling/plugin/post-processing)

• feat(tooling): split combined results JSONL by target (external-first) #364 per-target JSONL splitting
• feat(tooling): paired significance testing for result-file comparisons (external-first) #365 paired statistical significance testing
• feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366 aggregate win-rate summaries
• feat(tooling): benchmark summary report across models/metrics (external-first) #367 benchmark summary report
• feat(plugin): trial output-consistency metric via embedding similarity #368 trial output-consistency metric (plugin/reference)

If we implement an external-first item inside agentv, we keep it as optional utility/report tooling and avoid runtime/schema coupling.

2) Current Implementation Status (snapshot)

• feat(tooling): split combined results JSONL by target (external-first) #364: partially implemented foundation (matrix eval + target field), no auto split-by-target mode.
• feat(tooling): paired significance testing for result-file comparisons (external-first) #365: not implemented (compare has deltas only; no significance tests).
• feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366: partially implemented (counts exist; rates/per-metric aggregation missing).
• feat(tooling): benchmark summary report across models/metrics (external-first) #367: partially implemented building blocks (matrix summary, trace stats, wrappers), no benchmark report mode.
• feat(plugin): trial output-consistency metric via embedding similarity #368: partially implemented adjacent tooling; no embedding-based trial consistency workflow yet.
• feat(eval): suite-level total cost budget guardrail (core runtime) #369: partially implemented adjacent controls (trial cost limit + provider budget), no suite-level total budget.
• docs/examples: add multi-model x multi-metric x variability benchmark showcase #370: partially implemented via separate examples; no unified showcase.

3) Delivery Tracks

Track A: AgentV core/docs

• feat(eval): suite-level total cost budget guardrail (core runtime) #369 (core runtime)
• docs/examples: add multi-model x multi-metric x variability benchmark showcase #370 (docs/examples)

Track B: External analytics/plugin

• feat(tooling): split combined results JSONL by target (external-first) #364, feat(tooling): paired significance testing for result-file comparisons (external-first) #365, feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366, feat(tooling): benchmark summary report across models/metrics (external-first) #367, feat(plugin): trial output-consistency metric via embedding similarity #368

4) Dependency Graph

• feat(tooling): split combined results JSONL by target (external-first) #364 improves ergonomics for docs/examples: add multi-model x multi-metric x variability benchmark showcase #370 and feat(tooling): benchmark summary report across models/metrics (external-first) #367 outputs.
• feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366 should land before feat(tooling): paired significance testing for result-file comparisons (external-first) #365 (aggregate baseline before significance layer).
• feat(tooling): paired significance testing for result-file comparisons (external-first) #365 + feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366 feed feat(tooling): benchmark summary report across models/metrics (external-first) #367 reporting.
• feat(plugin): trial output-consistency metric via embedding similarity #368 is mostly independent (plugin-first).
• feat(eval): suite-level total cost budget guardrail (core runtime) #369 is independent of compare/report stack.

5) Parallel Waves (default)

Wave 1 (parallel)

• Agent A: feat(eval): suite-level total cost budget guardrail (core runtime) #369
• Agent B: feat(tooling): split combined results JSONL by target (external-first) #364
• Agent C: feat(plugin): trial output-consistency metric via embedding similarity #368

Wave 2 (parallel, once Wave 1 PRs are open)

• Agent D: feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366
• Agent E: docs/examples: add multi-model x multi-metric x variability benchmark showcase #370 (draft early, finalize against stable outputs)

Wave 3

• Agent F: feat(tooling): paired significance testing for result-file comparisons (external-first) #365 (after feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366)

Wave 4

• Agent G: feat(tooling): benchmark summary report across models/metrics (external-first) #367 (after feat(tooling): split combined results JSONL by target (external-first) #364/feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366/feat(tooling): paired significance testing for result-file comparisons (external-first) #365)

6) Merge Order (low-conflict default)

1. feat(eval): suite-level total cost budget guardrail (core runtime) #369
2. feat(tooling): split combined results JSONL by target (external-first) #364
3. feat(plugin): trial output-consistency metric via embedding similarity #368
4. feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366
5. feat(tooling): paired significance testing for result-file comparisons (external-first) #365
6. feat(tooling): benchmark summary report across models/metrics (external-first) #367
7. docs/examples: add multi-model x multi-metric x variability benchmark showcase #370 (can merge earlier if independent of unmerged behavior)

7) Subagent Contract

• We read CLAUDE.md in agentv first.
• We use branch + PR only (no direct push to main).
• We keep one issue per PR and avoid unrelated refactors.
• We include tests/docs required by each issue acceptance signals.
• We use squash merge and link each PR back to this tracking issue.

Suggested prompt:
"Implement EntityProcess/agentv issue #. First read CLAUDE.md in repo root. Create a feature branch, implement only this issue, add/update tests and docs, push, and open a PR that closes #. Keep changes scoped and avoid unrelated refactors."

8) Completion Criteria

• Core runtime: feat(eval): suite-level total cost budget guardrail (core runtime) #369 merged.
• Docs/examples: docs/examples: add multi-model x multi-metric x variability benchmark showcase #370 merged.
• External analytics/plugin deliverables available for feat(tooling): split combined results JSONL by target (external-first) #364/feat(tooling): paired significance testing for result-file comparisons (external-first) #365/feat(tooling): aggregate win-rate summaries for result comparisons (external-first) #366/feat(tooling): benchmark summary report across models/metrics (external-first) #367/feat(plugin): trial output-consistency metric via embedding similarity #368.
• End-to-end benchmark workflow executes without bespoke one-off scripts.