docs/examples: public benchmark starter pack for major open benchmark repos

[christso]

Objective

Add a first-party docs/examples starter pack showing how to use AgentV with the main public benchmark repos people already recognize, instead of leaving each benchmark as a one-off research note.

Why this is needed

AgentV already has the core primitives for benchmark-facing workflows:

• agentv import huggingface
• workspace isolation / Docker-oriented execution paths
• code, rubric, llm-judge, and tool-trajectory grading
• multi-trial aggregation (pass_at_k, mean, confidence_interval)

What is still missing is the starter layer: copy-paste examples and short guides that map well-known public benchmark repos into AgentV.

Recent MiniMax M2.7 follow-up research: https://github.com/agentevals/agentevals-research/pull/58

Scope

Create a curated benchmark example pack covering the highest-signal public repos:

1. SWE-bench/SWE-bench — canonical issue-resolution benchmark import example
2. multi-swe-bench/multi-swe-bench — multilingual SWE-style example
3. laude-institute/terminal-bench and harbor-framework/terminal-bench-2 — execution-based terminal task examples
4. multimodal-art-projection/NL2RepoBench — greenfield repo-generation example
5. openai/mle-bench — MLE-Bench Lite guide as a practical long-running benchmark example
6. anomalyco/opencode-bench — advanced multi-judge pattern reference

Deliverables

• docs page or example-pack index for public benchmark repos
• at least one runnable example/import path for each benchmark shape (issue-resolution, terminal tasks, repo generation, long-running optimization)
• clear caveats for heavyweight setups so examples stay honest about runtime costs
• explicit note that internal-only vendor eval names are not canonical public benchmark targets

Non-goals

• not a new benchmark ingestion subsystem
• not a public leaderboard product by itself
• not bundling huge benchmark datasets into the repo

Related

• feat: curated public benchmark dataset and leaderboard #966 curated public benchmark dataset and leaderboard
• feat: HuggingFace dataset import command #978 HuggingFace import command (already delivered)
• feat(core): wire base_commit into docker-workspace for SWE-bench evals #987 base_commit in docker workspace (already delivered)
• test: end-to-end SWE-bench Docker eval run #988 end-to-end SWE-bench Docker eval run (already delivered)

[christso]

Expanded scope: benchmark suite packaging, harness eval examples, and registry pattern

Based on competitive research and the harness-level evaluation finding, proposing three additions to this issue's scope:

1. Benchmark-native suite packaging format

Add an optional directory-based case format alongside EVAL.yaml for SWE-style benchmarks:

my-suite/
  suite.yaml           # case list, metadata
  cases/
    fix-null-check/
      prompt.md
      tests/verify.sh
      env/Dockerfile     # optional
      oracle/solve.sh    # optional (see #1136)

EVAL.yaml remains the top-level orchestrator — the suite directory is a packaging convention for benchmark case assets, not a replacement.

2. Harness evaluation examples

Add examples that evaluate how well the agent used its harness (skills, MCP tools, workspace instructions), not just whether the task passed:

• Skill effectiveness eval — tool-trajectory asserting the right skill fired, execution-metrics showing reduced tool usage
• MCP tool verification eval — tool-trajectory with arg matching on MCP tool calls
• Workspace instruction adherence eval — rubric scoring whether the agent followed AGENTS.md conventions
• Composite harness quality eval — composite weighting correctness + tool usage + efficiency + instruction adherence

These showcase AgentV's key differentiator: evaluating the configured agent, not just the naked model.

3. Remote suite references / registry pattern

Support git-hosted suite references for importing packaged benchmark collections:

agentv import suite git::https://github.com/example/benchmark-suites.git//swe-bench-lite

This helps AgentV compete for benchmark distribution workflows without abandoning its evaluator model.

Related new issues

• feat: immutable run bundle / reproducibility artifact #1133 — immutable run bundle
• feat: unified agent profile abstraction for cross-agent comparison #1134 — unified agent profile abstraction
• feat: snapshot MCP server pattern for reproducible benchmarks #1135 — snapshot MCP server pattern
• feat: oracle-run workflow for benchmark case validation #1136 — oracle-run workflow

[christso]

Benchmark packs should live in a separate repo

Margin's approach: main evals repo has framework code + one minimal test suite. Actual benchmarks live in swe-suites (392 MB) and test-suites (49 KB) repos.

AgentV should follow the same pattern:

• agentv repo — framework code + small example evals (already in examples/)
• Separate benchmark repo (e.g., EntityProcess/agentv-benchmarks) — actual benchmark packs, imported datasets, harness evaluation examples

Reasons:

1. Benchmark data is heavy — SWE-bench workspace templates, Dockerfiles, test scripts, oracle solutions add up fast
2. Benchmarks version independently from the framework
3. git clone agentv shouldn't pull hundreds of MB of benchmark assets
4. Benchmark packs can be referenced via agentv import suite git::https://... pattern

The examples/ directory in the main repo stays for small, self-contained feature demos. The benchmark repo is for production-scale benchmark packs.