docs(repo): make AGENTS.md the canonical instructions file

AGENTS.md

@@ -1,5 +1,260 @@
-# AgentV Repository
+# AgentV Repository Guidelines
 
-**FIRST ACTION**: Read `CLAUDE.md` for complete repository guidelines.
+This is a TypeScript monorepo for AgentV - an AI agent evaluation framework.
 
-This file exists for compatibility. All guidelines have been consolidated into `CLAUDE.md`.
+## High-Level Goals
+
+AgentV aims to provide a robust, declarative framework for evaluating AI agents.
+- **Declarative Definitions**: Define tasks, expected outcomes, and rubrics in simple YAML files.
+- **Structured Evaluation**: Use "Rubric as Object" (Google ADK style) for deterministic, type-safe grading.
+- **Multi-Objective Scoring**: Measure correctness, latency, cost, and safety in a single run.
+- **Optimization Ready**: Designed to support future automated hyperparameter tuning and candidate generation.
+
+## Design Principles
+
+These principles guide all feature decisions. **Follow these when proposing or implementing changes.**
+
+### 1. Lightweight Core, Plugin Extensibility
+AgentV's core should remain minimal. Complex or domain-specific logic belongs in plugins, not built-in features.
+
+**Extension points (prefer these over adding built-ins):**
+- `code-grader` scripts for custom evaluation logic
+- `llm-grader` evaluators with custom prompt files for domain-specific LLM grading
+- CLI wrappers that consume AgentV's JSON/JSONL output for post-processing (aggregation, comparison, reporting)
+
+**Ask yourself:** "Can this be achieved with existing primitives + a plugin or wrapper?" If yes, it should not be a built-in. This includes adding config overrides to existing evaluators — if a niche provider needs custom tool-name matching, that's a code-grader, not a new config field.
+
+### 2. Built-ins for Primitives Only
+Built-in evaluators provide **universal primitives** that users compose. A primitive is:
+- Stateless and deterministic
+- Has a single, clear responsibility
+- Cannot be trivially composed from other primitives
+- Needed by the majority of users
+
+If a feature serves a niche use case or adds conditional logic, it belongs in a plugin.
+
+### 3. Align with Industry Standards
+Before adding features, research how peer frameworks solve the problem. Prefer the **lowest common denominator** that covers most use cases. Novel features without industry precedent require strong justification and should default to plugin implementation.
+
+### 4. YAGNI — You Aren't Gonna Need It
+Don't build features until there's a concrete need. Before adding a new capability, ask: "Is there real demand for this today, or am I anticipating future needs?" Numeric thresholds, extra tracking fields, and configurable knobs should be omitted until users actually request them. Start with the simplest version (e.g., boolean over numeric range) and extend later if needed.
+
+### 5. Non-Breaking Extensions
+New fields should be optional. Existing configurations must continue working unchanged.
+
+### 6. AI-First Design
+AI agents are the primary users of AgentV—not humans reading docs. Design for AI comprehension and composability.
+
+**Skills over rigid commands:**
+- Use Claude Code skills (or agent skill standards) to teach AI *how* to create evals, not step-by-step CLI instructions
+- Skills should cover most use cases; rigid commands trade off AI intelligence
+- Only prescribe exact steps where there's an established best practice
+
+**Intuitive primitives:**
+- Expose simple, single-purpose primitives that AI can combine flexibly
+- Avoid monolithic commands that do multiple things
+- SDK internals should be intuitive enough for AI to modify when needed
+
+**Self-documenting code:**
+- File headers should explain what the file does, how it works, and how to extend it — no need to read other files to understand this one
+- Don't reference external projects, PRs, or issues in code comments; make everything standalone
+- Prefer data-driven patterns (static mappings, config tables) over conditional chains — AI can extend a mapping by adding an entry, but has to trace logic to extend an if/else tree
+- No dead code or speculative infrastructure; if it's unused, delete it
+- When a module has an extension point, include a short recipe in the header (e.g., "To add a new provider: 1. Create a matcher, 2. Add it to the mapping")
+
+**Scope:** Applies to skills, repo structure, documentation, SDK design, and source code — anything AI might need to reason about or extend.
+
+## Tech Stack & Tools
+- **Language:** TypeScript 5.x targeting ES2022
+- **Runtime:** Bun (use `bun` for all package and script operations)
+- **Monorepo:** Bun workspaces
+- **Bundler:** tsup (TypeScript bundler)
+- **Linter/Formatter:** Biome
+- **Testing:** Vitest
+- **LLM Framework:** Vercel AI SDK
+- **Validation:** Zod
+
+## Project Structure
+- `packages/core/` - Evaluation engine, providers, grading
+  - `src/evaluation/registry/` - Extensible evaluator registry (EvaluatorRegistry, assertion discovery)
+  - `src/evaluation/providers/provider-registry.ts` - Provider plugin registry
+  - `src/evaluation/evaluate.ts` - `evaluate()` programmatic API
+  - `src/evaluation/config.ts` - `defineConfig()` for typed agentv.config.ts
+- `packages/eval/` - Lightweight assertion SDK (`defineAssertion`, `defineCodeJudge`)
+- `apps/cli/` - Command-line interface (published as `agentv`)
+  - `src/commands/create/` - Scaffold commands (`agentv create assertion/eval`)
+- `examples/features/sdk-*` - SDK usage examples (custom assertion, programmatic API, config file)
+
+## Working Style
+
+### Planning
+- Use plan mode for any non-trivial task (5+ steps or architectural decisions).
+- If something goes sideways, STOP and re-plan immediately — don't keep pushing a broken approach.
+- For non-trivial changes, pause and ask: "Is there a more elegant solution?" before diving in.
+- Check in with the user before starting implementation on ambiguous tasks.
+
+### Subagent Strategy
+- Use subagents aggressively to keep the main context window clean.
+- Subagents for: research, file exploration, running tests, code review.
+- For complex problems, throw more subagents at it — parallelize where possible.
+- Name subagents descriptively.
+
+### Autonomous Bug Fixes
+- When you spot a bug, just fix it. Don't ask for hand-holding.
+- Point at logs, errors, failing tests — then resolve them.
+- Only ask when there's genuine ambiguity about intent.
+- Fix failing CI tests without being told.
+
+### Simplicity
+- Every change should be as simple as possible. Import existing code; don't reinvent.
+- Find root causes and fix them directly. No shotgun debugging.
+
+### Progress Updates
+- Provide high-level status updates at natural milestones.
+- When scope changes mid-task, communicate the shift and adjust the plan.
+
+## TypeScript Guidelines
+- Target ES2022 with Node 20+
+- Prefer type inference over explicit types
+- Use `async/await` for async operations
+- Prefer named exports
+- Keep modules cohesive
+
+## Wire Format Convention
+
+**All external-facing JSON and JSONL output uses `snake_case` keys.** This applies to:
+- JSONL result files on disk (`test_id`, `token_usage`, `duration_ms`)
+- Artifact-writer output (`pass_rate`, `tests_run`, `total_tool_calls`)
+- CLI command JSON output (`results summary`, `results failures`, `results show`)
+- YAML eval config fields
+
+**Internal TypeScript uses `camelCase`** as standard. Convert at the serialization boundary only:
+
+```typescript
+// Interfaces for JSON output use snake_case (they define the wire format)
+interface SummaryJson {
+  total: number;
+  pass_rate: number;
+  failed_test_ids: string[];
+}
+
+// Function internals use camelCase (idiomatic TypeScript)
+function formatSummary(results: EvaluationResult[]): SummaryJson {
+  const passRate = computePassRate(results);
+  const failedTestIds = findFailed(results);
+
+  return {
+    total: results.length,
+    pass_rate: passRate,
+    failed_test_ids: failedTestIds,
+  };
+}
+```
+
+**Reading back:** `parseJsonlResults()` in `artifact-writer.ts` converts snake_case → camelCase when reading JSONL into TypeScript.
+
+**Why:** Aligns with skill-creator (claude-plugins-official) and broader Python/JSON ecosystem conventions where snake_case is the standard wire format.
+
+## Testing & Verification
+
+### Pre-Push Hooks (Automated)
+
+The repository uses [prek](https://github.com/nickel-lang/prek) (`@j178/prek`) for pre-push hooks that automatically run build, typecheck, lint, and tests before pushing. **Do not manually run these checks before pushing** — just push to the feature branch and let the pre-push hook validate.
+
+**Setup (automatic):**
+The hooks are installed automatically when you run `bun install` via the `prepare` script. To manually install:
+```bash
+bunx prek install -t pre-push
+```
+
+**What runs on push:**
+- `bun run build` - Build all packages
+- `bun run typecheck` - TypeScript type checking
+- `bun run lint` - Biome linting
+- `bun run test` - All tests
+- `bun run validate:examples` - Validate example eval YAML files against the agentv schema
+
+If any check fails, the push is blocked until the issues are fixed.
+
+**Manual run (without pushing):**
+```bash
+bunx prek run --all-files --hook-stage pre-push
+```
+
+### Functional Testing (CLI)
+
+When functionally testing changes to the AgentV CLI, **NEVER** use `agentv` directly as it may run the globally installed version (bun or npm). Instead:
+
+- **From TypeScript source (preferred):** `bun apps/cli/src/cli.ts <args>` — always runs current code, no build step needed
+- **From built dist:** `bun apps/cli/dist/cli.js <args>` — requires `bun run build` first, can be stale
+- **From repository root:** `bun agentv <args>` — runs the locally built version (also requires build)
+
+**Prefer running from source** (`src/cli.ts`) during development. The dist build can silently serve stale code if you forget to rebuild after changes.
+
+### Browser E2E Testing (Docs Site)
+
+Use `agent-browser` for visual verification of docs site changes. Environment-specific rules:
+
+- **Always use `--session <name>`** — isolates browser instances; close with `agent-browser --session <name> close` when done
+- **Never use `--headed`** — no display server available; headless (default) works correctly
+
+### Verifying Evaluator Changes
+
+Unit tests alone are insufficient for evaluator changes. After implementing or modifying evaluators:
+
+1. **Copy `.env` to the worktree** if running in a git worktree (e2e tests need environment variables):
+   ```bash
+   cp /path/to/main/.env .env
+   ```
+   ```powershell
+   Copy-Item D:/path/to/main/.env .env
+   ```
+   Do not claim e2e or evaluator verification results unless this preflight has passed.
+
+2. **Run an actual eval** with a real example file:
+   ```bash
+   bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml --test-id <test-id>
+   ```
+
+3. **Inspect the results JSONL** to verify:
+   - The correct evaluator type is invoked (check `scores[].type`)
+   - Scores are calculated as expected
+   - Assertions array reflects the evaluation logic (each entry has `text`, `passed`, optional `evidence`)
+
+4. **Update baseline files** if output format changes (e.g., type name renames). Baseline files live alongside eval YAML files as `*.baseline.jsonl` and contain expected `scores[].type` values. There are 30+ baseline files across `examples/`.
+
+5. **Note:** `--dry-run` returns mock responses that don't match evaluator output schemas. Use it only for testing harness flow, not evaluator logic.
+
+### Completing Work — E2E Checklist
+
+Before marking any branch as ready for review, complete this checklist:
+
+1. **Preflight:** If in a git worktree, ensure `.env` exists in the worktree root.
+   ```bash
+   cp "$(git worktree list --porcelain | head -1 | sed 's/worktree //')/.env" .env
+   ```
+   Without this, any eval run or LLM-dependent test will fail with missing API key errors.
+
+2. **Run unit tests**: `bun run test` — all must pass.
+
+3. **Manual red/green UAT (REQUIRED for all changes):**
+   Automated tests are not sufficient. Every change must be manually verified from the end user's perspective using a red/green approach:
+   - **Red (before fix):** Reproduce the bug or demonstrate the missing feature on `main` (or before your change). Confirm the undesired behavior is observable from the CLI / user-facing output.
+   - **Green (after fix):** Run the same scenario with your changes applied. Confirm the fix or feature works correctly from the end user's perspective.
+   - Document both the red and green results in the PR or conversation so the user can see the before/after.
+
+   For evaluator changes, this means running a real eval (not `--dry-run`) and inspecting the output JSONL. For CLI/UX changes, this means running the CLI command and verifying the console output.
+
+4. **Verify no regressions** in areas adjacent to your changes (e.g., if you changed evaluator parsing, run an eval that exercises different evaluator types).
+
+5. **Mark PR as ready** only after all above steps pass.
+
+## Documentation Updates
+
+When making changes to functionality:
+
+1. **Docs site** (`apps/web/src/content/docs/`): Update human-readable documentation on agentv.dev. This is the comprehensive reference.
+
+2. **Skill files** (`plugins/agentv-dev/skills/agentv-eval-builder/`): Update the AI-focused reference card if the change affects YAML schema, evaluator types, or CLI commands. Keep concise — link to docs site for details.
+
+3. **Examples** (`examples/`): Update any example code, scripts, or eval YAML files that exercise the changed functionality. Examples are both documentation and integration tests.