feat(eval): composable quality gates with auto-remediation triggers

[christso]

Status Update (2026-04-05)

This issue remains valid as a core evaluation/CLI capability issue.

The earlier Studio expansion is no longer the default plan. Quality enforcement today is primarily handled by AgentV eval thresholds + GitHub Actions CI, so the dashboard-management portion of this issue should be treated as optional follow-up work, not part of the critical path.

Revised Scope

In scope

• severity: error | warning | info on evaluator configs
• non-blocking warnings / informational outcomes in result data
• optional remediation hooks if they are clearly useful in CLI/automation workflows
• reusable gate patterns via code graders / scripts / packages

Deprioritized

• Dashboard-driven gate CRUD
• Visual threshold editor in Studio
• Alert routing / remediation center in Studio
• Gate compliance dashboards in Studio

Notes

If Studio ever needs a UI for this later, it should come after the core primitives exist and only if CI + CLI workflows prove insufficient.

Acceptance Signals

• severity: warning|error|info works on evaluator configs
• warnings are visible in output/result data without failing the eval
• errors continue to block the eval as today
• any remediation hook design is usable from CLI/automation flows

[christso]

Extension: Dashboard-Driven Gate Configuration

This issue currently defines the core engine (severity levels, auto-remediation triggers, reusable gate library). The following dashboard surface should be added as part of the AgentV Studio (#563 upgrade):

Dashboard capabilities for quality gates

• Gate CRUD: Create, edit, delete quality gates from the dashboard UI (not just YAML)
• Visual threshold editor: Set thresholds with a slider, overlaid on historical score distribution so users can see where the threshold falls relative to actual performance
• Alert routing: Configure where regression notifications go (dashboard feed, webhook, email)
• One-click remediation: From a gate failure alert, trigger rerun, prompt mutation (feat(bench): mutator subagent — autonomous artifact rewriting from failure analysis #746), or Ralph Loop (feat(eval): Ralph Loop — iterative improvement with feedback injection #699) directly
• Gate compliance history: Timeline showing when gates passed/failed across runs
• Recommended thresholds: Based on historical baseline, suggest sensible gate thresholds for new evaluators

How this connects

• Gate engine (this issue) → stores gate definitions + evaluates them
• Dashboard (feat: AgentV Studio — local explorer hardening #563) → provides the UI for gate management + alert visualization
• Orchestration monitor (feat(dashboard): orchestration monitor — campaign management UI #785) → shows remediation campaigns triggered by gate failures
• Regression alerts (feat(eval): iteration tracking, termination taxonomy, and cross-run regression detection #335) → feeds regression data into gate compliance view

Research source

• melagiri/code-insights — demonstrates effective dashboard-driven configuration with real-time feedback

[christso]

Closing as YAGNI — the existing required + weight fields already express severity semantics:

• required: true = error (must pass)
• required: false (default) = warning (contributes to score)
• weight: 0 = info (tracked but doesn't affect score)

Auto-remediation hooks belong in CI/CD pipelines, not the eval framework. See #337 for full rationale.