Skip to content

chore: rename dataset → eval_set and derive from name field #679

@christso

Description

@christso

Objective

Rename the dataset field to eval_set throughout the codebase and change its derivation to use the YAML name field instead of the old dataset field or filename fallback.

Motivation

  • Align with agentv-bench which already uses eval_set terminology
  • dataset is generic and overloaded; eval_set is more precise
  • Many eval files share the same filename (dataset.eval.yaml), making filename-derived dataset values meaningless — the name field is a better source

Design

  • Rename: dataseteval_set in types (EvalTest, EvaluationResult), Zod schema, orchestrator, CLI commands, OTel exporter, tests, and JSONL baselines
  • Derivation change: eval_set reads from suite.name (already in schema) instead of suite.dataset, falling back to filename
  • Clean break: Remove dataset from RawTestSuite — no backward compat shim
  • CLI: --group-by dataset becomes --group-by eval-set
  • Examples: Add name: field to example YAMLs that only had description:

Acceptance Signals

  • All tests pass with renamed field
  • eval_set in results JSONL output reflects name from YAML (not filename)
  • CLI --group-by eval-set works
  • No remaining references to dataset as a field name in core/CLI code

Non-Goals

  • Renaming filenames (e.g., dataset.eval.yaml stays — it's a file naming convention)
  • Adding name: to every example YAML (just representative ones)

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions