Skip to content

Documentation: Add experiment creation API call to evaluation guide #331

@pierrevalade

Description

@pierrevalade

Issue Description

The evaluation guide at shows how to run evaluations with experiments, but doesn't demonstrate how to actually create an experiment via the API.

Current State

The documentation currently:

  • Shows generating an experiment ID locally: experiment_id = f"email-eval-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
  • References using this ID in completion metadata
  • Shows sending annotations for the experiment

However, it's missing the actual API call to create the experiment entity.

Proposed Improvement

The evaluation pipeline script should:

  1. Create a new experiment via the experiments API
  2. Add completions to this experiment (already shown)
  3. Send annotations for evaluation results (already shown)

This would provide a complete end-to-end example of programmatically running experiments.

Affected Section

Lines 471-472 in the "Create the Evaluation Pipeline" section currently show only local ID generation without the API call.

-- Claude Code

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions