Summary
Add cost tracking and estimation for LLM-based evaluations.
Tasks
- Track token usage in trace events.
- Calculate estimated cost based on model pricing.
- Add
--cost-limit flag to abort if budget exceeded.
- Include cost summary in reports.
Acceptance criteria
- Token counts captured in trace.
- Cost estimates shown in results.
- Evaluation stops gracefully at cost limit.
Summary
Add cost tracking and estimation for LLM-based evaluations.
Tasks
--cost-limitflag to abort if budget exceeded.Acceptance criteria