Skip to content

implement efficiency metrics#33

Merged
lintangsutawika merged 1 commit intomainfrom
feat/efficiency_metric
Dec 11, 2025
Merged

implement efficiency metrics#33
lintangsutawika merged 1 commit intomainfrom
feat/efficiency_metric

Conversation

@XuhuiZhou
Copy link
Contributor

solve #26

Tracks tokens (from vLLM TokenEvent IDs), steps, avg_tool_calls_per_step,
and wall_clock_duration

  • New utility module: efficiency_metrics.py for computation
  • Modified code_search_generator.py to capture timing and compute metrics
  • Metrics saved to trajectory JSONs automatically during training
  • Comprehensive test suite (19 tests, all passing)
  • Backward compatible, works with both sync and async training modes

Copy link
Collaborator

@lintangsutawika lintangsutawika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some adjustments:

  1. Moved efficiency_metrics.py to src/metrics/
  2. Adjust the parameters for the efficiency metrics that is returned from init_and_run as a dict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add efficiency metrics to the benchmarking code

2 participants

Comments