add a task dump in registry for better documentation of tasks#1052
add a task dump in registry for better documentation of tasks#1052
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Pull Request Overview
This PR adds a task dump functionality to the registry that exports all task names, metadata, and docstrings as JSON for better documentation. The changes also reduce the number of Global MMLU task variations.
- Adds
get_tasks_dump()method to the Registry class to serialize task configurations and metadata - Adds a new
dumpCLI command to output task information as JSON - Reduces Global MMLU task variations by removing CF and Hybrid formulations and limiting to "ALL" sensitivity label
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/lighteval/tasks/registry.py | Adds instance variables to store initialization parameters and implements get_tasks_dump() method with docstring parsing and value serialization logic |
| src/lighteval/tasks/multilingual/tasks/global_mmlu.py | Removes unused imports (CFFormulation, HybridFormulation) and reduces task variations |
| src/lighteval/main_tasks.py | Adds new dump CLI command that uses the registry's task dump functionality to output JSON |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…gface#1052) * add a task dump in registry for better documentation of tasks * Update src/lighteval/tasks/registry.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/lighteval/tasks/registry.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/lighteval/tasks/registry.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * remove * fix aimo --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
No description provided.