Hi! I submitted a PR to your HF dataset repo to register SciCode as a benchmark on the Hub's new Community Evals leaderboard system:
PR: https://huggingface.co/datasets/SciCode1/SciCode/discussions/5
This would give SciCode its own leaderboard page on Hugging Face where scores show up directly on model cards. The eval.yaml uses NeMo Evaluator as the evaluation framework, which already has a pre-built SciCode container for running the benchmark reproducibly.
Just needs a merge of the eval.yaml file on the HF dataset repo. Would love your thoughts!
Hi! I submitted a PR to your HF dataset repo to register SciCode as a benchmark on the Hub's new Community Evals leaderboard system:
PR: https://huggingface.co/datasets/SciCode1/SciCode/discussions/5
This would give SciCode its own leaderboard page on Hugging Face where scores show up directly on model cards. The
eval.yamluses NeMo Evaluator as the evaluation framework, which already has a pre-built SciCode container for running the benchmark reproducibly.Just needs a merge of the
eval.yamlfile on the HF dataset repo. Would love your thoughts!