Issue encountered
Unable to load a local dataset when creating a custom task.
Currently, as part of the Create new task workflow, the class LightevalTaskConfig only supports hugging face datasets via the parameter hf_repo. This does not allow us to test local datasets.
Solution/Feature
Under the hood, load_dataset from the datasets library is called that actually supports local datasets via the parameters data_files and data_dir. These parameters are not exposed thereby preventing us to load local datasets.
Please let me know if there is another way to achieve this if I've missed.
Issue encountered
Unable to load a local dataset when creating a custom task.
Currently, as part of the
Create new taskworkflow, the classLightevalTaskConfigonly supports hugging face datasets via the parameterhf_repo. This does not allow us to test local datasets.Solution/Feature
Under the hood,
load_datasetfrom thedatasetslibrary is called that actually supports local datasets via the parametersdata_filesanddata_dir. These parameters are not exposed thereby preventing us to load local datasets.Please let me know if there is another way to achieve this if I've missed.