Data inside workers

[Didayolo]

In CodaLab, we have the possibility to add data directly inside the compute workers, so it can be read by the ingestion or scoring program:

https://github.com/codalab/codalab-competitions/wiki/User_Using-your-own-compute-workers#put-data-directly-inside-the-worker

The way it is implemented is simply that the folder /data of the worker is mounted on the docker container of the submission.

Do we have this on Codabench? If no, should we implement this in the same way than in CodaLab or is there a better practice?

[dtuantran]

On Codabench we have HOST_DIRECTORY=/codabench env variable. I think we can do probably the same by putting data in /codabench/storage/. We need to try to see if it works.
Here is the code line related in compute_worker.py :

codabench/compute_worker/compute_worker.py

Line 543 in d8424b8

if self.input_data:

[Didayolo]

Thanks for your input. If this work, I'll update the documentation here: https://github.com/codalab/codabench/wiki/Compute-Worker-Management---Setup

[Didayolo]

To be confirmed:

• The data in HOST_DIRECTORY can be accessed
• The access is read-only