Reference data not found all of a sudden

[Irikos]

Hello,

This problem popped up seemingly out of nowhere.

I've made the evaluation script to access the reference data. Participants upload their json and the script compares it to the reference data and computes a score. It all worked fine for about 1.5 months, until 2 days ago when it stopped working, due to the script not being able to find the reference data anymore.

The data is still visible in "Files" tab and this error appeared with us making to changes to the competition.

I would very much like to understand what happened / is happening, both to fix it now and make sure I avoid this problem in the future.

Thank you!

Competition link: https://www.codabench.org/competitions/9109/

Full error example:
Traceback (most recent call last): File "/app/program/scoring.py", line 133, in map_score, ap50 = compute_Average_Precision(ground_truth_json, predictions_json) File "/app/program/scoring.py", line 11, in compute_Average_Precision gt_coco = COCO(ground_truth_json) File "/usr/local/lib/python3.10/site-packages/pycocotools/coco.py", line 81, in init with open(annotation_file, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/app/input/ref/reference_data/val_annotations.json'

[ihsaan-ullah]

@Didayolo Is #1905 already deployed? I think this problem is coming from there. When users use folders on purpose then they will get such errors

[gifale95]

Hello,

We are also experiencing the same problem for competitions 9482 and 9483. More precisely, submissions worked until July 17th, but from that day when submitting we get the error:

Traceback (most recent call last):
  File "/app/program/phase-2_model_selection_2025-06-18.py", line 254, in <module>
    target_samples = np.load(data_dir, allow_pickle=True).item()
  File "/opt/conda/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/app/input/ref/sub-01/target_sample_number/sub-01_ood_fmri_samples.npy'

We did not change neither the scoring program nor the reference data during this period. We would appreciate if you could fix this soon, so that submissions can resume.

Thanks,
Ale

[0oTyTo0]

same problem with gifale95 !

[ihsaan-ullah]

Hi @Irikos @0oTyTo0 @gifale95

Can you resubmit your submisison. There was a small problem on the compute worker side that we have fixed.

If you are using your own compute worker, use the tag cpu1.1 instead of latest (ignore this if you are using codabench worker)

[gifale95]

@ihsaan-ullah Submissions are now once again working, thank you very much for fixing this quickly!

[Irikos]

Yes, it works fine now. Thanks for the prompt response!