Compute worker issues

[Didayolo]

Submissions stuck in "Scoring" status

• Submissions stuck in "Scoring" state instead of "Failed" when the compute worker crashes

In the comment here, interesting explanation about why this problem happens and how to solve it:

• Submissions with exit code != 0 are marked as completed instead of failed #1641

Related issues:

• Docker pull fails + wrong status #1203

• submission in "Scoring" status for multiple hours on default queue #1184

• Failed statuses not updated #1257

• Submissions fail silently #1821

• Similarly, it looks like the status get stuck to "Preparing" when failing during this process.

Example failure during "Preparing":

[2025-09-18 11:25:05,234: ERROR/ForkPoolWorker-2] Task compute_worker_run[fd956bf5-3e2d-4168-ab48-f0896dc80993] raised unexpected: OSError(28, 'No space left on device')
Traceback (most recent call last):
[...]
OSError: [Errno 28] No space left on device

Duplication of submission files

• Compute worker - Avoid duplicating submission files #1874

Original issue:

• Discrepancy in Reported Storage Quota Usage #1833

Directory structure problem

• Compute worker - Allow to unzip files with parent directory #1905

Docker pull failing

• Docker pull failing

Pull for image: codalab/codalab-legacy:py39 returned a non-zero exit code! Check if the docker image exists on docker hub.

Related issues:

• submission in "Scoring" status for multiple hours on default queue #1184
• Solution is always running #1278
• Submission stuck on scoring status (Twice) #1263

Solution:

• To have more logs, we need to update compute_worker.py so we print more logs in the logger (More logs when docker pull fails in compute_worker.py #1283).

Logs at the wrong place

• Docker pull error during scoring are written in ingestion stderr instead of scoring stdrerr (Docker pull in scoring #1204)
  Solved by: Show error in scoring std_err #1214

No hostname is server status when status is "Preparing"

• The "Preparing" status means that the worker is downloading the necessary data and programs to run the submission. We should already have a hostname in the server status page during this phase, but it is not the case.

https://www.codabench.org/server_status

[Didayolo]

Docker pull failing: a worker was ill, we removed it from the queue.