Local setup - Compute worker cannot get submission from Minio - Connection refused

[ihsaan-ullah]

On my local setup on mac OS, my compute worker is giving me this issue:

Compute worker logs:

[2025-09-17 09:30:36,971: INFO/ForkPoolWorker-1] Getting bundle http://docker.for.mac.localhost:9000/private/dataset/2025-09-17-1758101436/000fd7159ab8/iris_submission.zip?AWSAccessKeyId=testkey&Signature=P7KdRXuvHfP2EBHSWMzwlN8iCJY%3D&Expires=1758533436 to unpack @ program
[2025-09-17 09:30:36,985: INFO/ForkPoolWorker-1] CODALAB_IGNORE_CLEANUP_STEP mode enabled, ignoring clean up of: /codabench/tmp81fz84vf
[2025-09-17 09:30:37,000: ERROR/ForkPoolWorker-1] Task compute_worker_run[400cd295-6f93-43ab-977d-9b719e20a45a] raised unexpected: URLError(ConnectionRefusedError(111, 'Connection refused'))
Traceback (most recent call last):
  File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib64/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/lib64/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/usr/lib64/python3.9/socket.py", line 856, in create_connection
    raise err
  File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/compute_worker.py", line 112, in run_wrapper
    run.prepare()
  File "/app/compute_worker.py", line 808, in prepare
    zip_file = self._get_bundle(url, path, cache=cache_this_bundle)
  File "/app/compute_worker.py", line 457, in _get_bundle
    urlretrieve(url, bundle_file)
  File "/usr/lib64/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

Any idea what could be wrong. I am on branch public-leaderboard

[Didayolo]

Hi @ihsaan-ullah,

I have the same problem on my local instance (branch develop). I thought it was only me:

codabench-compute_worker-1  | [2025-09-19 11:35:24,343: INFO/ForkPoolWorker-1] Submission updated successfully!
codabench-compute_worker-1  | [2025-09-19 11:35:24,352: INFO/ForkPoolWorker-1] Checking if cache directory needs to be pruned...
codabench-compute_worker-1  | [2025-09-19 11:35:24,355: INFO/ForkPoolWorker-1] Cache directory does not need to be pruned!
codabench-compute_worker-1  | [2025-09-19 11:35:24,355: INFO/ForkPoolWorker-1] Getting bundle http://docker.for.mac.localhost:9000/private/dataset/2025-09-19-1758281722/2ddf2be937a7/sample_code_submission.zip?AWSAccessKeyId=testkey&Signature=sUJVFS77dEu4hVp96z3esTQyQWA%3D&Expires=1758713723 to unpack @ program
codabench-compute_worker-1  | [2025-09-19 11:35:24,380: INFO/ForkPoolWorker-1] CODALAB_IGNORE_CLEANUP_STEP mode enabled, ignoring clean up of: /codabench/tmpliirrdqc
codabench-compute_worker-1  | [2025-09-19 11:35:24,421: ERROR/ForkPoolWorker-1] Task compute_worker_run[bf95baa8-125b-4501-bd51-0d19f845ffcc] raised unexpected: URLError(ConnectionRefusedError(111, 'Connection refused'))
codabench-compute_worker-1  | Traceback (most recent call last):
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open
codabench-compute_worker-1  |     h.request(req.get_method(), req.selector, req.data, headers,
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1285, in request
codabench-compute_worker-1  |     self._send_request(method, url, body, headers, encode_chunked)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
codabench-compute_worker-1  |     self.endheaders(body, encode_chunked=encode_chunked)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
codabench-compute_worker-1  |     self._send_output(message_body, encode_chunked=encode_chunked)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
codabench-compute_worker-1  |     self.send(msg)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 980, in send
codabench-compute_worker-1  |     self.connect()
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 946, in connect
codabench-compute_worker-1  |     self.sock = self._create_connection(
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/socket.py", line 856, in create_connection
codabench-compute_worker-1  |     raise err
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
codabench-compute_worker-1  |     sock.connect(sa)
codabench-compute_worker-1  | ConnectionRefusedError: [Errno 111] Connection refused
codabench-compute_worker-1  | 
codabench-compute_worker-1  | During handling of the above exception, another exception occurred:
codabench-compute_worker-1  | 
codabench-compute_worker-1  | Traceback (most recent call last):
codabench-compute_worker-1  |   File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 385, in trace_task
codabench-compute_worker-1  |     R = retval = fun(*args, **kwargs)
codabench-compute_worker-1  |   File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 650, in __protected_call__
codabench-compute_worker-1  |     return self.run(*args, **kwargs)
codabench-compute_worker-1  |   File "/app/compute_worker.py", line 112, in run_wrapper
codabench-compute_worker-1  |     run.prepare()
codabench-compute_worker-1  |   File "/app/compute_worker.py", line 808, in prepare
codabench-compute_worker-1  |     zip_file = self._get_bundle(url, path, cache=cache_this_bundle)
codabench-compute_worker-1  |   File "/app/compute_worker.py", line 457, in _get_bundle
codabench-compute_worker-1  |     urlretrieve(url, bundle_file)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 239, in urlretrieve
codabench-compute_worker-1  |     with contextlib.closing(urlopen(url, data)) as fp:
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen
codabench-compute_worker-1  |     return opener.open(url, data, timeout)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 517, in open
codabench-compute_worker-1  |     response = self._open(req, data)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open
codabench-compute_worker-1  |     result = self._call_chain(self.handle_open, protocol, protocol +
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
codabench-compute_worker-1  |     result = func(*args)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open
codabench-compute_worker-1  |     return self.do_open(http.client.HTTPConnection, req)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open
codabench-compute_worker-1  |     raise URLError(err)
codabench-compute_worker-1  | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

I even recreated the local instance from scratch after the MinIO upgrade PR.

[ObadaS]

It's working fine on my end, I think this might be a MacOS specific problem.
You could try to do a curl -v http://docker.for.mac.localhost:9000 and then a curl -v of the failing URL (must be done after a fresh failure as the URL has a time limit) to see what kind of error you guys get, it could help us debug it

[Didayolo]

~ codabench % curl -v http://docker.for.mac.localhost:9000
* Host docker.for.mac.localhost:9000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9000...
* Connected to docker.for.mac.localhost (::1) port 9000
> GET / HTTP/1.1
> Host: docker.for.mac.localhost:9000
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 254
< Content-Type: application/xml
< Server: MinIO
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Origin
< Vary: Accept-Encoding
< X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
< X-Amz-Request-Id: 1867A47CDFD91928
< X-Content-Type-Options: nosniff
< X-Ratelimit-Limit: 857
< X-Ratelimit-Remaining: 857
< X-Xss-Protection: 1; mode=block
< Date: Mon, 22 Sep 2025 15:30:35 GMT
< 
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host docker.for.mac.localhost left intact
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><RequestId>1867A47CDFD91928</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>%                                                                                                                                                                                                

[Didayolo]

Not sure what URL should be put there, tried the following one:

codabench-compute_worker-1  | [2025-09-22 15:33:32,150: INFO/MainProcess] Received task: compute_worker_run[33279a4c-4452-4d31-b963-a1538f39fae7]  
codabench-compute_worker-1  | [2025-09-22 15:33:32,192: INFO/ForkPoolWorker-1] Received run arguments: {'user_pk': 1, 'submissions_api_url': 'http://django:8000/api', 'secret': '7fa8129a-cbf5-4b42-8aaf-b2ad4a886134', 'docker_image': 'codalab/codalab-legacy:py39', 'execution_time_limit': 600, 'id': 6, 'is_scoring': False, 'prediction_result': 'http://docker.for.mac.localhost:9000/private/prediction_result/2025-09-22-1758555211/de4600033a46/prediction_result.zip?AWSAccessKeyId=testkey&Signature=Ty83SezvmA57taCVwAu9GK88qDM%3D&content-type=application%2Fzip&Expires=1758987211', 'ingestion_program': 'http://docker.for.mac.localhost:9000/private/dataset/2025-09-11-1757599157/a02044676b78/ingestion_program.zip?AWSAccessKeyId=testkey&Signature=z2p8k4RyK8TFGkqYTF12%2Bp16Bpw%3D&Expires=1758987211', 'input_data': 'http://docker.for.mac.localhost:9000/private/dataset/2025-09-11-1757599158/edb22a8df693/input_data.zip?AWSAccessKeyId=testkey&Signature=CZMvIk6%2BPvZnh0tL40i347uB0ak%3D&Expires=1758987211', 'ingestion_only_during_scoring': False, 'program_data': 'http://docker.for.mac.localhost:9000/private/dataset/2025-09-22-1758555207/0eb61d2a198c/sample_code_submission.zip?AWSAccessKeyId=testkey&Signature=6hLCuUWLzMtSw2N1RUZImo8PKLc%3D&Expires=1758987211', 'prediction_stdout': 'http://docker.for.mac.localhost:9000/private/submission_details/2025-09-22-1758555211/1f224c9fe7ad/prediction_stdout.txt?AWSAccessKeyId=testkey&Signature=%2BxUf9iVMpH%2BAiKbNDmyyTOJ0Ewk%3D&content-type=application%2Fzip&Expires=1758987211', 'prediction_stderr': 'http://docker.for.mac.localhost:9000/private/submission_details/2025-09-22-1758555211/c9e208bf5d0c/prediction_stderr.txt?AWSAccessKeyId=testkey&Signature=l9f03PeEIeJn%2BcsrT0QHyAC%2B%2Fy0%3D&content-type=application%2Fzip&Expires=1758987212', 'prediction_ingestion_stdout': 'http://docker.for.mac.localhost:9000/private/submission_details/2025-09-22-1758555212/95a0e10d0710/prediction_ingestion_stdout.txt?AWSAccessKeyId=testkey&Signature=PToRdWTB%2FKkG0PhaSftcFmWxPfk%3D&content-type=application%2Fzip&Expires=1758987212', 'prediction_ingestion_stderr': 'http://docker.for.mac.localhost:9000/private/submission_details/2025-09-22-1758555212/098780ce74b7/prediction_ingestion_stderr.txt?AWSAccessKeyId=testkey&Signature=oLkI0hT1PyBcMHtDgKmhrSmQMS4%3D&content-type=application%2Fzip&Expires=1758987212'}
codabench-compute_worker-1  | [2025-09-22 15:33:32,225: INFO/ForkPoolWorker-1] Updating submission @ http://django:8000/api/submissions/6/ with data = {'status': 'Preparing', 'status_details': None, 'secret': '7fa8129a-cbf5-4b42-8aaf-b2ad4a886134'}
codabench-compute_worker-1  | [2025-09-22 15:33:32,661: INFO/ForkPoolWorker-1] Submission updated successfully!
codabench-compute_worker-1  | [2025-09-22 15:33:32,666: INFO/ForkPoolWorker-1] Checking if cache directory needs to be pruned...
codabench-compute_worker-1  | [2025-09-22 15:33:32,668: INFO/ForkPoolWorker-1] Cache directory does not need to be pruned!
codabench-compute_worker-1  | [2025-09-22 15:33:32,668: INFO/ForkPoolWorker-1] Getting bundle http://docker.for.mac.localhost:9000/private/dataset/2025-09-22-1758555207/0eb61d2a198c/sample_code_submission.zip?AWSAccessKeyId=testkey&Signature=6hLCuUWLzMtSw2N1RUZImo8PKLc%3D&Expires=1758987211 to unpack @ program
codabench-compute_worker-1  | [2025-09-22 15:33:32,683: INFO/ForkPoolWorker-1] CODALAB_IGNORE_CLEANUP_STEP mode enabled, ignoring clean up of: /codabench/tmp4crtop5u
codabench-compute_worker-1  | [2025-09-22 15:33:32,689: ERROR/ForkPoolWorker-1] Task compute_worker_run[33279a4c-4452-4d31-b963-a1538f39fae7] raised unexpected: URLError(ConnectionRefusedError(111, 'Connection refused'))
codabench-compute_worker-1  | Traceback (most recent call last):
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open
codabench-compute_worker-1  |     h.request(req.get_method(), req.selector, req.data, headers,
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1285, in request
codabench-compute_worker-1  |     self._send_request(method, url, body, headers, encode_chunked)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
codabench-compute_worker-1  |     self.endheaders(body, encode_chunked=encode_chunked)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
codabench-compute_worker-1  |     self._send_output(message_body, encode_chunked=encode_chunked)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
codabench-compute_worker-1  |     self.send(msg)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 980, in send
codabench-compute_worker-1  |     self.connect()
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/http/client.py", line 946, in connect
codabench-compute_worker-1  |     self.sock = self._create_connection(
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/socket.py", line 856, in create_connection
codabench-compute_worker-1  |     raise err
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
codabench-compute_worker-1  |     sock.connect(sa)
codabench-compute_worker-1  | ConnectionRefusedError: [Errno 111] Connection refused
codabench-compute_worker-1  | 
codabench-compute_worker-1  | During handling of the above exception, another exception occurred:
codabench-compute_worker-1  | 
codabench-compute_worker-1  | Traceback (most recent call last):
codabench-compute_worker-1  |   File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 385, in trace_task
codabench-compute_worker-1  |     R = retval = fun(*args, **kwargs)
codabench-compute_worker-1  |   File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 650, in __protected_call__
codabench-compute_worker-1  |     return self.run(*args, **kwargs)
codabench-compute_worker-1  |   File "/app/compute_worker.py", line 112, in run_wrapper
codabench-compute_worker-1  |     run.prepare()
codabench-compute_worker-1  |   File "/app/compute_worker.py", line 808, in prepare
codabench-compute_worker-1  |     zip_file = self._get_bundle(url, path, cache=cache_this_bundle)
codabench-compute_worker-1  |   File "/app/compute_worker.py", line 457, in _get_bundle
codabench-compute_worker-1  |     urlretrieve(url, bundle_file)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 239, in urlretrieve
codabench-compute_worker-1  |     with contextlib.closing(urlopen(url, data)) as fp:
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen
codabench-compute_worker-1  |     return opener.open(url, data, timeout)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 517, in open
codabench-compute_worker-1  |     response = self._open(req, data)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open
codabench-compute_worker-1  |     result = self._call_chain(self.handle_open, protocol, protocol +
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
codabench-compute_worker-1  |     result = func(*args)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 1375, in http_open
codabench-compute_worker-1  |     return self.do_open(http.client.HTTPConnection, req)
codabench-compute_worker-1  |   File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open
codabench-compute_worker-1  |     raise URLError(err)
codabench-compute_worker-1  | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

~ codabench % curl -v http://docker.for.mac.localhost:9000/private/dataset/2025-09-22-1758555207/0eb61d2a198c/sample_code_submission.zip?AWSAccessKeyId=testkey&Signature=6hLCuUWLzMtSw2N1RUZImo8PKLc%3D
[1] 50163
zsh: no matches found: http://docker.for.mac.localhost:9000/private/dataset/2025-09-22-1758555207/0eb61d2a198c/sample_code_submission.zip?AWSAccessKeyId=testkey
[1]  + exit 1     curl -v                      

[ObadaS]

Can you get the UI if you open http://docker.for.mac.localhost:9000 in your browser ?

I get the same Access Denied error on my end, which is normal, when launching the curl -v http://docker.for.mac.localhost:9000 command (adapted for my machine, it's curl -v minio:9000)

[ihsaan-ullah]

Yes, I can access the UI

[Didayolo]

I remember reading somewhere that docker.for.mac.localhost was deprecated. Maybe we can try a new config.

---

EDIT - I'll try:
AWS_S3_ENDPOINT_URL=http://minio:9000/
and
AWS_S3_ENDPOINT_URL=http://host.docker.internal:9000

--> Then the MinIO access is even more bugous, the images / styles does not load fine