Remote workers not receiving submissions (private queue)

[johanneskruse]

Hi,

I'm running a competition and my remote workers has stopped receving any submissions.

I don't have any error logs from them. I might be related to #1446.

Best,
Johannes

[johanneskruse]

The public queue seems to be working; but I cannot use it for our competition because of the run time limit.

[johanneskruse]

I am now getting the following output logs:

$ docker logs -f compute_worker
/usr/local/lib/python3.8/site-packages/celery/platforms.py:800: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
 
 -------------- compute-worker@efe5103be5d4 v4.4.0 (cliffs)
--- ***** ----- 
-- ******* ---- Linux-5.15.0-1062-aws-x86_64-with-glibc2.34 2024-06-09 19:02:33
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x7f7a65017a30
- ** ---------- .> transport:   amqp://63a35e45-cb28-4eed-9c2c-af8072bf9d9c:**@www.codabench.org:5672/4d4c54a3-6e7f-418a-8d6b-36345ea8b37c
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> compute-worker   exchange=compute-worker(direct) key=compute-worker
                

[tasks]
  . compute_worker_run

[2024-06-09 19:02:33,405: INFO/MainProcess] Connected to amqp://63a35e45-cb28-4eed-9c2c-af8072bf9d9c:**@www.codabench.org:5672/4d4c54a3-6e7f-418a-8d6b-36345ea8b37c
[2024-06-09 19:02:33,523: INFO/MainProcess] mingle: searching for neighbors
[2024-06-09 19:02:34,879: INFO/MainProcess] mingle: all alone
[2024-06-09 19:02:35,079: CRITICAL/MainProcess] Unrecoverable error: PreconditionFailed(406, "PRECONDITION_FAILED - inequivalent arg 'x-max-priority' for queue 'compute-worker' in vhost '4d4c54a3-6e7f-418a-8d6b-36345ea8b37c': received the value '10' of type 'signedint' but current is none", (50, 10), 'Queue.declare')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/tasks.py", line 40, in start
    c.task_consumer = c.app.amqp.TaskConsumer(
  File "/usr/local/lib/python3.8/site-packages/celery/app/amqp.py", line 301, in TaskConsumer
    return self.Consumer(
  File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 386, in __init__
    self.revive(self.channel)
  File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 408, in revive
    self.declare()
  File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 421, in declare
    queue.declare()
  File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 611, in declare
    self._create_queue(nowait=nowait, channel=channel)
  File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 620, in _create_queue
    self.queue_declare(nowait=nowait, passive=False, channel=channel)
  File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 648, in queue_declare
    ret = channel.queue_declare(
  File "/usr/local/lib/python3.8/site-packages/amqp/channel.py", line 1148, in queue_declare
    return queue_declare_ok_t(*self.wait(
  File "/usr/local/lib/python3.8/site-packages/amqp/abstract_channel.py", line 88, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 508, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 514, in blocking_read
    return self.on_inbound_frame(frame)
  File "/usr/local/lib/python3.8/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.8/site-packages/amqp/connection.py", line 520, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
  File "/usr/local/lib/python3.8/site-packages/amqp/abstract_channel.py", line 145, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.8/site-packages/amqp/channel.py", line 279, in _on_close
    raise error_for_code(
amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'x-max-priority' for queue 'compute-worker' in vhost '4d4c54a3-6e7f-418a-8d6b-36345ea8b37c': received the value '10' of type 'signedint' but current is none

[Didayolo]

Hi,

It seems to be a general problem. I am not able to compute submissions on my custom queue.

The difference is that I just have not logs in my case.

[Didayolo]

@ObadaS Do you think this problem is happening since the upgrade of Caddy?

It is a very concerning issue, we need to solve it asap.

[ObadaS]

@ObadaS Do you think this problem is happening since the upgrade of Caddy?

Caddy is not used for workers since it only supports the HTTP(s) protocol and not AMQP that RabbitMQ uses, so I don't think it's connected to the Caddy upgrade

[ObadaS]

@johanneskruse Hello, have you tried creating a new worker on a fresh machine/virtual machine?

Also, just to make sure, you are using this command to launch the worker, without changing anything ?

docker run \
    -v /codabench:/codabench \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -d \
    --env-file .env \
    --name compute_worker \
    --restart unless-stopped \
    --log-opt max-size=50m \
    --log-opt max-file=3 \
    codalab/competitions-v2-compute-worker:latest 

[Didayolo]

@ObadaS I attached the worker wk8 to a custom queue named "TMP", and I confirm this is not working.

I added you as an admin of the queue if you want to do some tests.

[ObadaS]

@johanneskruse The problem should be fixed.

@Didayolo I got some logs that we can analyze later for this weird problem. I tried restarting the RabbitMQ container alone, but it did not fix the problem. It only got fixed after I restart all the containers (but I suspect that the problem might be coming from the site_worker container, we can try restarting it only, if the problem happens again in the future)

[johanneskruse]

@ObadaS - I can confirm that it is indeed working on my end.

Thank you both for your swift actions!

[johanneskruse]

Hi @Didayolo and @ObadaS.

The remote workers are now receiving jobs and running; however, before, a job was finished within 3-4 hours. Now jobs are running for more than 8 hours, and some haven't even stopped yet. Nothing has changed on my end. Is there any explanation or something that might fix this?