Tests can't run in parallel because of fixed ports

[QuLogic]

What happened:
Tests take a rather long time to run. So it would be nice to run them in parallel with pytest-xdist. However, they randomly fail to start, because they use the same port, and a second test is unable to open a port while another listens on a port. Thus you sometimes get a test error like:

OSError: [Errno 98] Address already in use

What you expected to happen:
Tests all run in parallel

Minimal Complete Verifiable Example:

pytest -n auto

Environment:

• Dask version: 2021.07.2
• Python version: 3.9.6
• Operating System: Fedora 34
• Install method (conda, pip, source): Source

[fjetter]

Locally I run the tests almost exclusively in parallel. There are a few isolated problematic tests but in general it works. Can you tell us which tests are not working for you?

In the past I thought about introducing pytest markers to disable tests which are known to not work for parallel execution, for whatever reason

Disclaimer: I often only run on the subset distributed/tests

That's not pretty but I actually wrote a small script to run exactly those tests. I wouldn't mind making this process a bit more streamlined if other folks are also using this.

#!/bin/bash

IGNORED="
not test_open_close_many_workers
and not rebalance
and not memory
and not upload
and not avoid_ci
and not test_recreate_error_array
and not test_open_close_many_workers
"
IGNORED=`echo $IGNORED | tr '\n' ' '`

pytest \
    -k "$IGNORED" \
    --runslow \
    -nauto \
    distributed/tests \
    -Wignore:Port \
    2>&1 \
    --timeout 60 \
    | tee test.log

[QuLogic]

I ran twice with -m 'not avoid_ci and not slow and not asyncio' -vra -n auto; this does 4 process parallelism.

First time failed on 18, and errored on 1:

ERROR distributed/deploy/tests/test_local.py::test_local_tls[True] - Assertio...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_dashboard - Excepti...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_defaults - OSError:...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_dashboard_whitelist
FAILED distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address[127.0.0.2---nanny]
FAILED distributed/cli/tests/test_dask_worker.py::test_nprocs_expands_name - ...
FAILED distributed/cli/tests/test_tls_cli.py::test_basic - distributed.comm.c...
FAILED distributed/deploy/tests/test_local.py::test_remote_access - OSError: ...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero
FAILED distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module
FAILED distributed/tests/test_scheduler.py::test_no_danglng_asyncio_tasks - A...
FAILED distributed/tests/test_versions.py::test_version_warning_in_cluster - ...
FAILED distributed/cli/tests/test_tls_cli.py::test_separate_key_cert - distri...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2
FAILED distributed/cli/tests/test_dask_worker.py::test_resources - AssertionE...
FAILED distributed/deploy/tests/test_local.py::test_local_tls[True] - OSError...
FAILED distributed/cli/tests/test_dask_worker.py::test_scheduler_file[--no-nanny]
FAILED distributed/cli/tests/test_dask_worker.py::test_contact_listen_address[tcp://0.0.0.0:39837---no-nanny]
FAILED distributed/cli/tests/test_tls_cli.py::test_nanny - distributed.comm.c...

Second time only failed 13, and none errored:

FAILED distributed/cli/tests/test_dask_scheduler.py::test_dashboard - Excepti...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_defaults - OSError:...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_pid_file - FileNotF...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_preload_module
FAILED distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default
FAILED distributed/cli/tests/test_dask_worker.py::test_scheduler_file[--nanny]
FAILED distributed/deploy/tests/test_local.py::test_remote_access - OSError: ...
FAILED distributed/cli/tests/test_dask_worker.py::test_nprocs_expands_name - ...
FAILED distributed/cli/tests/test_dask_worker.py::test_nprocs_negative - OSEr...
FAILED distributed/cli/tests/test_dask_worker.py::test_contact_listen_address[tcp://0.0.0.0:39837---no-nanny]
FAILED distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address[127.0.0.2---no-nanny]
FAILED distributed/cli/tests/test_tls_cli.py::test_nanny - distributed.comm.c...
FAILED distributed/comm/tests/test_ws.py::test_expect_scheduler_ssl_when_sharing_server

Some of the failures are 'address already in use'; some are timeouts (presumably a different test was blocking the port and then went away); a few just crash entirely

[fjetter]

note that we're also struggling with flaky tests in general. especially in parallel execution these seem to be more severe in my experience. Therefore, a few "random" failures are unfortunately expected every now and then

Looks like this issue affects mostly deploy and CLI tests. Would you be interested in rewriting some of the tests to use random/free ports? I can see a lot of hard coded URLs in these tests but I'm generally confident that this can be resolved

[QuLogic]

Would it make sense to have the scheduler accept -p 0 and then print out the real port on which it's listening?

[fjetter]

I know that when using kwargs, the scheduler accepts something like dashboard_address=":0" which does exactly that. I wouldn't be surprised if the CLI supports this indirectly already if provided with the correct input.

Just one example in out tests:

distributed/distributed/deploy/tests/test_adaptive.py

Lines 25 to 30 in 2801f11

	with LocalCluster(
	n_workers=0,
	silence_logs=False,
	dashboard_address=":0",
	loop=loop,
	) as cluster: