Default dashboard address configurable

[fjetter]

This complements #6591

The default port for the dashboard or http server is hard coded to 8787. If this port is available, great. However, if a test doesn't properly cleanup after itself all following tests that are hard coded against this port for whatever reason are doomed to fail. We have a couple of these cascading failures on CI lately

This PR introduces a couple of changes

• Allow the default port to be configurable. This is mostly useful for tests
• Set the default port to :0 such that all tests that don't really care about it use the kernel to get a suitable port w/out risk of collision
• I extended our test setup with a couple of fixtures to allow certain tests to override this, e.g. if they specifically want to test against the dashboard port they can use open_port. I didn't want to use open_port for all tests. Maybe that's paranoid?
• our magical popen will now automatically forward the local environment and respect config changes to dask.config by creating a new config file for that process

If we do not wan to have this kind of fixture magic in here, I can simplify this to use open_port all the time.

[fjetter]

One of these cascades https://github.com/dask/distributed/runs/6972028225?check_suite_focus=true

FAILED distributed/cli/tests/test_dask_scheduler.py::test_signal_handling[Signals.SIGINT]
FAILED distributed/cli/tests/test_dask_scheduler.py::test_signal_handling[Signals.SIGTERM]
FAILED distributed/cli/tests/test_dask_worker.py::test_scheduler_file[--nanny]
FAILED distributed/cli/tests/test_dask_worker.py::test_scheduler_file[--no-nanny]
FAILED distributed/cli/tests/test_tls_cli.py::test_basic - distributed.comm.c...
FAILED distributed/cli/tests/test_tls_cli.py::test_nanny - distributed.comm.c...
FAILED distributed/cli/tests/test_tls_cli.py::test_separate_key_cert - distri...
FAILED distributed/cli/tests/test_tls_cli.py::test_use_config_file - distribu...
FAILED distributed/comm/tests/test_ws.py::test_expect_scheduler_ssl_when_sharing_server
FAILED distributed/comm/tests/test_ws.py::test_large_transfer_with_no_compression
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[True-ws://-None-8787]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[True-wss://-True-8787]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[False-ws://-None-8787]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[False-wss://-True-8787]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[True-ws://-None-8786]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[True-wss://-True-8786]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[False-ws://-None-8786]
FAILED distributed/comm/tests/test_ws.py::test_http_and_comm_server[False-wss://-True-8786]
FAILED distributed/deploy/tests/test_local.py::test_local_tls[True] - Runtime...
FAILED distributed/deploy/tests/test_local.py::test_local_tls[False] - Runtim...
FAILED distributed/deploy/tests/test_local.py::test_local_tls_restart - Runti...
FAILED distributed/deploy/tests/test_local.py::test_cluster_info_sync - Attri...
FAILED distributed/deploy/tests/test_local.py::test_cluster_info_sync_is_robust_to_network_blips
FAILED distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[True-None]
FAILED distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[True-127.0.0.1]
FAILED distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[False-None]
FAILED distributed/deploy/tests/test_local.py::test_cluster_host_used_throughout_cluster[False-127.0.0.1]
FAILED distributed/deploy/tests/test_local.py::test_connect_to_closed_cluster
FAILED distributed/deploy/tests/test_local.py::test_localcluster_start_exception
FAILED distributed/tests/test_preload.py::test_web_preload - UserWarning: Por...
FAILED distributed/tests/test_preload.py::test_web_preload_worker - UserWarni...
ERROR distributed/cli/tests/test_dask_worker.py::test_scheduler_file[--nanny]
ERROR distributed/cli/tests/test_dask_worker.py::test_scheduler_file[--no-nanny]

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   10h 23m 22s ⏱️ - 35m 58s
  2 890 tests ±0    2 798 ✔️  -   5    83 💤 ±0    8 ❌ +  5  1 🔥 ±0 
21 408 runs  ±0  20 381 ✔️  - 57  968 💤 +4  56 ❌ +51  3 🔥 +2 

For more details on these failures and errors, see this check.

Results for commit af462c4. ± Comparison against base commit 4779f84.

♻️ This comment has been updated with latest results.