Make CI happy again

[milesgranger]

xref: dask/dask-expr#945

Tried to split each 'fix' into it's own commit, I think some may need their own follow-up issue.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±    0      27 suites  ±0   10h 26m 24s ⏱️ + 1h 48m 13s
 4 048 tests  -     2   3 932 ✅ +  109    110 💤  -   3   6 ❌  - 108 
50 828 runs  +5 654  48 459 ✅ +6 274  2 336 💤 +180  33 ❌  - 800 

For more details on these failures, see this check.

Results for commit 1ccfad6. ± Comparison against base commit f595c9e.

This pull request removes 2 tests.

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]

♻️ This comment has been updated with latest results.

[crusaderky]

@fjetter @phofl can I have a secondo opinion on this? Not sure I understand why we don't have this task in dask-expr. Are we missing a line in distributed.scheduler.default-task-durations?

• XREF Ensure shuffle split default durations uses proper prefix #4991

[phofl]

There should be a split_task, e.g. the materialisation happens in TaskShuffle in _layer, but I have never look into how this is actually materialized

[crusaderky]

Same discussion for test_blocklist_shuffle_split

[phofl]

@hendrikmakait and I stumbled upon the root cause, dask-expr doesn't properly respect the context manager. I am preparing a fix at the moment, should be there in a minute

[phofl]

dask/dask-expr#958

[phofl]

pr is merged

[crusaderky]

@phofl I'm now seeing split-taskshuffle and split-stage. Are they both supposed to have 1us default-duration?

[phofl]

Yeah the naming is confusing and something we probably should fix, but it's ok conceptually

[crusaderky]

Potential blocker.
Discussion on test_scheduler.py::test_default_task_duration_splits above.

[crusaderky]

I'm aiming to merge this PR with these tests still failing, as they are more involved:

• test_dataframe_set_index_sync (test_dask_collections.py::test_dataframe_set_index_sync passes locally, fails on CI #8561)
• test_default_task_duration_splits (Mark fast tasks of tasks-based shuffle when dask-expr is enabled #8563)
• test_blocklist_shuffle_split (Mark fast tasks of tasks-based shuffle when dask-expr is enabled #8563)

[hendrikmakait]

Thanks, everyone! I agree with @crusaderky that we should go ahead and merge if nothing else pops up on CI.