WIP Rework control flow for missing tasks

[fjetter]

This reworks the control flow for missing tasks which should get rid of all assert ts.who_has issues. The essence is that all _to_fetch transitions will check on who_has and transition to missing instead. Combined with the check in the exception handler of gather_dep AND the guard in ensure_communicating, this should plug all holes where this could be introduced 🤞

It also includes #6248

Locally, the test_chaos_rechunk hasn't failed in 100 tries (running concurrently) (This already includes the fail_hard decorator)

I would like to get my other PRs merged that modify the state machine and revisit this afterwards

[github-actions]

Unit Test Results

       16 files  ±  0         16 suites  ±0   7h 13m 50s ⏱️ - 9m 33s
  2 745 tests +  2    2 662 ✔️ +  3       80 💤  - 1  3 ❌ ±0 
21 861 runs  +16  20 812 ✔️ +20  1 043 💤  - 5  6 ❌ +1 

For more details on these failures, see this check.

Results for commit 9682b97. ± Comparison against base commit 70e1fca.

[mrocklin]

For reference, on my computer test_chaos_rechunk would fail every ~20 iterations. We're going to need to find a way to amp up the chaos soon.

…

On Fri, Apr 29, 2022 at 11:44 AM Florian Jetter ***@***.***> wrote: This reworks the control flow for missing tasks which should get rid of all assert ts.who_has issues. The essence is that all _to_fetch transitions will check on who_has and transition to missing instead. Combined with the check in the exception handler of gather_dep AND the guard in ensure_communicating, this should plug all holes where this could be introduced 🤞 It also includes #6248 <#6248> Locally, the test_chaos_rechunk hasn't failed in 100 tries (running concurrently) (This already includes the fail_hard decorator) I would like to get my other PRs merged that modify the state machine and revisit this afterwards ------------------------------ You can view, comment on, or merge this pull request online at: #6251 Commit Summary - 05c3929 <05c3929> Fix recommendation ordering in transition_memory_released - a03c5c6 <a03c5c6> fixup! Fix recommendation ordering in transition_memory_released - e095b79 <e095b79> Ensure intermediate release transitions will not forget tasks - d332e6a <d332e6a> Allow cancelled missing transition (to released) - 9682b97 <9682b97> Rework missing transitions File Changes (4 files <https://github.com/dask/distributed/pull/6251/files>) - *M* distributed/scheduler.py <https://github.com/dask/distributed/pull/6251/files#diff-bbcf2e505bf2f9dd0dc25de4582115ee4ed4a6e80997affc7b22122912cc6591> (12) - *M* distributed/tests/test_worker.py <https://github.com/dask/distributed/pull/6251/files#diff-ee66fe9141ad4e11edb332284757324f3026e7126c39b881aa7d637224b4c92f> (7) - *M* distributed/tests/test_worker_state_machine.py <https://github.com/dask/distributed/pull/6251/files#diff-56e1acc8ea64d9ab2fc573fd8c53fe0bf3afa1ac06986feed27670a1e9d5fefb> (47) - *M* distributed/worker.py <https://github.com/dask/distributed/pull/6251/files#diff-0c7375c95aaa92986f8ed5460bae4e98720a14fca13556e01fd44ab6b927c24d> (54) Patch Links: - https://github.com/dask/distributed/pull/6251.patch - https://github.com/dask/distributed/pull/6251.diff — Reply to this email directly, view it on GitHub <#6251>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTHL3GXVADF3U5APXTLVHQGXNANCNFSM5UWT3BDA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[fjetter]

For reference, on my computer test_chaos_rechunk would fail every ~20
iterations.

We're going to need to find a way to amp up the chaos soon.

Can you provide a traceback or smth like that?

[mrocklin]

About a week ago a bunch of messages were sent out from my account to ancient issues. I think that github was having some issue at the time. If there are messages from me at around that time that seem oddly timed then please ignore.