`test_stress_scatter_death` fails with state machine AssertionError

[fjetter]

https://github.com/dask/distributed/runs/5837063883?check_suite_focus=true

2022-04-06 00:06:59,109 - distributed.worker - ERROR - 
Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3894, in validate_task
    self.validate_task_fetch(ts)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3836, in validate_task_fetch
    assert ts.who_has
AssertionError
2022-04-06 00:06:59,111 - distributed.core - ERROR - Invalid TaskState encountered for <TaskState 'slowadd-1-23' fetch>.
Story:
[('slowadd-1-23', 'ensure-task-exists', 'released', 'compute-task-1649203619.045564', 1649203619.050943), ('slowadd-1-23', 'released', 'fetch', 'fetch', {}, 'compute-task-1649203619.045564', 1649203619.051032), ('gather-dependencies', 'tcp://127.0.0.1:61560', {'slowadd-1-23'}, 'ensure-communicating-1649203619.0522308', 1649203619.052306), ('slowadd-1-23', 'fetch', 'flight', 'flight', {}, 'ensure-communicating-1649203619.0522308', 1649203619.05233), ('request-dep', 'tcp://127.0.0.1:61560', {'slowadd-1-23'}, 'ensure-communicating-1649203619.0522308', 1649203619.05341), ('receive-dep', 'tcp://127.0.0.1:61560', {'slowadd-1-23'}, 'ensure-communicating-1649203619.0522308', 1649203619.064134), ('slowadd-1-23', 'put-in-memory', 'ensure-communicating-1649203619.0522308', 1649203619.064231), ('slowadd-1-23', 'flight', 'memory', 'memory', {'slowadd-2-22': 'ready'}, 'ensure-communicating-1649203619.0522308', 1649203619.064246), ('slowadd-1-23', 'compute-task', 'compute-task-1649203619.057579', 1649203619.0708082), ('remove-replicas', ('slowadd-1-23',), 'ensure-communicating-1649203619.0522308', 1649203619.089152), ('slowadd-1-23', 'remove-replica-confirmed', 'ensure-communicating-1649203619.0522308', 1649203619.08917), ('slowadd-1-23', 'release-key', 'ensure-communicating-1649203619.0522308', 1649203619.089201), ('slowadd-1-23', 'memory', 'released', 'released', {'slowadd-1-23': 'forgotten'}, 'ensure-communicating-1649203619.0522308', 1649203619.0893), ('slowadd-1-23', 'released', 'forgotten', 'forgotten', {}, 'ensure-communicating-1649203619.0522308', 1649203619.089324), ('slowadd-1-23', 'ensure-task-exists', 'released', 'compute-task-1649203619.0898051', 1649203619.109652), ('slowadd-1-23', 'released', 'fetch', 'fetch', {}, 'compute-task-1649203619.0898051', 1649203619.1097448)]
Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3894, in validate_task
    self.validate_task_fetch(ts)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3836, in validate_task_fetch
    assert ts.who_has
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/core.py", line 625, in handle_stream
    handler(**merge(extra, msg))
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 1895, in handle_compute_task
    self.transitions(recommendations, stimulus_id=stimulus_id)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 2579, in transitions
    self.validate_task(ts)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3904, in validate_task
    raise AssertionError(
AssertionError: Invalid TaskState encountered for <TaskState 'slowadd-1-23' fetch>.
Story:
[('slowadd-1-23', 'ensure-task-exists', 'released', 'compute-task-1649203619.045564', 1649203619.050943), ('slowadd-1-23', 'released', 'fetch', 'fetch', {}, 'compute-task-1649203619.045564', 1649203619.051032), ('gather-dependencies', 'tcp://127.0.0.1:61560', {'slowadd-1-23'}, 'ensure-communicating-1649203619.0522308', 1649203619.052306), ('slowadd-1-23', 'fetch', 'flight', 'flight', {}, 'ensure-communicating-1649203619.0522308', 1649203619.05233), ('request-dep', 'tcp://127.0.0.1:61560', {'slowadd-1-23'}, 'ensure-communicating-1649203619.0522308', 1649203619.05341), ('receive-dep', 'tcp://127.0.0.1:61560', {'slowadd-1-23'}, 'ensure-communicating-1649203619.0522308', 1649203619.064134), ('slowadd-1-23', 'put-in-memory', 'ensure-communicating-1649203619.0522308', 1649203619.064231), ('slowadd-1-23', 'flight', 'memory', 'memory', {'slowadd-2-22': 'ready'}, 'ensure-communicating-1649203619.0522308', 1649203619.064246), ('slowadd-1-23', 'compute-task', 'compute-task-1649203619.057579', 1649203619.0708082), ('remove-replicas', ('slowadd-1-23',), 'ensure-communicating-1649203619.0522308', 1649203619.089152), ('slowadd-1-23', 'remove-replica-confirmed', 'ensure-communicating-1649203619.0522308', 1649203619.08917), ('slowadd-1-23', 'release-key', 'ensure-communicating-1649203619.0522308', 1649203619.089201), ('slowadd-1-23', 'memory', 'released', 'released', {'slowadd-1-23': 'forgotten'}, 'ensure-communicating-1649203619.0522308', 1649203619.0893), ('slowadd-1-23', 'released', 'forgotten', 'forgotten', {}, 'ensure-communicating-1649203619.0522308', 1649203619.089324), ('slowadd-1-23', 'ensure-task-exists', 'released', 'compute-task-1649203619.0898051', 1649203619.109652), ('slowadd-1-23', 'released', 'fetch', 'fetch', {}, 'compute-task-1649203619.0898051', 1649203619.1097448)]

Dump test_stress_scatter_death.yaml.zip