Mock process memory readings in test_worker.py (v2)

[crusaderky]

In scope

• Change all tests in test_worker.py that rely on process memory readings to use a deterministic, robust and fast mock design instead
• Replace fragile timing-based test design with events.
• Supersedes Work around flakyness in spill hysteresis #5850
• Supersedes Mock process memory readings in test_worker.py #5870
• Mitigates Memory may not shrink fast enough #5840
• Closes test_spill_hysteresis flaky on ubuntu #5848
• Partially reverts Allow memory monitor to evict data more aggressively #3424

Out of scope

• same treatment to test_scheduler.py::test_memory
• remediation to test_active_memory_manager.py, which should be carried out by adding a switch to use managed memory
• remediation to all tests around rebalance(): postponed indefinitely due to the whole function to be reimplemented on top of the Active Memory Manager

[github-actions]

Unit Test Results

       12 files  ±0         12 suites  ±0   7h 24m 25s ⏱️ - 3m 58s
  2 621 tests +1    2 540 ✔️ +3    80 💤 ±0  1 ❌  - 2 
15 650 runs  +6  14 784 ✔️ +5  864 💤 +2  2 ❌  - 1 

For more details on these failures, see this check.

Results for commit 3484f0c. ± Comparison against base commit fb8484e.

[fjetter]

I think these tests are much more readable and easier to maintain, thank you!

[fjetter]

This is a bit fragile. As soon as the while loop in the memory monitor changes anything about how it awaits, e.g. another async call to evict data, this condition will break since we rely on skipping exactly one loop iteration with sleep(0).

I can't come up with a nicer way other than sleeping for a longer time so this is fine for me. I just wanted to raise this in case somebody else thinks of something

[fjetter]

Another ugly thing we can do but I think I dislike this even more is to wait for _memory_monitoring to toggle since then we know for sure the memory_monitor is done. I think I prefer sticking with sleep(0) and deal with a test failure/test rewrite if that ever happens