Refactor semaphore._Watch into general-purpose Deadline utility

[hendrikmakait]

This PR extracts some changes from #7184 into a stand-alone PR. The Deadline utility can be used in multiple places where timeout-based cancellation of pipelined operations is desired. For example, Worker.close currently uses its timeout to limit the duration of individual operations which makes it impossible to estimate (or limit) its own total duration.

Note

I'm not entirely sure where to best put this, distributed.utils feels overused and a dedicated new module feels like a bit too much for this. We may want to drop the private scope, but I'd leave it for now to avoid over-exposing API details.

• Tests added / passed
• Passes pre-commit run --all-files

cc @fjetter

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±  0         15 suites  ±0   6h 41m 21s ⏱️ - 1m 33s
  3 173 tests +  7    3 089 ✔️ +  8    84 💤 ±0  0 ❌  - 1 
23 476 runs  +52  22 571 ✔️ +48  905 💤 +5  0 ❌  - 1 

Results for commit 69d0cca. ± Comparison against base commit 5a14053.

♻️ This comment has been updated with latest results.

[fjetter]

The module name is a bit confusing. Cancellation is a bit too generic for my taste. I'm not sure if this thing warrants a dedicated module

[hendrikmakait]

I'm not sure if this thing warrants a dedicated module

Agreed, as noted in the description I'm not sure where to best put this. I'm not a huge fan of cramming everything into distributed.utils either but that might be the best spot for now. Any suggestions?