Replace worker use of `threading.local()` with ContextVar

[gjoseph92]

Context variables are now the recommended way to manage thread/context-local state:

Context managers that have state should use Context Variables instead of threading.local() to prevent their state from bleeding to other code unexpectedly, when used in concurrent code.
https://docs.python.org/3/library/contextvars.html

They work for threads, but as a bonus, also work for async code.

In many places in the worker (and even more in distributed in general), we use threading.local() variables currently. However, workers also support being used in an async mode (used in most tests), and increasingly support running async functions (#5151). The fact that thread-local variables can have the wrong value when used in an async context means there are likely lots of bugs around async mode. #4959 is just one example.

If we want to support async mode properly, switching Worker use of thread-local variables to ContextVars would be a good place to start. These sorts of bugs are generally very confusing and tedious to track down. It might be worth getting ahead of these bugs instead of repeatedly hunting them down.

As a bonus, we're inconsistent about ContextVars vs thread-local currently (for example, get_client checks a ContextVar, but get_worker checks a thread-local var). Consolidating on one interface might make future development smoother.

cc @jcrist @crusaderky @fjetter

[fjetter]

I often stumbled over incorrect async detection when using a Client due to this. This feels like it should be a more robust way of working with it?

[jrbourbeau]

Thanks for the suggestion @gjoseph92. It sounds like switching to using context variables will resolve some tricky issues that already exist and help prevent future ones from popping up. The fact that it's also explicitly recommended over threading.local() in the Python docs is a good signal too

[crusaderky]

+1 for replacing thread-local variables with contetxvars wherever possible.

[fjetter]

See also #6107 for a summary of an earlier attempt to introduce contextvars specifically for usage with stimulus_ids

[fjetter]

FYI This is soft-blocked by us still using tornado callbacks. Tornado callbacks are not dealing with ctxvars properly as pointed out in #6107 and we'd need to build our own handling for this. A refactoring like

• Replace loop.call_later and loop.add_callback with background tasks #6359
• and possibly replace gen.coroutine usage with async def #6221
  should be done first

[gjoseph92]

I'm surprised to hear that. I thought I found the opposite was the case in #5517.