the "distributed.worker" logger gains ~thousand DequeHandler instances during a pytest run

[graingert]

running len(logging.getLogger("distributed.worker").handlers) shows a few thousand DequeHandler instances in there.
This means that logging messages from the worker or adding handlers to the worker gradually gets slower and slower as the test suite progresses.

the handlers are supposed to be cleared when the worker is GC'd by

distributed/distributed/node.py

Line 88 in 1da5199

weakref.finalize(self, logger.removeHandler, self._deque_handler)

however

distributed/distributed/utils.py

Lines 1131 to 1142 in 1da5199

	@functools.lru_cache(None)
	def _iscoroutinefunction_cached(f):
	return _iscoroutinefunction(f)
	def iscoroutinefunction(f):
	# Attempt to use lru_cache version and fall back to non-cached version if needed
	try:
	return _iscoroutinefunction_cached(f)
	except TypeError: # unhashable type
	return _iscoroutinefunction(f)

keeps a permanent strong reference to the worker:

see #4481 and #4469

[graingert]

the reference chain is from objgraph.find_backref_chain https://mg.pov.lt/objgraph/#memory-leak-example

[fjetter]

If the performance problem is an actual concern, ensuring all handlers are async is probably the most straight forward and would resolve a few other issues around exception handling

some earlier and related work

• RFC Allow a loop cycle after every message in the batch #5443
• Ensure exceptions in handlers are handled equally for sync and async #4734
• Is it intended that any error from a handler makes Server.handle_stream close the comm? #5483