Flaky `test_threadpoolworkers_pick_correct_ioloop`

[jrbourbeau]

distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop had a random failure over in #4677 (see this CI build).

Full traceback:

2021-04-07T03:42:22.3799290Z =================================== FAILURES ===================================
2021-04-07T03:42:22.3800010Z __________________ test_threadpoolworkers_pick_correct_ioloop __________________
2021-04-07T03:42:22.3800550Z 
2021-04-07T03:42:22.3801040Z cleanup = None
2021-04-07T03:42:22.3801420Z 
2021-04-07T03:42:22.3802080Z     def test_threadpoolworkers_pick_correct_ioloop(cleanup):
2021-04-07T03:42:22.3802760Z         # gh4057
2021-04-07T03:42:22.3803240Z     
2021-04-07T03:42:22.3803780Z         with dask.config.set(
2021-04-07T03:42:22.3804410Z             {
2021-04-07T03:42:22.3806610Z                 "distributed.scheduler.locks.lease-validation-interval": 0.01,
2021-04-07T03:42:22.3808540Z                 "distributed.scheduler.locks.lease-timeout": 0.05,
2021-04-07T03:42:22.3809450Z             }
2021-04-07T03:42:22.3809890Z         ):
2021-04-07T03:42:22.3810550Z             with Client(processes=False, threads_per_worker=4) as client:
2021-04-07T03:42:22.3811330Z                 sem = Semaphore(max_leases=1, name="database")
2021-04-07T03:42:22.3812020Z                 protected_ressource = []
2021-04-07T03:42:22.3812550Z     
2021-04-07T03:42:22.3813160Z                 def access_limited(val, sem):
2021-04-07T03:42:22.3813740Z                     import time
2021-04-07T03:42:22.3814200Z     
2021-04-07T03:42:22.3814670Z                     with sem:
2021-04-07T03:42:22.3815780Z                         assert len(protected_ressource) == 0
2021-04-07T03:42:22.3816940Z                         protected_ressource.append(val)
2021-04-07T03:42:22.3817610Z                         # Interact with the DB
2021-04-07T03:42:22.3818190Z                         time.sleep(0.1)
2021-04-07T03:42:22.3818840Z                         protected_ressource.remove(val)
2021-04-07T03:42:22.3819420Z     
2021-04-07T03:42:22.3820070Z >               client.gather(client.map(access_limited, range(10), sem=sem))
2021-04-07T03:42:22.3820630Z 
2021-04-07T03:42:22.3821230Z distributed/tests/test_semaphore.py:532: 
2021-04-07T03:42:22.3821850Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2021-04-07T03:42:22.3822460Z distributed/client.py:1975: in gather
2021-04-07T03:42:22.3823070Z     return self.sync(
2021-04-07T03:42:22.3823740Z distributed/client.py:843: in sync
2021-04-07T03:42:22.3824410Z     return sync(
2021-04-07T03:42:22.3825010Z distributed/utils.py:353: in sync
2021-04-07T03:42:22.3825630Z     raise exc.with_traceback(tb)
2021-04-07T03:42:22.3826280Z distributed/utils.py:336: in f
2021-04-07T03:42:22.3826860Z     result[0] = yield future
2021-04-07T03:42:22.3828160Z ../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/tornado/gen.py:762: in run
2021-04-07T03:42:22.3829060Z     value = future.result()
2021-04-07T03:42:22.3829700Z distributed/client.py:1840: in _gather
2021-04-07T03:42:22.3830430Z     raise exception.with_traceback(traceback)
2021-04-07T03:42:22.3831780Z ../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/dask/utils.py:34: in apply
2021-04-07T03:42:22.3832660Z     return func(*args, **kwargs)
2021-04-07T03:42:22.3833210Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2021-04-07T03:42:22.3833610Z 
2021-04-07T03:42:22.3834400Z val = 6, sem = <distributed.semaphore.Semaphore object at 0x7fd31e927280>
2021-04-07T03:42:22.3835110Z 
2021-04-07T03:42:22.3835640Z     def access_limited(val, sem):
2021-04-07T03:42:22.3836190Z         import time
2021-04-07T03:42:22.3836670Z     
2021-04-07T03:42:22.3837130Z         with sem:
2021-04-07T03:42:22.3837720Z >           assert len(protected_ressource) == 0
2021-04-07T03:42:22.3838300Z E           assert 1 == 0
2021-04-07T03:42:22.3838770Z E             +1
2021-04-07T03:42:22.3839550Z E             -0
2021-04-07T03:42:22.3839950Z 
2021-04-07T03:42:22.3840630Z distributed/tests/test_semaphore.py:526: AssertionError

cc @fjetter due to the Semaphore connection

[hristog]

@jrbourbeau, not that it's a big deal, but it seems to me there's a typo in the issue summary.

[jrbourbeau]

Thank for reading carefully @hristog. Do you mean the protected_ressource variable? Any interest in submitting a PR to fix the typo? (no obligation though)

[hristog]

Hi @jrbourbeau - I've raised #4689 to address the typo in protected_resource you've brought up.

I was referring to the typo in issue summary here ("Falky test_threadpoolworkers_pick_correct_ioloop") - thanks for updating it! I spotted it as I was catching up with the discussion in #4677 (and the associated chain of issues/PRs), where it appeared as referencing the latter. I hadn't been aware of the protected_resource typo at the time.