Tests fail to run when pytest-asyncio>0.14.0 installed in environment

[jacobtomlinson]

A while ago distributed dropped pytest-asyncio in favour of gen_test (see #6050). Other projects in the Dask ecosystem continue to use pytest-asyncio and I've noticed recently that having pytest-asyncio>0.14.0 simply installed in the conda env causes the distributed tests to fail.

Error when running tests

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/_pytest/main.py", line 264, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/_pytest/config/__init__.py", line 992, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pluggy/hooks.py", line 308, in call_historic
INTERNALERROR>     res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 191, in pytest_configure
INTERNALERROR>     config.issue_config_time_warning(LEGACY_MODE, stacklevel=2)
INTERNALERROR>   File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1339, in issue_config_time_warning
INTERNALERROR>     warnings.warn(warning, stacklevel=stacklevel)
INTERNALERROR> DeprecationWarning: The 'asyncio_mode' default value will change to 'strict' in future, please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.

I typically have a development conda env with many Dask projects installed from source. Something must've bumped the pytest-asyncio version and with the pin in distributed gone I have to manually downgrade.

Once quick fix for this seems to be to add asyncio_mode=strict to the pytest section in setup.cfg. But that's a little frustrating given that we don't even use it here.

Another option is to go through the other Dask subprojects and remove pytest-asyncio in favour of something else. But that does open up a bunch of additional work.

Given that @fjetter and @graingert made this change I'm keen to get your advice on a good course of action here? #6050 and #6063 are pretty information light so it would be good to know what the issues were that caused it to be removed because I expect the same problems to be happening in other projects. Is it ok to rely on gen_test in other projects that have distributed as a dependency?

[fjetter]

I tried to remove the pinning once for distributed but failed to do so for different reasons #4858
I haven't looked into it since. For distributed this was an easy call since our own gen_test would provide similar functionality.
FWIW I think this is not a great user experience to raise if a config option is not set when it wasn't necessary before.

Once quick fix for this seems to be to add asyncio_mode=strict to the pytest section in setup.cfg. But that's a little frustrating given that we don't even use it here.

I'm open to setting this config in our repo if that alleviates any pain but I do not know if pytest-asyncio would still mess with our asyncio setup regardless.

Maybe @graingert has an answer for this?

[fjetter]

Is it ok to rely on gen_test in other projects that have distributed as a dependency?

I think other projects are using gen_cluster as well. I consider gen_cluster and gen_test on the same level of availability.
I would phrase it roughly like, public for developers with an asterisk indicating that we may change stuff occasionally for the better and we'd expect developers to adjust to small changes.

[graingert]

I'm happy to see asyncio_mode=strict set until a version of pytest-asyncio releases with strict mode enabled by default

DeprecationWarning: The 'asyncio_mode' default value will change to 'strict' in future, please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.

we can also use addopts = -p no:asyncio which will also work with anyone using --strict-config

[jacobtomlinson]

Thanks both! I like the idea of just disabling the plugin so I'll raise a PR to do that.

I'll also explore migrating away from it in other projects, would be good to be consistent with distributed to reduce maintenance overhead.

[leycec]

Necrobump. Unrelatedly, I just hit this in @beartype. The newest version of the pytest-asyncio plugin now insanely imports your entire test suite at early test collection time. Since this includes Python submodules not prefixed by test_*, pytest-asyncio now breaks all test suites that contain Python submodules that are:

• Not prefixed by test_*.
• Not intended to be imported at early test collection time.

In short, pytest-asyncio is insane. addopts = -p no:asyncio is the answer. Thanks so much for this issue discussion, which directed me directly to a trivial solution. Guido save us from pytest plugins. They are the road to QA Hell.