Add a warning when collecting 'async def' functions

[nicoddemus]

As discussed in #2175, while it does not seem possible at the moment to support asyncio functions out of the box because it would be implemented in detriment of other async frameworks, we should at the very least emit a warning when collecting awaitable functions.

[pawelad]

How do you see that working with mentioned plugins like pytest-asyncio and pytest-tornado?

I think the correct behaviour would be to not display the warning if the test was successfully marked by them, but as fas as I understand (and please correct me if I'm wrong) it would mean we have to keep a list of 'allowed' markers/plugins.

I would be happy to help with the implementation but just want to make sure that I understand what is the expected behaviour.

[nicoddemus]

Unless someone has a better idea, I think just checking for the known marks would be sufficient. We should even add to the warning message a recommendation for the user to check out pytest-asyncio or pytest-tornado for async goodness, so having a list of known marks doesn't seem problematic to me.

[pawelad]

If it's good enough for you then it's good enough for me : )

But do you have in mind some kind of a 'vetting' process or would you rather just allow all async helping plugins, no matter how active they are? In other words, where do you draw the line? I'm fine with there not being a line, but then again it won't be a list of 'recommended' plugins but a list of 'known' ones. And please do tell me if I'm overthinking this : -)

That said - any other suggestions for the list? From my quick research I also found pytest-tornasync and pytest-aiohttp (although they use fixtures instead of markers, which may complicate the solution a little).

[nicoddemus]

But do you have in mind some kind of a 'vetting' process or would you rather just allow all async helping plugins, no matter how active they are? In other words, where do you draw the line?

My initial reaction is to list all "known" plugins, except if one of them is marked as unmaintained by the author.

From my quick research I also found pytest-tornasync and pytest-aiohttp (although they use fixtures instead of markers, which may complicate the solution a little).

Hmmm perhaps a solution would be to look at the list of installed plugins instead, and when one of the known async plugins is installed we just suppress all warnings related to async tests.

[RonnyPfannschmidt]

can we please assume people can at least google and just say that they need a pytest plugin fitting the async framework they use

[RonnyPfannschmidt]

also the async plugins should handle the async tests

[nicoddemus]

can we please assume people can at least google and just say that they need a pytest plugin fitting the async framework they use

Fair enough.

also the async plugins should handle the async tests

They already do, I'm not sure I get your point...

[RonnyPfannschmidt]

then they should not see a warning

[nicoddemus]

then they should not see a warning

Do you mean that we should just emit a warning and assume that the plugins will override the appropriate hook (pytest_runtest_call) effectively silencing our warning?

[RonnyPfannschmidt]

correct

[nicoddemus]

Oh yeah you are right, no need for complicated checks then, which makes everything much easier.

@pawelad is this course of action clear or do you need some pointers?

Also, should this go into master or features? I believe it is harmless to add it to master, I don't see how it would break any test suite today (and if it breaks, it is because people are using async tests without a proper plugin installed).

[pawelad]

As far as I understand - no checking, no special conditions, just display the warning when collected test is an awaitable.

To be honest I don't really see how's that better, but then again I didn't go into code structure yet.