Incorrect return type for asyncio.gather?

[s-kovacevic]

asyncio.gather return type from tasks.pyi seem to be different from the actual runtime type.

example.py

import asyncio
from typing import List

async def five() -> int:
    return 5

async def runner() -> List[int]:
    fives = await asyncio.gather(five(), five(), five())
    print(type(fives))  # prints `<class 'list'>`
    print(fives)  # prints `[5, 5, 5]`
    return fives

loop = asyncio.get_event_loop()
result: List[int] = loop.run_until_complete(runner())

$  python3 --version
Python 3.6.6
$  python3 example.py               
<class 'list'>
[5, 5, 5]
$  mypy --version
mypy 0.641
$ mypy example.py                  
example.py:11: error: Incompatible return value type (got "Tuple[int, int, int]", expected "List[int]")

By looking at tasks.pyi I see no overload or any case where gather does not return a Tuple.

[srittau]

Looking at the implementation of gather() in both Python 3.5 and 3.7, it seems to always return a list future. The problem is that there is no way to annotate it to return a list with different types at each position. Philosophically, gather() should return a tuple.

All fixes I can see have other downsides. -> List[Any] and -> List[Union[_T1, _T2]] etc. lose type and length information. The latter also makes it cumbersome to retrieve values in the general case, where _T1 != _T2.

[s-kovacevic]

Can't think of any better solution right now. I feel like it is better to lose type information than have it wrong. Were there any similar issues in the past?

[DevilXD]

I've just ran into this one, and I agree that List[Any] would be a better solution here. I understand that it loses typing information, but it's better than having an incorrect type altogether. It seems that MyPy agrees to this, as the current return type as reported by reveal_type there is asyncio.Future[builtins.list[Any]].

[aawilson]

Is the base issue here that the return type of gather changed after Python 3.7, which means supporting the correct typing would break compatibility with those versions?

[aawilson]

Nevermind, I poked around about in tasks.pyi and I see that the problem I'm getting locally has to do with my interpreter version running LSP not being the version I expect, I see the issue.

[Avasam]

Would TypeVarTuple help with this now? https://peps.python.org/pep-0646/#implications
Once mypy supports it of course python/mypy#12280

[JelleZijlstra]

No, because TypeVarTuple can't be used to type a heterogeneous list.

[Avasam]

Edit: I misunderstood this issue for something else. The tuple to list issue by OP is still present. And the problem is fundamental enough that I don't see a viable fix that avoids the downsides already mentioned.

However, #9678 still improved things significantly where there's more possible workarounds that don't require casting or ignoring, but have a runtime cost:

# Hide the tuples behind an iterable
async def runner() -> list[int]:
    fives = await asyncio.gather(*[five(), five(), five()]) # not efficient, more of a demonstration for `Iterable[int]`
    return fives
    
# Explicitly convert to list now that type is inferred
async def runner() -> list[int]:
    fives = list(await asyncio.gather(five(), five(), five()))
    return fives

[JelleZijlstra]

To summarize, we have to make a tradeoff here:

• We could reflect that gather always returns a list, but then we have no way to precisely type e.g. a, b = await asyncio.gather(returns_int(), returns_str())
• Or we could get precise types for the above sample, but lie and pretend that it returns a tuple instead of a list

We've chosen the second option, and I think that's still the right tradeoff, since in my experience it's common to immediately unpack the return value of gather(). I propose to close this issue as there's nothing more we can do in typeshed.