Change in behavior with __import__('pkgutil').extend_path() # type: Iterable[str]

[ericvw]

I noticed a change in behavior between versions 0.720 and 0.730 when using pkgutil-style namespace packages.

For the following __init__.py file:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

mypy == 0.720 will report:

$ mypy --version
mypy 0.720

$ mypy --show-traceback __init__.py
__init__.py:2: error: Cannot determine type of '__path__'

Thus, I "resolved" this with the following in, in 0.720:

from typing import Iterable
__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: Iterable[str]

With mypy ==0.730, I get the following:

$ mypy --version
mypy 0.730

mypy --show-traceback __init__.py
__init__.py:2: error: Module has no attribute "extend_path"
Found 1 error in 1 file (checked 1 source file)

When researching this issue, #7501 (comment) indicates that __import__ is not followed. However, I don't fully understand the change in behavior and had trouble using git-bisect to narrow it down.

[gvanrossum]

The root cause is in python/typeshed#3188, which was incorporated into mypy 0.730. Before that PR, __import__() was declared as returning Any. Now it's declared as ModuleType. While Any was to wide (we do know that it will return a module), ModuleType is probably too narrow, at least in case the argument is a literal.

I'm thinking that maybe this should be rolled back -- it doesn't make sense to try and do strict type checking on the module returned by __import__.

@srittau @ilevkivskyi @JukkaL Thoughts?

[ilevkivskyi]

I am fine either way. Short term we can set it back to Any and later probably add some special casing if the argument is a literal string or a final value (low priority probably).

[gvanrossum]

@ericvw If you can submit a PR to typeshed to change the return type of __import__ to Any (link to this issue) that would help.

[msullivan]

I think making it return Any is a good idea, since that is probably needed for most uses. I don't think that it is even close to worth trying to support this properly as a special case.

[msullivan]

So I'm going to close this issue, but I would merge a typeshed PR to change the return type of __import__.

[ericvw]

I have created python/typeshed#3383. Thanks!

[tigerjack]

I'm having the same behavior in my __init__.py file containing

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

and mypy 0.780

[UnexpectedLobster]

Same with mypy 0.782.
This is troublesome when working on ROS because every catkin packages uses the following template for python modules: https://github.com/ros/catkin/blob/noetic-devel/cmake/templates/__init__.py.in

[mfarrugi]

Is there a workaround for this that doesn't require annotating every __path__ reference?

[gvanrossum]

The last two comments seem to be about an unrelated issue. This issue is only about the type returned by ‘import’.

[mfarrugi]

Ah, looks like we're complaining specifically about __init__.py:2: error: Cannot determine type of '__path__' which fits into #1422.