Run without module detection logic

[tuukkamustonen]

Feature

Allow mypy --no-module-detection (or similar flag) to run without mypy attempting to map files as modules. For example, linting standalone scripts should not require module detection magic.

Pitch

Mypy has (somewhat confusing?) file-to-module mapping logic explained in https://mypy.readthedocs.io/en/latest/running_mypy.html#mapping-file-paths-to-modules. The key take away is that mypy attempts to determine a module name for each imported file (in various ways). For example, a/b/c.py -> a.b.c.

However, the documentation doesn't give rationale for why this actually needs to be always done. If you merely want to check standalone files like mypy one.py then why does this require determining module name for the file?

(AFAIK, other linters like pylint don't have this kind of magic, perhaps because they don't follow imports like mypy does.)

If we have two files mypy one.py two.py and two imports one, then the detection logic is needed, so that module one is found (and it's fine).

However, if I have sub-directories with similarly named files, doing mypy one/file.py two/file.py throws:

two/file.py: error: Duplicate module named 'file' (also at 'one/file.py')

But these are two standalone files, in separate directories, not part of the same package. Why is the error there?

Similar trouble is faced when maintaining a monorepo with multiple packages:

package1/
    setup.py
    ...
package2/
    setup.py
   ...

Mypy will throw error about duplicate module name on setup.py files (and others).

Now, without arguing on how mypy should determine module name of the files is looks up (surely there's reasoning for that), I'm suggesting to add an option to completely skip this magic. In my case, I always pip install -e . the packages that I'm developing, so whatever imports the code (wherever), those imports are already resolvable. All the packages are already importable and whatever files reside out of the packages, are stand-alone scripts and never imported.

One downside is that something like this would no longer work:

package1/
    src/
        ...
    setup.py
    tasks.py  # imports util.py
    util.py

For that I don't have suggestion :( Could it be solved by multiple entries in MYPYPATH or PYTHONPATH?

[atrigent]

Does using --namespace-packages help with this?

[tuukkamustonen]

I'm not using namespace packages so no.

And mypy nags about duplicate modules Even with MYPYPATH=pkg1,pkg2 mypy --namespace-packages --explicit-package-bases ..

[ns-dmichelini]

I'm having the same issue. I'm working with a mono repo with multiple packages and getting duplicate module errors because of the same situation as your example above with the 2 setup.py's. Have you found a workaround or a way to scan these separately in the config file?

[hauntsaninja]

You could try --exclude '/setup\.py$' if you don't care about checking your setup.py's. Alternatively, something like MYPYPATH=pkg1:pkg2 mypy --namespace-packages --explicit-package-bases . could help (note that MYPYPATH is colon separated, which is maybe why it didn't work for @tuukkamustonen )

[tuukkamustonen]

Not sure if my earlier paste is a typo or if I actually misused MYPYPATH but trying it now with colon and there's no change - still getting the error.

[MattF-NSIDC]

My team has a case where we are using python scripts to generate a large, relatively-complex configuration tree. The directory structure and directory names of these scripts is a meaningful part of the configuration tree and that includes spaces in the directory names (ruling out making these directories in to packages without significant overhaul). Some of the scripts also have meaningful names, e.g. settings.py in multiple locations in the tree, so we get this "Duplicate module" error. We haven't found any way yet to typecheck this structure of configuration scripts alongside our project with mypy after digging through this and related issues (--scripts-as-modules and --namespace-packages don't help us). We really don't want to exclude any of these scripts from typechecking, but that has been our unhappy workaround so far.

Maybe we can pass the list of files in each directory to a separate call of mypy using a helper script? We're just surprised that there seems to be no escape hatch that enables us to typecheck a directory structure including one/foo.py and two/foo.py where one and two are not packages and shouldn't be made in to packages.

I feel a new flag, e.g. --skip-module-detection <dir> or --skip-module-detection <regex> makes sense to dodge this bullet for a subset of a project.

[erictraut]

I've faced mypy's "duplicate module" issue in a number of source bases, and I haven't found a good workaround short of renaming modules or telling mypy to ignore parts of the code base. I agree that a new mode would be really useful here. I'm not clear on why this limitation exists in the first place, since it's legal for a code base to have duplicate module names as long as they don't result in a namespace conflict. Perhaps someone who is familiar with the inner workings of mypy could explain why this check exists?

For what it's worth, pyright doesn't have this limitation, so it might suit your needs.

[MattF-NSIDC]

Interesting. Thanks for your input, though I would have appreciated disclosure that you are the author of Pyright ;) We tried Pyright out, and it was indeed able to typecheck parts of the codebase which mypy refused to check due to the issue in this thread. It's certainly strange adding Node as a development dependency to our project, on the flipside.

[MattF-NSIDC]

We tried out PyType because we liked that it could be installed with Python ecosystem tooling (pip), but it had the same problem with "duplicate" modules.

We landed on running mypy in 3 phases:

1. For the whole project, excluding the problem scripts directory (LAYERS_CFG_DIR in the code)
2. For the set of files with unique names in the problem directory, all at once.
3. For the set of files with "duplicate" names in the problem directory, one file at a time.

Now our entire project is being typechecked again! But we have to deal with dozens of runs of mypy. We would really like to achieve this in one run with a flag like --skip-module-detection-in-dir {LAYERS_CFG_DIR}.

Here's the monstrosity if you'd like to adapt it to your project:
https://github.com/nsidc/qgreenland/blob/700c9b9812dff4d2ee6827902572b1d59e76cc77/tasks/test.py#L46-L115