0.971: Namespace packages -> Source file found twice under different module names

[pawamoy]

Bug Report

With mypy 0.961 I was able to scan my namespace packages without any issue. I've just upgraded today to 0.971, and I'm now getting a Source file found twice under different module names error.

$ pdm run mypy src
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

$ pdm run mypy src --namespace-packages
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

$ pdm run mypy src --namespace-packages --explicit-package-bases
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

$ pdm run mypy src/namespace
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

$ pdm run mypy src/namespace --namespace-packages
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

$ pdm run mypy src/namespace --namespace-packages --explicit-package-bases
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

To Reproduce

Create a namespace package and try to run mypy on it. I'll try to give an actual reproduction example tomorrow.

Expected Behavior

No "found twice" error.

Actual Behavior

"Found twice" error.

Your Environment

• Mypy version used: 0.971
• Mypy command-line flags:
• Mypy configuration options from mypy.ini (and other config files):

[mypy]
ignore_missing_imports = true
show_error_codes = true
warn_unused_ignores = true
warn_unused_configs = true

• Python version used: 3.8.11
• Operating system and version: Linux 3.10.0-862.6.3.el7.x86_64

Seems related to:

• "Source file found twice under different module names" with namespace package installed in virtualenv by "pip install -e ." #10172
• Mypy issue validating package with directory and -p arg #11234
• Support package and module in config files #10728
• ("unable to merge overloads" reported from site-packages #12162)
• (Namespace packages and __init__ imports #9716)

[hauntsaninja]

Thanks for reporting! I haven't yet tried to confirm, but do let me know if you have an example repro :-)

[pawamoy]

OK I got something 🙂

Setup:

pipx install pdm==1.15.4  # certainly possible to replicate with another build backend / project manager
mkdir repro-mypy
cd repro-mypy

cat <<EOF > pyproject.toml
[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

[project]
name = "namespace-package"
description = "My namespace package."
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.8"

[tool.pdm.build]
includes = ["src/namespace"]

[tool.pdm.dev-dependencies]
typing = ["mypy>=0.971"]
EOF

touch README.md
mkdir -p src/namespace/package
touch src/namespace/package/py.typed
echo "SOME_VAR: int = 0" > src/namespace/package/module.py
cat <<EOF > src/namespace/package/__init__.py
from namespace.package.module import SOME_VAR
__all__ = ["SOME_VAR"]
EOF

pdm install

Repro:

$ pdm run mypy src/ --verbose

LOG:  Mypy Version:           0.971
LOG:  Config File:            Default
LOG:  Configured Executable:  /home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/bin/python3
LOG:  Current Executable:     /home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/bin/python3
LOG:  Cache Dir:              .mypy_cache
LOG:  Compiled:               True
LOG:  Exclude:                []
LOG:  Found source:           BuildSource(path='src/namespace/package/__init__.py', module='package', has_text=False, base_dir='/home/user/repro-mypy/src/namespace')
LOG:  Found source:           BuildSource(path='src/namespace/package/module.py', module='package.module', has_text=False, base_dir='/home/user/repro-mypy/src/namespace')
LOG:  Could not load cache for package: package/__init__.meta.json
LOG:  Metadata not found for package
LOG:  Parsing src/namespace/package/__init__.py (package)
LOG:  Metadata fresh for package.module: file src/namespace/package/module.py
LOG:  Could not load cache for namespace.package.module: namespace/package/module.meta.json
LOG:  Metadata not found for namespace.package.module
LOG:  Parsing /home/user/repro-mypy/src/namespace/package/module.py (namespace.package.module)
LOG:  Build finished in 0.009 seconds with 2 modules, and 1 errors
src/namespace/package/module.py: error: Source file found twice under different module names: "package.module" and "namespace.package.module"
Found 1 error in 1 file (errors prevented further checking)

Have to go but will expand in the afternoon 🙂

[pawamoy]

Possibly not a bug but a change in behavior to be more strict.

I'm able to work around the above issue with the following:

MYPYPATH=src pdm run mypy src/ --verbose --namespace-packages --explicit-package-bases

Ideally mypy would be able to understand that the root package is namespace (not src nor package) thanks to sys.path, without the need for MYPYPATH=src and --namespace-packages --explicit-package-bases:

$ pdm run python
Python 3.8.11 (default, Jul  9 2021, 15:44:14) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print("\n".join(sys.path))

/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python38.zip
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8
/home/user/.basher-packages/pyenv/pyenv/versions/3.8.11/lib/python3.8/lib-dynload
/home/user/repro-mypy/__pypackages__/3.8/lib
/home/user/repro-mypy/src

(emphasis on /home/user/repro-mypy/src being in sys.path)

I think the workaround also means I have to run mypy several times with different options to be able to scan both my namespace package and other standalone scripts.

[pawamoy]

Ah, the verbose mode actually outputs helpful hints 🙂

src/namespace/package/module.py: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules for more info
src/namespace/package/module.py: note: Common resolutions include: a) adding `__init__.py` somewhere, b) using `--explicit-package-bases` or adjusting MYPYPATH

Didn't see those before 👍

[hauntsaninja]

Awesome, thank you for the thorough repro.

Yeah, we include an example of running mypy with namespace packages in a src layout here: https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules

MYPYPATH=src mypy --namespace-packages --explicit-package-bases .

But the behaviour in this case has been this way for a while.

I wasn't able to repro your example here #13202 (comment) . Specifically, I got the error on both 0.971 and 0.961 (and this is what I expected). Note that I wasn't able to run your repro exactly because pdm didn't work for me (seemed to install x86 stuff on my m1 mac), so I did the editable install myself with pip.

Agreed that in general this could be more usable + your suggestion of using sys.path to help out here is a good one. But that's a slightly different matter from addressing regressions between 0.961 and 0.971, which is typically more urgent. I'll continue to play around and see if I can find some behaviour that's regressed, but also curious for your thoughts. Also you've provided the mypy -v log of the failed run with 0.971, do you also have the logs from a successful run with 0.960?
(Also fwiw, if there is a regression, I'm suspicious of #11143 )

[edwardpeek-crown-public]

I've encountered a similar regression in handling of namespace packages going from 0.961 to 0.971, for which I can't find working mypy config/options: repro_v2.zip

$ unzip repro.zip
$ python3 -m venv venv
$ . ./venv/bin/activate
$ pip install -r requirements.txt
$ mypy
.../foo/src is in the MYPYPATH. Please remove it.
See https://mypy.readthedocs.io/en/stable/running_mypy.html#how-mypy-handles-imports for more info

The particular issue appears to be a bad interaction between:

• mypy_path configuration for correct lookup of namespace packages
• the packages being installed in editable mode

Edit: fixed issue in repro

[hauntsaninja]

Thanks, that's almost certainly a different issue: #13214 (comment)
I think I fixed the regression on master, could you confirm the fix? python3 -m pip install -U git+https://github.com/python/mypy.git to install latest code

[hauntsaninja]

Unfortunately, there's no known workaround on 0.971 :-( I asked for a release with the fix here, but we'll see what happens: #13089 (comment)

[edwardpeek-crown-public]

Can confirm master fixes the repro above and in our actual codebase. Thanks!