Add support for pkgutil-style namespace packages

[alxroyer]

🚀 Feature

Add support for pkgutil-style namespace packages.

Pitch

In october 2018, PR #5691 brought support for PEP420 namespace packages (i.e. native namespace packages), which closed issue #1645.

However, the native PEP420 method only works for Python 3.

This type of namespace package is defined in PEP 420 and is available in Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3 and installation via pip.

According to Python documentation, for Python 2/3 compatible code, the pkgutil-style remains the recommended way.

This is recommended for new packages that need to support Python 2 and 3 and installation via both pip and python setup.py install.

Let the following project structure, with PYTHONPATH and MYPYPATH pointing to the src/ and 'tests/src/' directories:

my-project
+-- src/
|   +-- mynamespace/
|       +-- __init__.py
+-- tests/
    +-- data/
    +-- src/
        +-- mynamespace/
            +-- test/
            |   +-- __init__.py
            +-- __init__.py

The mynamespace namespace is splitted in two directories, in order not to mix up the test code with the deliverable code in this project.
Within both src/mynamespace/__init__.py and tests/src/mynamespace/__init__.py, the following code implements the pkgutil-style namespace package:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Python 2 and Python 3 execute this code correctly.

But, unfortunately, mypy (as of 0.770) generates a false 'Duplicate module named 'mynamespace'' error (same error as the former #1645 issue).

[alxroyer]

Checked with mypy 0.782, still no support for pkgutil-style namespace packages, even though the namespace_packages option is activated:

test\src\mynamespace\__init__.py: error: Duplicate module named 'mynamespace' (also at 'src\mynamespace\__init__.py')
test\src\mynamespace\__init__.py: error: Are you missing an __init__.py?

[alxroyer]

Could it be limited work based on PR #5691?

[alxroyer]

I've just found a way to adjust the source files configuration that makes mypy work with pkgutil-style namespace packages.

In my mypy.conf file:

# Previously:
# files = src/**/*.py, tests/src/**/*.py
# Now:
files = src/**/*.py, tests/src/mynamespace/test/**/*.py

The trick, I guess, is that only one mynamespace/__init__.py file should be set in the source files list.
With the latter configuration:

• src/mynamespace/__init__.py remains in the source files list,
• but tests/src/mynamespace/__init__.py is now skipped over.

Not sure however whether this is the expected configuration, or just a specific workaround for pkgutil-style namespace packages.
The --namespace-packages documentation does not seem to tell anything special about that.

Another observation:
Once this configuration is done, whatever the --python-version, the --namespace-packages option seems to have no effect.

[hterik]

This limitation is still present in mypy 0.910 on Python 3.9

The trick with skipping the init file at root level and going for mypy namespace/**/*.py saved the day. Thanks. 👍

Another trick is to simply skip namespace packages altogether and just use underscores on normal packages instead, like mynamespace_test, mynamespace_foo, etc. Some times feels like it would make things a lot easier. Not just mypy but almost every other tool, linter or editor interacting with packages requires some kind of quirk for namespaces to work (using pkgutil-style instead of implicit PEP420 was one of them, can't remember why).

[jake-sigtech]

+1 - Would be nice to have a proper fix for this.

[xmo-odoo]

According to Python documentation, for Python 2/3 compatible code, the pkgutil-style remains the recommended way.

FWIW pkgutil-style namespacing is also necessary if trying to "merge" a namespace package with an existing package which contains code at this location, or when trying to make a sub-package of that a namespace package.

[alxroyer]

For the info, I've recently updated my mypy configuration file with mypy v1.0.1
(executed with Python 3.7.9, but not sure that makes a difference).

exclude does not help

The new exclude configuration does not help when sticking with files.

According to the --exclude documentation:

"Note that this flag only affects recursive directory tree discovery, that is, when mypy is discovering files within a directory tree or submodules of a package to check. If you pass a file or module explicitly it will still be checked."

Indeed, I could not exclude 'tests/src/mynamespace/__init__.py' this way in the initial example.

modules / packages could be the solution?

See modules and packages documentations.

The following configurations seem to work:

[mypy]
modules = mynamespace, mynamespace.test

or:

[mypy]
packages = mynamespace, mynamespace.test

Confusingly, with modules, mypy tells "Success: no issues found in 2 source files", even though we're talking about packages...
With packages, the number of files seems to be more consistent, but I've not checked whether all module dependencies were covered by this number.

As far as I've tested, even with modules, errors are detected in module dependencies, i.e. the full packages, which can be much more than "2 sources files".

---

Memo: modules and packages incompatible with files

Using either modules or packages with files leads to the following error:

usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: May only specify one of: module/package, files, or command.

[hauntsaninja]

Python 2 is very dead, mypy is not going to gain support for pkgutil style namespaces.