How should I correctly lint an implicit namespace package?

[dalai4git]

I am trying to lint a namespace package (PEP420), but I can't get it to work without getting import-error. I am not sure my approach is correct, so I'd appreciate some pointers. In the example below I just want to lint org.company.package2.

Steps to reproduce

# mkdir -p org/company/package1
# mkdir -p org/company/package2
# touch org/company/package1/__init__.py
# echo "class Module:\n    pass\n" >> org/company/package1/module.py
# touch org/company/package2/__init__.py
# echo "from org.company.package1.module import Module\n" >> org/company/package2/module.py
# python
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from org.company.package1.module import Module
>>> 

Current behavior

Specifying the package does not work at all

# pylint org.company.package2
No config file found, using default configuration
************* Module org.company.package2
F:  1, 0: No module named org.company.package2 (fatal)

Specifying the path gives me an import-error

# pylint -r -n org/company/package2 
No config file found, using default configuration
************* Module package2.module
C:  2, 0: Trailing newlines (trailing-newlines)
C:  1, 0: Missing module docstring (missing-docstring)
E:  1, 0: Unable to import 'org.company.package1.module' (import-error)
W:  1, 0: Unused Module imported from org.company.package1.module (unused-import)

pylint --version output

pylint 1.6.5, 
astroid 1.4.9
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609]

With master version of pylint and astroid

This seems to work correctly but is not what I want

# ~/workspace/venv/bin/pylint org       
No config file found, using default configuration
************* Module company.package1.module
C:  3, 0: Trailing newlines (trailing-newlines)
C:  1, 0: Missing module docstring (missing-docstring)
C:  1, 0: Missing class docstring (missing-docstring)
R:  1, 0: Too few public methods (0/2) (too-few-public-methods)
************* Module company.package2.module
C:  2, 0: Trailing newlines (trailing-newlines)
C:  1, 0: Missing module docstring (missing-docstring)
W:  1, 0: Unused Module imported from org.company.package1.module (unused-import)

This doesn't seem to be doing anything

# ~/workspace/venv/bin/pylint org.company       
No config file found, using default configuration

and this apparently crashes

# ~/workspace/venv/bin/pylint org.company.package       
No config file found, using default configuration
Traceback (most recent call last):
  File "/home/dalai/workspace/venv/bin/pylint", line 9, in <module>
    load_entry_point('pylint==1.7.0', 'console_scripts', 'pylint')()
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/__init__.py", line 13, in run_pylint
    Run(sys.argv[1:])
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/lint.py", line 1300, in __init__
    linter.check(args)
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/lint.py", line 726, in check
    self._do_check(files_or_modules)
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/lint.py", line 837, in _do_check
    for descr in self.expand_files(files_or_modules):
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/lint.py", line 869, in expand_files
    self.config.black_list_re)
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/utils.py", line 894, in expand_modules
    modpath = _modpath_from_file(subfilepath, is_namespace)
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg/pylint/utils.py", line 833, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(filename, is_package_cb=_is_package_cb)
  File "/home/dalai/workspace/venv/lib/python3.5/site-packages/astroid-1.5.0-py3.5.egg/astroid/modutils.py", line 310, in modpath_from_file_with_callback
    filename, ', \n'.join(sys.path)))
ImportError: Unable to find module for /home/dalai/workspace/org/company/package2/module.py in /home/dalai/workspace, 
., 
/home/dalai/workspace/venv/bin, 
/home/dalai/workspace/venv/lib/python3.5/site-packages/pylint-1.7.0-py3.5.egg, 
/home/dalai/workspace/venv/lib/python3.5/site-packages/editdistance-0.3.1-py3.5-linux-x86_64.egg, 
/home/dalai/workspace/venv/lib/python3.5/site-packages/astroid-1.5.0-py3.5.egg, 
/usr/lib/python35.zip, 
/usr/lib/python3.5, 
/usr/lib/python3.5/plat-x86_64-linux-gnu, 
/usr/lib/python3.5/lib-dynload, 
/home/dalai/workspace/venv/lib/python3.5/site-packages, 
/home/dalai/workspace/venv/lib/python3.5/site-packages/astroid-1.5.0-py3.5.egg/astroid/brain

[PCManticore]

Interesting, thank you for reporting this issue! I thought it was solved in the master branch. Why it is not what you want, can you clarify a bit?

[dalai4git]

I want to only lint the org.company.package2 package and not both. Pylint is being run in jenkins and ideally the report should be clean of warnings/errors from other (possibly external) packages. Currently I just use pylint 1.6.5, add __init__.py files automatically before running and ignore the import-errors. I was just wondering if I could simplify this with a newer version of pylint.

So if I understand correctly, implicit namespace packages are only supported by the version in the master branch and not the currently released one?

[brycepg]

I'm getting an error for implicit namespace packages on master. The workaround I currently have is making implicit namespace packages explicit.

I'll try to work on this issue

[tgpfeiffer]

With

pylint 2.5.3
astroid 2.4.2
Python 3.8.3 (default, May 17 2020, 14:48:56) [GCC]

I still see the problem. The minimal example from @dalai4git shows

$ pylint org 
************* Module org.company.package1.module
org/company/package1/module.py:3:0: C0305: Trailing newlines (trailing-newlines)
org/company/package1/module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
org/company/package1/module.py:1:0: C0115: Missing class docstring (missing-class-docstring)
org/company/package1/module.py:1:0: R0903: Too few public methods (0/2) (too-few-public-methods)
************* Module org.company.package2.module
org/company/package2/module.py:2:0: C0305: Trailing newlines (trailing-newlines)
org/company/package2/module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
org/company/package2/module.py:1:0: W0611: Unused Module imported from org.company.package1.module (unused-import)

$ pylint org/company 
************* Module company
org/company/__init__.py:1:0: F0010: error while code parsing: Unable to load file org/company/__init__.py:
[Errno 2] No such file or directory: 'org/company/__init__.py' (parse-error)

$ pylint org/company/package2
************* Module package2.module
org/company/package2/module.py:2:0: C0305: Trailing newlines (trailing-newlines)
org/company/package2/module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
org/company/package2/module.py:1:0: E0401: Unable to import 'org.company.package1.module' (import-error)
org/company/package2/module.py:1:0: W0611: Unused Module imported from org.company.package1.module (unused-import)

Is this expected to be working in the latest release?

[bdegreve]

I also have this problem. It seems to originate from this bit in pylint/utils/utils.py:

def _modpath_from_file(filename, is_namespace, path=None):
    def _is_package_cb(path, parts):
        return modutils.check_modpath_has_init(path, parts) or is_namespace

    return modutils.modpath_from_file_with_callback(
        filename, path=path, is_package_cb=_is_package_cb
    )

_modpath_from_file fails to find a modpath for org/company/package2/module.py because org.company.package2 is not a namespace (so is_namespace is false), but neither do all package levels in org.company.package2 have a __init__.py so modutils.check_modpath_has_init returns False too.

I'm not sure what the right behavior should be. I guess if the different package levels are org, company and package, then going from the left to the right, you can have as many namespace packages without __init__.py, but as soon as one level has one, then all the ones on the right have to have one too? Is that right? I'm not familiar with the reason d'être of _is_package_cb so if this would be the right approach ...

pylint 2.7.4
astroid 2.5.2
python 3.7.9

[hippo91]

@bdegreve thanks for your report and your analysis. We will work on this ASAP.

[bdegreve]

@hippo91 Great! Let me know if I can help ...

[sak96]

the issue is that when running

pylint org
... import-error

but not when

pylint .

so when using . the namespace org.company is found.
one work around i found is to use init hook

pylint --init-hook='import sys; sys.path.append(".")'  org/

[tjulinfan]

@sak96 Thanks! Your workaround works for me.

This is suffer, do we have any plan to fix it? or can we have an option to determine if to support the implicit namespace package.

[jacobtylerwalls]

The original example presented by @dalai4git was fixed very recently in astroid, see pylint-dev/astroid#1576, but not released in any version of pylint yet, which is why you're not seeing it. It's targeted for release in astroid 2.12 / pylint 2.15.

Output of first example in OP no longer has import-error as of pylint-dev/astroid@18483dc:

% python -m pylint org.company.package2
************* Module org.company.package2.module
org/company/package2/module.py:2:0: C0305: Trailing newlines (trailing-newlines)
org/company/package2/module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
org/company/package2/module.py:1:0: W0611: Unused Module imported from org.company.package1.module (unused-import)

------------------------------------------------------------------
Your code has been rated at 0.00/10 (previous run: 0.00/10, +0.00)

---

I would be very grateful if those who participated in this discussion would verify if the latest main branch of astroid solves any issues or makes any issues worse. If so, please open new issues to discuss them. 🙏🏻

To test, make sure you install astroid bleeding-edge after installing pylint: this is because pip install pylint will give you astroid==2.11.4.

---

Leaving this issue open for the moment to see if we need documentation improvements or test cases in pylint.

[jacobtylerwalls]

I don't see a significant difference with #1667, so I'll close as a duplicate.