import pkg_resources.py31compat causes no module error

[marikgoldstein]

While pip installing jupyter with a version of setuptools cloned just before this post, I came across a no module error coming from setuptools:

ImportError: No module named py31compat

which comes from setuptools/setuptools/sandbox.py, line 15

import pkg_resources.py31compat

However, in setuptools/pkg_resources/__init__.py, line 72, py31compat is imported there:

from . import py31compat

Therefore, it seems unnecessary to do

import pkg_resources.py31compat

in sandbox.py. It's a python file already included when importing the pkg_resources submodule. One can just do:

import pkg_resources

This fixed the problem for me.

Any reason for using an import statement directly to the python file within the submodule that already includes it in init.py?

I've also had this problem on older python environments while installing other libraries. Not sure what would change in an existing environment to cause this problem.

[marikgoldstein]

Actually, I see now that this also happens in easy_install.py.

from setuptools.py31compat import get_path, get_config_vars
from setuptools.py27compat import rmtree_safe

It would be easy enough to fix, but, is there a clear reason why this is causing errors? These individual files are not recognized as modules on their own, and are included by their containing submodules.

[jaraco]

Any reason for using an import statement directly to the python file within the submodule that already includes it in init.py?

Yes. pkg_resources isn't importing py31compat for the purpose of exporting it, but instead for the purpose of using it locally. It doesn't appear in pkg_resources.__all__. If one removed all usage of py31compat in pkg_resources, linters would indicate py31compat as an unused import and it would get removed. It's best for modules to import the requisite modules from their canonical locations.

Actually, I see now that this also happens in easy_install.py.

Really? You get "no module errors" in attempts to import setuptools.py31compat? That's even more surprising.

I'm going to have to ask you to do more investigation, as these modules work just fine in CI tests and for most users all day long.

My suspicion is that there's something unique to your environment that's corrupting or changing your installation of setuptools after your environment is started up. Maybe you're starting Jupyter, then installing packages or downgrading packages. Such a change might cause parts of setuptools to be imported from one install and others from another install.

If you can provide steps to replicate the issue, especially if the steps are minimally producing the error, someone here may be able to help investigate.

[gweis]

Seeing a similar issue, but when starting up Zope. (I know Zope does a lot of module path scanning and importing which may cause this.

However, when I replace the line in pkg_resources.__init__.py
from . import py31compat
with
py31compat = __import__('pkg_resources.py31compat')
it works without a problem. (but I have no idea what the actual difference is there)
even the next line works which should be the equivalent of the relative import.
py31compat = __import__('', globals(), locals(), ['py31compat'], 1)

Really no idea what's happening here and where the actual problem is. Just thought I'll throw this in to provide a potential pointer into the issue.

[mtdeguzis]

Came upon this thread when following http://webapp.org.ua/build-automation/using-buildout-system-on-windows/ (used latest where I could). Python 2.7 installed via https://www.python.org/ftp/python/2.7.14/python-2.7.14.amd64.msi

C:\Users\mtdeguzis> easy_install --version
Traceback (most recent call last):
  File "C:\Python27\Scripts\easy_install-script.py", line 11, in <module>
    load_entry_point('setuptools==36.7.1', 'console_scripts', 'easy_install')()
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 2291, in load
    return self.resolve()
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", line 47, in <module>

  File "build\bdist.win-amd64\egg\setuptools\sandbox.py", line 15, in <module>
ImportError: No module named py31compat

[jaraco]

@mtdeguzis I think there's evidence of a corrupted install in your traceback. Notice how pkg_resources is in C:\Python27\lib\site-packages\pkg_resources but setuptools is found at build\bdist.win-amd64\egg\setuptools (presumably installed as a .egg).

So you probably have two installs of setuptools, one as an egg and another 'externally managed' (maybe pip or virtualenv installed).

If you're using buildout or any installer than pip, your best bet is to work with that installer project to help ascertain where things went awry.

Most likely, you can work around the issue by removing the offending packages (i.e. setuptools and pkg_resources directories in site-packages), especially if you have a suitable setuptools*.egg there.

Based on this new evidence, I don't believe there's a flaw in Setuptools that needs to be corrected, but rather we're seeing evidence of improper installs that's only been exaggerated by the introduction of this (proper) import.

Feel free to continue to investigate. If you can find a valid scenario that I can replicate, especially without involving external installers like buildout, I can continue to investigate. If you can replicate the issue with buildout only, I suggest working with that project to isolate the issue.

[mtdeguzis]

That was the case, extra package, thanks

[shigemk2]

Me too.

• Ubuntu 17.10
• Python 3.6.2 (pyenv)
• setuptools 28.8.0 to 36.7.2

When I upgrade easy_install with easy_install -U setuptool, I got this error.

easy_install --version
Traceback (most recent call last):
  File "/path/to/.pyenv/versions/3.6.2/bin/easy_install", line 11, in <module>
    load_entry_point('setuptools==36.7.2', 'console_scripts', 'easy_install')()
  File "/path/to/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/path/to/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/path/to/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/path/to/.pyenv/versions/3.6.2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/path/to/.pyenv/versions/3.6.2/lib/python3.6/site-packages/setuptools-36.7.2-py3.6.egg/setuptools/command/easy_install.py", line 47, in <module>
  File "/path/to/.pyenv/versions/3.6.2/lib/python3.6/site-packages/setuptools-36.7.2-py3.6.egg/setuptools/sandbox.py", line 15, in <module>
ModuleNotFoundError: No module named 'pkg_resources.py31compat'

[jaraco]

You've bumped up against limitations in easy_install. It's not viable to use easy_install to upgrade packages installed by other installers such as pip or Debian apt (known to setuptools as externally-managed). What you're getting is corrupted environment where both versions of setuptools are installed, but Python import semantics are allowing the older pkg_resources to take precedence. Here's what's happening:

1. The easy_install upgrade has added setuptools 36.7.2 to your site-packages, but left setuptools 28.8.0 installed as externally-managed packages.
2. easy_install command imports pkg_resources, which uses Python's import semantics and since setuptools 25, eggs no longer get precedence. So the pkg_resources from setuptools 28.8.0 gets imported.
3. pkg_resources loads setuptools 36.7.2 as indicated in the easy_install script and invokes the command. Presumably, this gives precedence now to the 36.7.2 package before setuptools is imported, so setuptools module comes from 36.7.2.
4. When setuptools from 36.7.2 attempts to import something from pkg_resourecs, because it was imported in in step (2) from 28.8.0, it doesn't have the module that setuptools expects and requires.

The recommendation here is to not use easy_install to install anything. Instead, use pip install -U setuptools.

[gweis]

Hi, ... I know this issue is closed, but here is my case...

I have pip 9.0.1 installed, which has a 'vendored in' version of pkg_resources without py31compat package....

Whichever pkg_resources get's loaded first is the winner ;(

[jaraco]

@gweis: Even though pip has a vendored pkg_resources, I wouldn't expect that version of pkg_resources to be loaded as sys.modules['pkg_resources'].

>>> import pip
>>> pip._vendor.pkg_resources
<module 'pip._vendor.pkg_resources' from '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py'>
>>> import sys
>>> 'pkg_resources' in sys.modules
False

So in that case, if one were to import pkg_resources, I would expect it to get that package from a top-level package and not from pip.

[gweis]

Finally tracked it down.

I suspected that pip may be the issue because, that was the only other pkg_resources module I could find in my file system. (also there is no other setupttools version installed anywhere)

However, in my case I have a namespace module org with an __init__.py file containing:
__import__('pkg_resources').declare_namespace(__name__)

When my app starts up, something is importing stdlib copy module which tries to import from org.python.core import PyStringMap which will load my __init__.py from my namespace package, which again tries to use copy (within pkg_resources/_vendor/pyparsing) and then somewhere further up in the call stack, something was catching the AttributeError on import and seemed to have left some of the module imports in a funny state. etc....

Essentially, using __import__('pkg_resources').declare_namespace(__name__) created an recursive import chain in my case.

Lesson to take form this. Don't use org as namespace package or use pkgutil.extend_path instead of pkg_resources.declare_namespace if unavoidable. Not sure whether pkgutil.extend_path would have any side effects or not. The docs are not very specific about the difference of the approaches.

(Still not clear why using __import__ works but from . import doesn't though.)