ImportError during pip build with Setuptools 34 and pip 6.0

[cjw296]

This is part of the traceback:

      File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/setuptools/__init__.py", line 10, in <module>
    
        from six.moves import filter, map
    
    ImportError: No module named six.moves

This started when 34.0.2 was released, here are some examples of the nightlies it's causing the fail:

https://travis-ci.org/python-excel/xlrd
https://travis-ci.org/Mortar/mortar_import

[mgedmin]

As a workaround, try pip install -U setuptools before you try to upgrade everything else.

[cjw296]

@mgedmin - where would I put that in a .travis.yml? What's the real fix here?

[jaraco]

I took a stab in python-excel/xlrd#194 and it seems to be a suitable workaround. The issue lies with pip and I've detailed the problem and proposed solution in pypa/pip#4253.

[cjw296]

@jaraco - if this is a pip issue, why did a setuptools release cause it? Would be great if you could do a setuptools release to revert until there's a real fix rather than requiring everyone-who-does-nightly-builds to go and change their CI scripts...

[jaraco]

Well, it was an unanticipated issue, and I didn't encounter it during my testing. Indeed, no one identified any issues in the two beta releases I made.

This change was released as a backward-incompatible release knowing that it might have some issues.

In fact, it's only affecting a small number of projects that happen to be updating setuptools as part of another pip upgrade operation. Since the impact is fairly small and the workaround simple enough, it's not worth the cost and effort of backing out these changes and rolling them out at another time.

[jacebrowning]

In fact, it's only affecting a small number of projects that happen to be updating setuptools as part of another pip upgrade operation.

I believe this is affecting ALL of my projects that use Python 2 on Travis CI.

[jaraco]

@jacebrowning: Can you share an example? Is the issue identical? In the aforementioned xlrd workaround, I've suggested perhaps developing a script to run across the affected projects and apply a workaround. Surely the presence of your additional use-cases is pertinent and an indicator of a more widespread issue.

[jacebrowning]

@jaraco This build passed a few days ago, but failed on rebuild: https://travis-ci.org/jacebrowning/verchew/jobs/190372989

docopt fails to install setuptools?

  Running setup.py install for docopt
  Found existing installation: py 1.4.26
    Uninstalling py-1.4.26:
      Successfully uninstalled py-1.4.26
  Found existing installation: setuptools 12.0.5
    Uninstalling setuptools-12.0.5:
      Successfully uninstalled setuptools-12.0.5
  Compiling /tmp/pip-build-qlDqhr/astroid/astroid/tests/testdata/python3/data/module2.py
  
  Running setup.py install for configparser
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/__init__.py", line 10, in <module>
        from six.moves import filter, map
    ImportError: No module named six.moves
    Complete output from command /home/travis/virtualenv/python2.7.9/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-qlDqhr/configparser/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-OTem_o-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/travis/virtualenv/python2.7.9/include/site/python2.7:
    Traceback (most recent call last):
    
      File "<string>", line 1, in <module>
    
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/__init__.py", line 10, in <module>
    
        from six.moves import filter, map
    
    ImportError: No module named six.moves

[jaraco]

By digging into that project, I see that it effectively invokes pip install -U pytest, which has a dependency on setuptools and attempts to upgrade it. That usage is likely to be fairly common.

[jaraco]

Annoyingly, I've been unable to replicate the issue. Even if I use Python 2.7 and download the requirements.txt from xlrd, they install fine. I can't get setuptools to build/install before its dependencies. I see on the failing test that it's using pip 6.0.7. Perhaps that's a factor also.

[jaraco]

I was able to replicate the issue by pinning to pip 6.0.7 and running this script:

#!/bin/sh
rm -Rf ~/.virtualenvs/issue-945
virtualenv --python python2.7 --no-setuptools --no-wheel ~/.virtualenvs/issue-945
~/.virtualenvs/issue-945/bin/pip install 'setuptools<34' pip==6.0.7

export LDFLAGS="-L$(brew --prefix libyaml)/lib"
export CFLAGS="-I$(brew --prefix libyaml)/include"

~/.virtualenvs/issue-945/bin/pip install -U nose nose_fixes nose-cov coveralls tox sphinx pkginfo six panci setuptools-git wheel twine

[jaraco]

Bisecting, it seems the issue goes away between pip 6.0.8 and pip 6.1.

[jaraco]

The issue was fixed in pypa/pip#2478 released April 2015.

Given that the issue is isolated only to old versions of pip and is triggered by aggressive upgrading of packages, I'm going to say that this has to be a won't fix, as I don't think there's a reasonable way to support an arbitrary mix of old and new packages... and since the final fix is already in place in pip, the best recommendation will be to upgrade pip.

[jaraco]

@cjw296, @jacebrowning I unfortunately don't see a solution for this issue in Setuptools. I could backout the changes for the v34 release, but then at what point could those be introduced? It appears the only option we have is to recommend that the projects (or CI environments perhaps) either update pip or don't upgrade setuptools. Can you think of any other options?

[jacebrowning]

either update pip or don't upgrade setuptools

Or upgrade both (pip install --upgrade pip setuptools) before using either.

[cjw296]

Interestingly, this only appears to be affecting some of my projects, notably ones that use requirements.txt but also a couple that don't. Very strange. Anyway, since it seems like there's no choice but to insert the hack, I'll do that and see what happens.

[cjw296]

Here's a build where the hack recommended doesn't work:

https://travis-ci.org/Simplistix/picky/jobs/196432515

Here's the CI file:
https://github.com/Simplistix/picky/blob/master/.travis.yml