Fallback to 'python setup.py bdist' when 'python setup.py bdist_dumb' fails?

[jzoldak]

One of the entries in my requirements file is paver, but pip-accel is choking on installing it.
You can reproduce with:

pip-accel install paver -vv --no-clean --build /tmp

This will give you something like:

pip_accel.bdist[79437] DEBUG Executing external command: /Users/foo/.virtualenvs/test/bin/python setup.py bdist_dumb --format=tar
Traceback (most recent call last): paver (1.2.3) .. 
  File "/Users/foo/.virtualenvs/test/bin/pip-accel", line 9, in <module>
    load_entry_point('pip-accel==0.13.5', 'console_scripts', 'pip-accel')()
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/__init__.py", line 117, in main
    install_requirements(requirements)
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/__init__.py", line 280, in install_requirements
    install_binary_dist(members, prefix=install_prefix, python=python)
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/bdist.py", line 248, in install_binary_dist
    for member, from_handle in members:
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/bdist.py", line 84, in get_binary_dist
    raw_file = build_binary_dist(package, version, directory, python=python)
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/bdist.py", line 156, in build_binary_dist
    filenames = os.listdir(dist_directory)
OSError: [Errno 2] No such file or directory: '/tmp/paver/dist'

And now check out paver's build directory with:

ls -l /tmp/paver

[xolox]

Hi Jesse and thanks for the feedback. I think this is actually a bug in Paver's setup.py script (which is by the way the most non standard setup.py script I've seen so far, and I've seen hundreds of them, even fixed some along the way):

peter@macbook> python setup.py bdist_dumb --format=tar

*** Problem with pavement:
/home/peter/tmp/2014-11-07/paver/pavement.py
In setting command options for Task: distutils.command.bdist_dumb,
    option -d/--dist-dir for Task: distutils.command.bdist_dumb is already in use
    by another task in the dependency chain.

However I did notice that the python setup.py bdist command works fine and produces the desired result. There's something to argue for making pip-accel as robust as possible by falling back to that option. Nevertheless the command used by pip-accel is documented in the Python standard library and should Just Work (TM), that must be a bug.

[xolox]

This is definitely coming from the convoluted way that Paver insists on bootstrapping its own installation using itself. To be honest I really don't get what the point of that exercise is (except for showcasing creativity :-). I guess I can still compensate for this, but it does leave me with a bit of a bad taste in my mouth :-). On the other hand when pip-accel gets you 99% of the way there, it should try to support that last 1% as well, otherwise you can't use it or have to work around this yourself, also not nice. I'll get back to this after merging the S3 support that you are also waiting for ;-).

[jzoldak]

ah, that makes sense. Thanks for tracking this down @xolox

[xolox]

For anyone reading along, in case my previous comments sounded a bit harsh: I like installation procedures to be as declarative (introspectable and safe) as possible. The setup.py script of Paver is the absolute opposite, it seems to be running hundreds to thousands of lines of code just in order to call distutils.setup() or setuptools.setup()... When I find these kinds of things I usually try to fix them but that assumes the project author(s) agree with the general sentiment that setup.py scripts should be as declarative as possible. That's clearly not the case here so I assume there's no point in trying to convince anyone :-).

[xolox]

Hi Jesse,

I believe this is now fixed, can you try the latest version to confirm?

Thanks for the feedback!

[jzoldak]

Peter I'm now getting this. Is it what you expected?
My command is pip-accel install -r requirements.txt where my requirements.txt file includes Paver==1.2.1

2014-11-24 15:57:07 precise64 pip_accel.bdist[1799] INFO Building Paver (1.2.1) binary distribution ..
2014-11-24 15:57:07 precise64 pip_accel.cli[1799] ERROR Caught unhandled exception!
Traceback (most recent call last):
  File "/home/foo/src/pip-accel/pip_accel/cli.py", line 53, in main
    accelerator.install_from_arguments(arguments)
  File "/home/foo/src/pip-accel/pip_accel/__init__.py", line 180, in install_from_arguments
    self.install_requirements(requirements, **kw)
  File "/home/foo/src/pip-accel/pip_accel/__init__.py", line 365, in install_requirements
    self.bdists.install_binary_dist(binary_distribution, **kw)
  File "/home/foo/src/pip-accel/pip_accel/bdist.py", line 284, in install_binary_dist
    for member, from_handle in members:
  File "/home/foo/src/pip-accel/pip_accel/bdist.py", line 71, in get_binary_dist
    raw_file = self.build_binary_dist(requirement)
  File "/home/foo/src/pip-accel/pip_accel/bdist.py", line 156, in build_binary_dist
    name=requirement.name, version=requirement.version)
NoBuildOutput: Build of Paver (1.2.1) did not produce a binary distribution archive!

Please check the build output because it will probably provide a hint about what went wrong.

Build output:

*** Problem with pavement:
/tmp/tmpyd0u3d/Paver/pavement.py
In setting command options for Task: distutils.command.bdist_dumb,
    option -d/--dist-dir for Task: distutils.command.bdist_dumb is already in use
    by another task in the dependency chain.

[xolox]

Hi Jesse,

No that output is not what I expected. It looks like you're still running pip-accel 0.18 while the bug fix relevant to this issue was included in pip-accel 0.19 (or I'm overlooking some dumb mistake on my side, but I checked and pip-accel 0.19 is available on PyPI ;-). Are you sure you upgraded to the latest version? For reference, I just tested this again and it works fine for me based on pip-accel 0.19. The output quoted below shows the new behavior in the line Build of paver (1.2.1) failed, falling back to alternative method ..

peter@macbook> pip-accel install paver==1.2.1
2014-11-27 21:56:04 pip_accel[3320] INFO Unpacking source distribution(s) ..
2014-11-27 21:56:04 pip_accel[3320] INFO Executing command: pip install --no-install --no-index --find-links=file:///home/peter/.pip-accel/sources --build-directory=/tmp/tmpMnHDMF paver==1.2.1
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking paver==1.2.1
  Could not find a version that satisfies the requirement paver==1.2.1 (from versions: 1.2.3)
No distributions matching the version for paver==1.2.1
Storing complete log in /home/peter/.pip/pip.log
2014-11-27 21:56:04 pip_accel[3320] INFO We don't have all source distribution archives yet!
2014-11-27 21:56:04 pip_accel[3320] INFO Downloading missing source distribution(s) ..
2014-11-27 21:56:04 pip_accel[3320] INFO Executing command: pip install --no-install --download-cache=/home/peter/.pip/download-cache --find-links=file:///home/peter/.pip-accel/sources --build-directory=/tmp/tmpMnHDMF paver==1.2.1
Downloading/unpacking paver==1.2.1
  Downloading Paver-1.2.1.tar.gz (305kB): 305kB downloaded
  Storing download in cache at /home/peter/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FP%2FPaver%2FPaver-1.2.1.tar.gz
  Running setup.py egg_info for package paver
Successfully downloaded paver
2014-11-27 21:56:05 pip_accel[3320] INFO Finished downloading source distribution(s) in 1.2 second.
2014-11-27 21:56:05 pip_accel[3320] INFO Installing from binary distributions ..
2014-11-27 21:56:06 pip_accel[3320] INFO Uninstalled previously installed package paver.
2014-11-27 21:56:06 pip_accel.caches.s3[3320] INFO Checking if distribution archive is available in S3 bucket: v7/paver:1.2.1:CPython-2.7.tar.gz
2014-11-27 21:56:06 pip_accel.bdist[3320] INFO Building paver (1.2.1) binary distribution ..
2014-11-27 21:56:06 pip_accel.bdist[3320] WARNING Build of paver (1.2.1) failed, falling back to alternative method ..
2014-11-27 21:56:06 pip_accel.bdist[3320] INFO Building paver (1.2.1) binary distribution ..
2014-11-27 21:56:06 pip_accel.bdist[3320] INFO Finished building paver (1.2.1) in 0.41 seconds.
2014-11-27 21:56:06 pip_accel[3320] INFO Finished installing 1 requirement in 1.05 second.

[jzoldak]

argh, no you're probably right - I probably had the wrong installed version. I'll have to try it again and let you know.

[xolox]

@jzoldak: The pip-accel test suite now includes a test to verify that paver==1.2.3 does not cause any issues so I'm pretty sure this issue has been resolved. Can you confirm?

[jzoldak]

Looks great! Thanks for implementing this.
👍

[xolox]

@jzoldak: Thanks for the confirmation!