Dependencies among editable packages

[kunalparmar]

I have downloaded the latest release of pyobjc and am trying to install it using a requirements file which fails.

$> cat requirements.txt 
-e pyobjc/pyobjc-core
-e pyobjc/pyobjc-framework-Cocoa

$> pip install -r requirements.txt 
Obtaining file:///Users/kparmar/overflow/pyobjc/pyobjc-core (from -r requirements.txt (line 1))
  Running setup.py egg_info for package from file:///Users/kparmar/overflow/pyobjc/pyobjc-core

    warning: no directories found matching 'Scripts'
    warning: no directories found matching 'setup-lib'
    warning: no directories found matching 'source-deps'
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
Obtaining file:///Users/kparmar/overflow/pyobjc/pyobjc-framework-Cocoa (from -r requirements.txt (line 2))
  Running setup.py egg_info for package from file:///Users/kparmar/overflow/pyobjc/pyobjc-framework-Cocoa
    No local packages or download links found for pyobjc-core>=2.4a0
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/Users/kparmar/overflow/pyobjc/pyobjc-framework-Cocoa/setup.py", line 68, in <module>
        extra_link_args=["-framework", "Foundation"]),
      File "pyobjc_setup.py", line 360, in setup
        **k
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 260, in __init__
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 284, in fetch_build_eggs
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 563, in resolve
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 799, in best_match
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 811, in obtain
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 327, in fetch_build_egg
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 434, in easy_install
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/package_index.py", line 475, in fetch_distribution
    AttributeError: 'NoneType' object has no attribute 'clone'
    Complete output from command python setup.py egg_info:
    No local packages or download links found for pyobjc-core>=2.4a0

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/Users/kparmar/overflow/pyobjc/pyobjc-framework-Cocoa/setup.py", line 68, in <module>

    extra_link_args=["-framework", "Foundation"]),

  File "pyobjc_setup.py", line 360, in setup

    **k

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 260, in __init__

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 284, in fetch_build_eggs

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 563, in resolve

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 799, in best_match

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 811, in obtain

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 327, in fetch_build_egg

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 434, in easy_install

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/package_index.py", line 475, in fetch_distribution

AttributeError: 'NoneType' object has no attribute 'clone'

[mcdonc]

Ref: http://bugs.python.org/setuptools/issue90

[carljm]

Sounds like the bug is fixed in distribute, so "virtualenv --distribute" should be a functional workaround.

Closing as it doesn't appear to be a bug in pip.

[kunalparmar]

@carljm That did not solve the issue. I've pasted the output below. Any idea where the bug is, if its not in pip.

$> virtualenv --distribute ~/.virtualenvs/pyobjc
New python executable in /Users/kparmar/.virtualenvs/pyobjc/bin/python
Installing distribute.................................................................................................................................................................................................done.
Installing pip................done.

$> source ~/.virtualenvs/pyobjc/bin/activate
(pyobjc)
$> cat requirements.txt 
-e pyobjc/pyobjc-core
-e pyobjc/pyobjc-framework-Cocoa
(pyobjc)
$> pip install -r requirements.txt 
Obtaining file:///Users/kparmar/overflow/pyobjc/pyobjc/pyobjc-core (from -r requirements.txt (line 1))
  Running setup.py egg_info for package from file:///Users/kparmar/overflow/pyobjc/pyobjc/pyobjc-core

    warning: no directories found matching 'Scripts'
    warning: no directories found matching 'setup-lib'
    warning: no directories found matching 'source-deps'
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
Obtaining file:///Users/kparmar/overflow/pyobjc/pyobjc/pyobjc-framework-Cocoa (from -r requirements.txt (line 2))
  Running setup.py egg_info for package from file:///Users/kparmar/overflow/pyobjc/pyobjc/pyobjc-framework-Cocoa
    No local packages or download links found for pyobjc-core>=2.4a0
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/Users/kparmar/overflow/pyobjc/pyobjc/pyobjc-framework-Cocoa/setup.py", line 68, in <module>
        extra_link_args=["-framework", "Foundation"]),
      File "pyobjc_setup.py", line 360, in setup
        **k
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/dist.py", line 221, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs
        parse_requirements(requires), installer=self.fetch_build_egg
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/pkg_resources.py", line 580, in resolve
        dist = best[req.key] = env.best_match(req, self, installer)
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/pkg_resources.py", line 825, in best_match
        return self.obtain(req, installer) # try and download/install
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/pkg_resources.py", line 837, in obtain
        return installer(requirement)
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/dist.py", line 294, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/command/easy_install.py", line 584, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pyobjc-core>=2.4a0')
    Complete output from command python setup.py egg_info:
    No local packages or download links found for pyobjc-core>=2.4a0

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/Users/kparmar/overflow/pyobjc/pyobjc/pyobjc-framework-Cocoa/setup.py", line 68, in <module>

    extra_link_args=["-framework", "Foundation"]),

  File "pyobjc_setup.py", line 360, in setup

    **k

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/dist.py", line 221, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs

    parse_requirements(requires), installer=self.fetch_build_egg

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/pkg_resources.py", line 580, in resolve

    dist = best[req.key] = env.best_match(req, self, installer)

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/pkg_resources.py", line 825, in best_match

    return self.obtain(req, installer) # try and download/install

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/pkg_resources.py", line 837, in obtain

    return installer(requirement)

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/dist.py", line 294, in fetch_build_egg

    return cmd.easy_install(req)

  File "/Users/kparmar/.virtualenvs/pyobjc/lib/python2.7/site-packages/distribute-0.6.27-py2.7.egg/setuptools/command/easy_install.py", line 584, in easy_install

    raise DistutilsError(msg)

distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pyobjc-core>=2.4a0')

[carljm]

@kunalparmar It appears that did fix the original problem; note that you're getting an entirely different error now. Does the pyobjc-core source you have in that directory specify a version that is >=2.4a0?

[kunalparmar]

@carljm Yes, its set to 2.4a0.

[carljm]

@kunalparmar Sorry, I'm not able to reproduce, so I'm not sure what else to recommend. I tried setting up a similar situation locally with Sphinx and its dependencies (docutils, Jinja2, pygments) where I unpacked each one as a directory and then set up a requirements file using -e and the local directory for each one, and pip install -r requirements.txt worked fine for me in that case. I'm not on OS X so I can't try with pyobjc. If you can put together a simpler reliably-failing case, I can take a look.

Also, what version of pip are you on? Try with the latest release (from just a few days ago)?

[kunalparmar]

@carljm I was on version 1.1. No luck with version 1.2.1 either. Its possible that the issue is with the setup.py scripts of pyobjc. If you dont mind, maybe you can take a quick look at the scripts here (https://bitbucket.org/ronaldoussoren/pyobjc/src) to catch any obvious errors in the setup.py scripts.

If you'd like me to try something to help get to the root of this problem, please let me know. Really appreciate your help!

[mcdonc]

This is actually a problem with using setup_requires. The code which attempts to process setup_requires (in either setuptools or distribute) apparently doesn't take into the account any index or other find-links specified by the user doing the installing. A minimal test package which exhibits the bug is available from http://plope.com/static/misc/apackage.tgz (it will fail similarly to the OP's failures when "pip install -r requirements.txt" is executed).

This is an upstream setuptools/distribute bug that @carljm seems to believe we can't work around in pip.

[carljm]

Use of setup_requires in a setup.py file is just another case of the common problem: "pip must be able to get metadata about your package; the only way to get metadata about a package in the current Python packaging ecosystem is to execute setup.py, so pip must be able to execute setup.py reliably in all situations." setup_requires names things that must be installed in order to run setup.py, so it makes it less likely that setup.py can be reliably executed in all situations.

There's a solution proposed near the end of the thread in #25, which is to add an optional flag to pip install that would cause it to install the listed requirements sequentially rather than first collecting all packages and metadata, and then installing them. In most cases, this is worse, because it allows a broken package somewhere in the chain to cause things to be left in a partially-installed state, but in cases of setup_requires dependencies like this it would be useful.

[kunalparmar]

Looking through the setup.py of pyobjc-framework-Cocoa I came across setup_requires and install_requires both of which had pyobjc-core>=2.4a0. I removed that from setup_requires and then ran pip and everything installed. Is that the right fix or does that work for other reasons?

[mcdonc]

Unknown. If it works, then it's the right fix. But it's completely dependent on the particulars of the package.