BUG?: extra in extra not working

[schlichtanders]

Dear team,

I just stumbled upon a bug, and also tried to figure out what is going on, however it is too complicated for me unfortunately... I hope you can help:

In principle I have a setup.py for parentpackage with essentially the following code:

setup(
...
install_requires = ["package"]
extras_require = {
  "extra" : ["package[extra]", "otherpackage"]
}
...
)

This however does not work as expected. When installing the parentpackage[extra], otherpackage gets installed, however only package and NOT package[extra] (i.e. nothing of the extra requirements listed under package[extra] is installed).

I felt like the reason lies here https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L261
in that if the second requirement is added ("package[extra]"), than both install_req.constraint == False and existing_req.constraint == False, which prevent the crucial code part from being executed.

I just changed this that it gets executed for testing (i.e. made itif not install_req.constraint and not existing_req.constraint). Surprisingly for me, the nested extra packages are STILL NOT installed.

Any help would be great

[piotr-dobrogost]

You did not provide version of pip.
This is duplicate of #3147 and was already fixed in pull #3198.

[schlichtanders]

Thanks for the fast response,
All the tests I did, I did on pip@develop. So this is not completely fixed apparently.

After searching through the duplicates I found that #3105 is already merged... but #2716 not yet. Maybe this may help?

If this issue is indeed meant to be completely fixed, here my setup.py which dependencies are all open accessible (however the source code of the mypackage itself is closed source):

#!/usr/bin/python
# -*- coding: utf-8 -*-
__author__ = 'Stephan Sahm'

import os
import shutil
from setuptools import setup
from distutils.command.clean import clean as Clean

class CleanCmd(Clean):

    description = "Cleans ..."

    def run(self):
        Clean.run(self)
        if os.path.exists('build'):
            shutil.rmtree('build')

        for dirpath, dirnames, filenames in os.walk('.'):
            for filename in filenames:
                if (filename.endswith('.so') or \
                    filename.endswith('.pyd') or \
                    filename.endswith('.pyc') or \
                    filename.endswith('_wrap.c') or \
                    filename.startswith('wrapper_') or \
                    filename.endswith('~')):
                        os.unlink(os.path.join(dirpath, filename))

            for dirname in dirnames:
                if dirname == '__pycache__' or dirname == 'build':
                    shutil.rmtree(os.path.join(dirpath, dirname))
                if dirname == "pymscons.egg-info":
                    shutil.rmtree(os.path.join(dirpath, dirname))

setup(
    name='mypackage',
    version='0.1',
    description='description of mypackage',
    author=__author__,
    author_email='[email protected]',
    license='closed source',
    packages=['mypackage', 'mypackage.subpackage'],
    zip_safe=False,
    install_requires=["pyparsing>=2.0.3",
                      "pyparsingOD>=2.0.3",
                      "pyparsing_regex>=0.1.0"],
    extras_require = {
        'PyPy':  [],
        # BUG CPython call to regex does not work at all.
        # As a workaround I added the extra dependencies explicitly:
        'CPython': ["pyparsing_regex[CPython]>=0.1.0", "yappi>=0.94"],
    },
    dependency_links = [
        "git+https://github.com/schlichtanders/pyparsing_regex.git#egg=pyparsing_regex-0.1.0",
        "git+https://github.com/schlichtanders/pyparsing-2.0.3-OrderedDict#egg=pyparsingOD-2.0.3.1"
    ],
    # include_package_data=True,  # should work, but doesn't, I think pip does not recognize git automatically
    package_data = {
        'pymscons': ['schemes/*.pkl', 'schemes/*.txt'],
    },
    cmdclass={'clean': CleanCmd}
)

Maybe someone can reproduce the weird behaviour I experience

EDIT: I know recognize a maybe crucial difference: I install all these with pip --process-dependency-links as direct requirements are not yet supported by setuptolls (at least to my knowledge and tests)

[piotr-dobrogost]

#2716 is not needed for this, #3105 was closed and not merged and the issue was supposed to be fixed in #3198 as I wrote. Looks like it's not fixed.

[dstufft]

Closing this, it is either already fixed or if it isn't it is fundamentally a duplicate of #988.

[thomasjthomasj]

This is not a duplicate issue, I am currently experiencing this on version 9.0.1.

The difference is that #3147 is talking about extras appearing in extra_requires in a general sense, e.g.:

setup(
    install_requires=['some-package'],
    extras_require=[
        'a-different-package[an-extra]'
    ]
)

The issue I am having that is described here is when the extra requirement matches the package in install_requires, e.g.:

setup(
    install_requires=['the-same-package'],
    extras_require=[
        'the-same-package[an-extra]'
    ]
)

Because install_requires gets installed first, when Pip checks to see if requirements declared in extras_require exist it resolves to true.

[nickwilliams-eventbrite]

I am also still having this problem in Pip 9.0.3 and 10.0.1. I'll provide a specific example. PySOA provides a PyTest plugin, and that plugin has a set of requirements, represented in the extra pysoa[pytest]. Projects that use PySOA would put pysoa in their normal requirements (so that the PyTest plugin extras aren't included in the distribution), but would put pysoa[pytest] in their testing extras so that the PyTest plugin extras are installed for testing:

    install_requires={ ... 'pysoa' ...},
    tests_require={ ... 'pysoa[pytest]' ... },
    extras_require={
        'testing': { ... 'pysoa[pytest]' ...},
    },

In our CI environment, when we run python setup.py test, it works perfectly. The pysoa[pytest] requirement, with extras, is installed, and tests pass. However, locally, we install it with pip install -e .[testing], and that approach does not install the extras from pysoa[pytest], so our tests fail locally unless we manually install the extra requirements.

This really seems like a fundamental problem to me, and doesn't seem like a duplicate of any other issue that has been reported except for maybe #3147, which was marked as a duplicate of #3189, which is not at all what the problem here is. Multiple pull requests (#2716, #3105, et. al) have purported to fix this, but it's clearly not fixed. Is there a non-closed issue somewhere that describes this problem that hasn't been marked as a duplicate of some other issue that doesn't, actually, describe this problem? (Addendum: I just found #4957 ... that might fit the notion of a non-closed issue that describes this problem ... ?)

[ankostis]

I'm having the same problem. And it included the case where the extras define an existing package with version-constraints.

setup(name='foo',
      install_requires=['bar'],
      extras_require={'pinbar': 'bar==1.2.3']})

Please reopen.

[ankostis]

By debugging i think that the comparison between the main and the extra requirement for the same distribution at

pip/src/pip/_internal/req/req_set.py

Line 123 in 2b3609a

if install_req.constraint or not existing_req.constraint:

misses the cases where they have different pinned-versions or extras.
Could this be the source for the problems described here?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.