Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extras not getting installed when the package that includes the extras has already been installed #4957

Closed
eeshangarg opened this issue Jan 5, 2018 · 10 comments
Labels
C: dependency resolution About choosing which dependencies to install C: extras Handling optional dependencies

Comments

@eeshangarg
Copy link

  • Pip version: 9.0.1
  • Python version: 3.5.2
  • Operating system: Ubuntu 16.04.3 LTS

Description:

Note: I initially reported this here pypa/setuptools#1247, but was advised to report it here instead.

A similar issue was reported and fixed here for pip.

In our case:

  1. We have a repo with multiple packages in it.
  2. We have the zulip_bots package that depends on requests and simple_salesforce. simple_salesforce further depends on requests[security].
  3. The zulip_bots package also depends on the zulip package in the same repo which also depends on/installs requests. So, it is clear that requests does get installed whether you install zulip or zulip_bots.
  4. We usually use our provision script to set up a dev env and install the zulip and zulip_bots packages.
  5. But recently, when we ran this executable (part of the zulip_bots package), we ran into:
pkg_resources.DistributionNotFound: The 'cryptography>=1.3.4; extra == "security"' distribution was not found and is required by requests

The above error led us to conclude that for some reason, requests[security] was not getting satisfied as expected.

It would mean a lot to us if someone could please shed some light on whether this is a re-manifestation of #3189 and #3198 or simply a quirk in our understanding of how extra dependencies are resolved! Thanks! :)

roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 5, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 5, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 5, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 5, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 6, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 6, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to roberthoenig/python-zulip-api that referenced this issue Jan 6, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
roberthoenig added a commit to zulip/python-zulip-api that referenced this issue Jan 6, 2018
This makes it straightforward to add dependencies for a bot,
and works around pypa/pip#4957.
@mitar
Copy link

mitar commented Jan 9, 2018

I think I have a similar case. Reproduction is as follows. If I have a requirements.txt file like:

git+https://gitlab.com/datadrivendiscovery/[email protected]
scikit-learn

and I do:

pip install --process-dependency-links -r requirements.txt

(--process-dependency-links has nothing with this report, this is just because the package I am demonstrating with requires it)

Then scipy is not installed, despite being and extra dependency alldeps in scikit-learn[alldeps] from d3m_metadata package (installed from the git repo above). scipy installs correctly if there is no scikit-learn entry in requirements.txt. It seems extras do not get combined as one would expect. This is really tricky because one cannot really know all dependencies packages can have. So If in my project I need d3m_metadata package and just basic scikit-learn, I would like to list only those. But now what happens i that d3m_metadata package does not get all expected dependencies and it fails.

@nickwilliams-eventbrite
Copy link

nickwilliams-eventbrite commented Apr 26, 2018

I am having the following problem (commented on #3516). Is this issue the cause of that problem?

I am also still having this problem in Pip 9.0.3 and 10.0.1. I'll provide a specific example. PySOA provides a PyTest plugin, and that plugin has a set of requirements, represented in the extra pysoa[pytest]. Projects that use PySOA would put pysoa in their normal requirements (so that the PyTest plugin extras aren't included in the distribution), but would put pysoa[pytest] in their testing extras so that the PyTest plugin extras are installed for testing:

   install_requires={ ... 'pysoa' ...},
   tests_require={ ... 'pysoa[pytest]' ... },
   extras_require={
       'testing': { ... 'pysoa[pytest]' ...},
   },

In our CI environment, when we run python setup.py test, it works perfectly. The pysoa[pytest] requirement, with extras, is installed, and tests pass. However, locally, we install it with pip install -e .[testing], and that approach does not install the extras from pysoa[pytest], so our tests fail locally unless we manually install the extra requirements.

@fpagnoux
Copy link

fpagnoux commented Aug 2, 2018

Same issue here 😞

Extract from my setup.py:

    extras_require = {
        'api': [
            'Some-Dep[api]'
            ],
        },
    install_requires = [
        'Some-Dep',
        ],

pip install --editable ".[api]" does not install the extra dependencies api of Some-Dep.

@TheKevJames
Copy link

TheKevJames commented Aug 9, 2018

This just bit us as well. I've run a few test cases:

$ pip install 'requests[security]' 'requests'  # installs `requests[security]`
$ pip install 'requests' 'requests[security]'  # BUG: only installs `requests`
$ pip install 'requests' 'requests[security]' && pip install 'requests[security]' # BUG: only installs `requests`, then installs `requests[security]` in the second command
$ pip install 'requests[security]' 'requests[socks]'  # BUG: only installs `requests[security]`
$ pip install 'requests[socks]' 'requests[security]'  # BUG: only installs `requests[socks]`

This bug also applies to setup.py dependencies, even if the local setup.py is not itself making use of extras:

$ cat setup.py
import setuptools; setuptools.setup(
    name='test-0', version='1.0.0', description='test 0',
    author='me', author_email='[email protected]', url='https://domain.tld',
    packages=[],
    install_requires=['requests[security]'])
$ pip install .  # installs `requests[security]`
$ pip install requests .  # BUG: only installs `requests` (and `test-0`)

The above test cases have been replicated on:

  • OSX 10.12.6, Python 3.7, pip 18.0
  • python:3.5.4 docker image, pip 9.0.1
  • python:2.7.11 docker image, pip 8.1.2
  • python:2.7.10 docker image, pip 8.0.0 (updated with pip install pip==8.0.0)

And versions of pip prior to 8.0.0 raise an error when passing requirements with two different extras, as described in #3189.

After going through this exercise, It seems clear to me that #3189 as well as its associated PR #3198 solve only a small portion of this issue, namely running rather than erroring out. It is worth noting that #988 "build a dependency resolver" is still open and describes the above issues quite clearly in point 2:

a. "first found, wins" (where the order is breadth first)

Given the above, I'm pretty sure this boils down to #988, #4957, etc, and many other bugs on various other repos (eg. googleapis/google-cloud-python#5023, googleapis/google-cloud-python#5776) all being dupes of this.

ltalirz added a commit to ltalirz/aiida-diff that referenced this issue Aug 28, 2018
ltalirz added a commit to aiidateam/aiida-diff that referenced this issue Aug 28, 2018
address aiidateam/aiida-plugin-cutter#20

 * limit aiida version to <1.0
 * switch from aiida to aiida-core
* add dependencies explicitly, working around pypi bug pypa/pip#4957
ltalirz added a commit to ltalirz/aiida-plugin-cutter that referenced this issue Aug 28, 2018
fix aiidateam#20

 * limit aiida version to <1.0
 * switch from aiida to aiida-core
 * add dependencies explicitly, working around pypi bug pypa/pip#4957
 * switch to aiida 0.12 as default minimum version
@springheeledjak
Copy link

springheeledjak commented May 2, 2019

I'm seeing this same issue with pip version 19.1 as well. As an example, if I have two extras specified, prod and test, with extras_require that look something like the following:

setup(
    [...],
    extras_require={
        'prod': ['foo'],
        'test': ['foo[bar]'],
    },
)

then I can verify that both pip install .[prod,test] and pip install .[test,prod] will install foo but fail to install it with extras as foo[bar].

I don't have a good solution for the case when two extras specify the same package but with different extras (e.g. for the above example, if test were to require foo[bar] and prod required foo[baz]) other than either merging all of the specified extras together and requiring all of them (e.g. foo[bar,baz]), or simply raising an exception because of the conflict, but either would be better than the current behavior of silent failure.

joshuagl added a commit to joshuagl/in-toto that referenced this issue Jul 3, 2019
install_requires should list "what a project minimally needs to run
correctly"[1] and the presence of these dependencies is checked at runtime
when the project is installed via pip.

The securesystemslib pynacl extras allow for faster implementations of
cryptographic algorithms, but aren't required in order to run - therefore
they should not be listed in install_requires.

Unfortunately pypa/pip#4957 makes including the securesystemslib[pynacl] extra
in extras_require a noop when securesystemslib is already installed, therefore
we add pynacl itself to extras_require so that we can keep in-synch with the
pynacl dependency in securesystemslib

1. https://packaging.python.org/discussions/install-requires-vs-requirements

Signed-off-by: Joshua Lock <[email protected]>
@chrahunt chrahunt added the C: extras Handling optional dependencies label Oct 26, 2019
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Oct 26, 2019
@NeolithEra

This comment has been minimized.

@SaschaHeyer
Copy link

any updates?

@uranusjr
Copy link
Member

uranusjr commented Aug 4, 2020

I believe the 2020 resolver does this correctly. Can anyone confirm?

@uranusjr uranusjr added the S: awaiting response Waiting for a response/more information label Aug 4, 2020
@zhyu
Copy link

zhyu commented Aug 6, 2020

I'm seeing the same issue with pip 20.2.1.
And I can confirm using pip --use-feature 2020-resolver does solve the issue for me.

@uranusjr
Copy link
Member

uranusjr commented Aug 6, 2020

Thanks for the confirmation. I’ll merge this into the #988 mega issue then.

@uranusjr uranusjr closed this as completed Aug 6, 2020
@uranusjr uranusjr removed the S: awaiting response Waiting for a response/more information label Aug 6, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: dependency resolution About choosing which dependencies to install C: extras Handling optional dependencies
Projects
None yet
Development

No branches or pull requests