pip download --platform --python-version does not work for some packages

[chadrik]

Environment

• pip version: 18.0
• Python version: 2.7
• OS: centos 7

Description

When using pip to download wheels for it does not find them despite the fact that there are obvious matches available on PyPI

How to Reproduce

I tried all of the following commands <and many other permutations) to try to make this work

pip download --python-version 36 --abi cp36 --platform linux_x86_64  "hiredis==0.3.1" --only-binary=:all: --dest ./wheelhouse -v
pip download --python-version 36 --abi cp36 --platform manylinux1_x86_64  "hiredis==0.3.1" --only-binary=:all: --dest ./wheelhouse -v
pip download --python-version 36 --abi cp36 --platform linux_x86_64  --implementation py "hiredis==0.3.1" --only-binary=:all: --dest ./wheelhouse -v

...
    Skipping link https://files.pythonhosted.org/packages/0d/d1/f0346030e4e5fbd931c7d81acdb8c81ea57733f6d87bc15f3dc28a51eb30/hiredis-0.3.1-cp36-cp36m-manylinux1_x86_64.whl#sha256=736a30992ede64b79989869e1a068fec93405dac71d87a316f22ee72e75cb6a5 (from https://pypi.org/simple/hiredis/) (requires-python:>=2.6, !=3.0.*, !=3.1.*); it is not compatible with this Python
...
No matching distribution found for hiredis==0.3.1

The wheel it skips looks perfectly valid to me: hiredis-0.3.1-cp36-cp36m-manylinux1_x86_64.whl . I assume linux_x86_64 matches manylinux1_x86_64, and even if it doesn't, I tried specifiying manylinux1_x86_64 just to be safe.

The same thing happens when trying to download wheels for python 2.7.

The documentation on how these work together is pretty sorely lacking (

• what is the valid list of platforms and what is the logic for how they match?
• the provided examples about macosx use the wrong specifier, AFAICT, using a dash instead of an underscore.
• the pep that is referred to is far too long to grok and it also seems out of date
• the verbose output about the skipped wheels above provides no information about how pip came to the conclusion that the wheel "is not compatible with this Python". It would be nice if the note provided some more info about why the wheel is not considered compatible.

Along with the other issue I posted #5369 it feels like the intersection of what is required to be useful and what actually works is frustratingly small.

[noonat]

I would just like to mention that I am also running into this issue on CentOS while trying to download MacOS wheels. This issue appears to also be reproducible for both Ubuntu and CentOS via Docker. For Ubuntu:

# create a docker container
$ docker run -it ubuntu bash

# install pip and try to download a macos package
$ apt-get update
$ apt-get install -y python3 python3-virtualenv
$ python3.6 -m virtualenv --python=python3.6 env
$ ./env/bin/python3.6 -m pip install -U pip
$ ./env/bin/python3.6 -m pip download --only-binary :all: --platform macosx_10_10_intel --python-version 27 --abi cp27 -v numpy==1.16.0

And the snipped output from pip:

...
    Skipping link https://files.pythonhosted.org/packages/e4/97/167eb80dadcf2905b58d66ada6c128d3ec5e8595beb02457b881e7399be3/numpy-1.16.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=a80ecac5664f420556a725a5646f2d1c60a7c0489d68a38b5056393e949e27ac (from https://pypi.org/simple/numpy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*); it is not compatible with this Python
...
  Could not find a version that satisfies the requirement numpy==1.16.0 (from versions: 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.1, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.1, 1.10.2, 1.10.4)
...
Removed build tracker '/tmp/pip-req-tracker-0ljej8rn'
No matching distribution found for numpy==1.16.0
Exception information:
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/env/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 164, in run
    resolver.resolve(requirement_set)
  File "/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 269, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/env/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 196, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/env/lib/python3.6/site-packages/pip/_internal/index.py", line 688, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for numpy==1.16.0

These steps on CentOS have the same result:

# create a docker container
$ docker run -it centos bash

# install pip and try to download a macos package
$ yum install -y centos-release-scl
$ yum install -y rh-python36
$ scl enable rh-python36 bash
$ python3.6 -m virtualenv env
$ ./env/bin/python3.6 -m pip install -U pip
$ ./env/bin/python3.6 -m pip download --only-binary :all: --platform macosx_10_10_intel --python-version 27 --abi cp27 -v numpy==1.16.0

Note that it still seems to exhibit this behavior even if you are actually running under CPython 2.7 (but on Linux instead of MacOS). The packages download without issue when running the same command on MacOS.

[AbdealiLoKo]

Was trying pip download and had some very similar issues. The documentation was not super useful, but after some trial and error, the command that got it working for me for hiredis is:

pip download \
  --python-version 36 --abi cp36m --platform manylinux1_x86_64  --implementation cp \
  "hiredis==0.3.1" --only-binary=:all: --dest /tmp/deps

Observations:

• The --implementation py doesnt look right as it is CPython specific and there is no implementation agnostic wheel available.
• The api in the wheel says cp36m, while the download command being tried originally was cp36

Similarly, for the numpy example, changing the abi to cp27m works for me:

pip download \
  --python-version 27 --abi cp27m --platform macosx_10_10_intel \
  "numpy==1.16.0" --only-binary=:all: --dest /tmp/deps/pypi

Pretty much got this via trial and error ... Better documentation would be really helpful

[cjerdonek]

FYI, I just posted PR #6540 to address this portion of the original comment:

• the verbose output about the skipped wheels above provides no information about how pip came to the conclusion that the wheel "is not compatible with this Python". It would be nice if the note provided some more info about why the wheel is not considered compatible.

[cjerdonek]

It would be nice if the note provided some more info about why the wheel is not considered compatible.

I was thinking more about this because #6540 only provides half of the information (namely the wheel's file tags). Currently, I don't think there's any way for the user to know the other half of the information -- namely what tags the current Python version matches (or the requested version in the case of options like --python-version being passed). Since this list can be very long (e.g. ~400 on my system), this info can't simply be logged.

I was thinking then that a new command called something like pip tags could be useful. It could start out simply by printing out a list of tags for the current Python in sorted order, and the command could accept the same environment options like --platform and --python-version that pip install accepts. This would give people a way to see what tags their options are resulting in. Looking at the get_supported() function inside pip's pep425tags.py, I don't think there's any other way for the user to get this information, since you can really only get it empirically by running the code. A command like this should be relatively simple because it would just be a matter of calling get_supported() and printing out the resulting list.

Maybe later the command could accept arguments for things like evaluating certain wheel filenames against the current Python version, with more verbose logging, etc. The command could be thought of as a helper for working with PEP 425 compatibility tags as implemented by pip: https://www.python.org/dev/peps/pep-0425/#details

[cjerdonek]

FYI, yesterday I posted PR #6638 that will help with this. It implements an initial pip debug command that displays, among other things, the list of compatible pep 425 tags. It also accepts the same --python-version, --platform, etc. options that pip download and install support so that one would have a way to experiment and see what tags result in those cases, too.

So combined with PR #6540 (already merged), this would give you a better pathway to troubleshoot issues like the ones in this issue. Namely, with PR #6540 merged, the logs will tell you for each wheel what tags the wheel generates. And PR #6638 would give you an interactive way to see whether you can generate any of those tags (and locally, without having to do any actual installs, etc).

[jacobscgc]

I have a similar issue but more related to that it does not recognize --python-version 36 as being python >3.5. This leads pip to download an older version, for example when trying to download numpy:

pip download -v -v -v --no-cache-dir --python-version 36 --abi cp36m --platform manylinux1_x86_64 --only-binary=:all: --no-deps --exists-action i numpy

I get a whole list of output, stating that version 1.16.4 is the newest version available for this version of python, despite the fact that version 1.17.0 is available for python 3.6.

Using version 1.16.4 (newest of versions: 1.11.3, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.13.3, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4)

It does accept 1.16.4 because the versions accepted are all != tests:

Found link https://files.pythonhosted.org/packages/87/2d/e4656149cbadd3a8a0369fcd1a9c7d61cc7b87b3903b85389c70c989a696/numpy-1.16.4-cp36-cp36m-manylinux1_x86_64.whl#sha256=27e11c7a8ec9d5838bc59f809bfa86efc8a4fd02e58960fa9c49d998e14332d5 (from https://pypi.org/simple/numpy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 1.16.4

However, python version >=3.5 is not accepted when --python-version=36:

The package https://files.pythonhosted.org/packages/19/b9/bda9781f0a74b90ebd2e046fde1196182900bd4a8e1ea503d3ffebc50e7c/numpy-1.17.0-cp36-cp36m-manylinux1_x86_64.whl#sha256=9588c6b4157f493edeb9378788dcd02cb9e6a6aeaa518b511a1c79d06cbd8094 (from https://pypi.org/simple/numpy/) (requires-python:>=3.5) is incompatible with the pythonversion in use. Acceptable python versions are:>=3.5

Which seems like an error to me but maybe I'm mistaken. The same issues occur for pandas (downloads 0.24.2 instead of 0.25.0) and scipy (downloads 1.2.2 instead of 1.3.1).

[cjerdonek]

@jacobscgc What version of pip are you using?

[cjerdonek]

@jacobscgc Actually, file a new issue for your issue because yours seems different, and then it can be discussed there.

[maltfield]

+1 for this feature. I've encountered this issue, but also I'm trying to write a script to pip download files for all my app's target platforms, but the script only runs on linux. So it would be great if I could tell pip to download the packages as if I were on Windows or MacOS when the script is running on linux.

Edit: In my case I discovered that pip was too old of a version. Upgrading pip allowed the wheel to be downloaded.

It would be great if pip's --verbose output would indicate that the link was skipped because pip was too old.

[uranusjr]

Unfortunately we can’t go back in time and modify the error message in old pip versions. New pip versions allow you to download the wheel, and contain improved error messaging if it can’t.

[katzdm]

I wonder if the most straightforward patch here could be to augment the semantics of --platform and --abi flags, to accept comma-separated lists of acceptable values?

Looking at get_supported(), and given that the underlying functions from packaging.tags already accept Iterable[str] for both abis and platforms, this looks like it could be a minor (and non-breaking?) change.

We could then invoke

pip download \
    --python-version 36 \
    --abi "cp36m,cp36,abi3,none" \
    --platform "manylinux2014_x86_64,manylinux2010_x86_64,manylinux1_x86_64,linux_x86_64,any" \
    --no-deps hiredis==0.3.1

which, while a bit less than ideal, might at least get us the right .whl from a single command. Thoughts?

[katzdm]

Wanted to ping this thread - Would the maintainers be open to a PR allowing multiple --platform and --abi values for pip download? Would be happy to put one together.

[pradyunsg]

I'm not sure I understand the nuances here well enough, but if it is indeed simple enough to file an implementation PR, please feel free to do so. It'll even bring us to a nice round number of 80 open PRs. :P

[katzdm]

@pradyunsg Sounds great! I've opened #8820 with the idea I had in mind - the example above seems to work.

    --python-version 36 \
    --abi "cp36m,cp36,abi3,none" \
    --platform "manylinux2014_x86_64,manylinux2010_x86_64,manylinux1_x86_64,linux_x86_64,any" \
    --no-deps hiredis==0.3.1

Let me know if the general shape of it looks good, and whether you have any comments/suggestions - Happy to augment tests, once (and if) there's agreement that this is a good direction. Thanks for your time!

[pradyunsg]

Consolidating this into #11664