--ignore-requires-python has no effect

[asottile]

Environment

• pip version:

$ ./venv/bin/pip --version
pip 19.0.3 from /tmp/t/venv/site-packages/pip (python 3.5)

• Python version:

$ ./venv/bin/python --version
Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04)
[PyPy 6.0.0 with GCC 6.2.0 20160901]

• OS:

$ uname -a
Linux asottile-VirtualBox 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

Description / Expected behaviour / How to Reproduce

I expect this to succeed:

$ ./venv/bin/pip install --ignore-requires-python black -v

black requires python>=3.6, though I am running pypy3.5 and explicitly asking pip to --ignore-requires-python

Output

$ ./venv/bin/pip install --ignore-requires-python black -v
Created temporary directory: /tmp/pip-ephem-wheel-cache-7_gvlskb
Created temporary directory: /tmp/pip-req-tracker-t4ef_ow9
Created requirements tracker '/tmp/pip-req-tracker-t4ef_ow9'
Created temporary directory: /tmp/pip-install-3x0i9w7v
Collecting black
  1 location(s) to search for versions of black:
  * https://pypi.org/simple/black/
  Getting page https://pypi.org/simple/black/
  Looking up "https://pypi.org/simple/black/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/black/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/black/
    The package https://files.pythonhosted.org/packages/22/5a/ebe5881cbf07435ea1ad76119ee142f42fb7a333d562c202db015b959ae7/black-18.3a0-py3-none-any.whl#sha256=7183263650ba3071034e90b40a1ea74abccbd32cf525cef6d7914479dbe7f2fb (from https://pypi.org/simple/black/) (requires-python:>=3.6) is incompatible with the python version in use. Acceptable python versions are: >=3.6
    The package https://files.pythonhosted.org/packages/83/a2/fae3ddb516a52a0f37e6e2ef3bd789b6d093d2c7015a41a41bff30f39897/black-18.3a0.tar.gz#sha256=0461c7a52b5beb378936bf642753dec7a45305c96c6129d540b9c53227121a5a (from https://pypi.org/simple/black/) (requires-python:>=3.6) is incompatible with the python version in use. Acceptable python versions are: >=3.6
    The package https://files.pythonhosted.org/packages/61/b4/3a39dcf59342f53680c59a5e44c0b172ea5d6470a5aaba8ae3cdbfccc1e7/black-18.3a1-py3-none-any.whl#sha256=04edc87b51b34a7aa093f39bf207e2ebc3612b26f2c79fdf8ff13b2d60622c9a (from https://pypi.org/simple/black/) (requires-python:>=3.6) is incompatible with the python version in use. Acceptable python versions are: >=3.6

[ many repeated lines removed ]


    The package https://files.pythonhosted.org/packages/e8/5f/0f79fcd943ba465cbd4bf303c9794970c13a95e5456630de9f72e7f37ad4/black-18.9b0.tar.gz#sha256=e030a9a28f542debc08acceb273f228ac422798e5215ba2a791a6ddeaaca22a5 (from https://pypi.org/simple/black/) (requires-python:>=3.6) is incompatible with the python version in use. Acceptable python versions are: >=3.6
    Skipping link https://files.pythonhosted.org/packages/30/62/cf549544a5fe990bbaeca21e9c419501b2de7a701ab0afb377bc81676600/black-19.3b0-py36-none-any.whl#sha256=09a9dcb7c46ed496a9850b76e4e825d6049ecd38b611f1224857a79bd985a8cf (from https://pypi.org/simple/black/) (requires-python:>=3.6); it is not compatible with this Python
    The package https://files.pythonhosted.org/packages/89/07/aebb10fb8f2ffbac672dfbebffa724643bc84cf012a57737a622d1dabddb/black-19.3b0.tar.gz#sha256=68950ffd4d9169716bcb8719a56c07a2f4485354fec061cdd5910aa07369731c (from https://pypi.org/simple/black/) (requires-python:>=3.6) is incompatible with the python version in use. Acceptable python versions are: >=3.6
  Could not find a version that satisfies the requirement black (from versions: )
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-t4ef_ow9'
No matching distribution found for black
Exception information:
Traceback (most recent call last):
  File "/tmp/t/venv/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/tmp/t/venv/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/tmp/t/venv/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/tmp/t/venv/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/tmp/t/venv/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/tmp/t/venv/site-packages/pip/_internal/operations/prepare.py", line 269, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/tmp/t/venv/site-packages/pip/_internal/req/req_install.py", line 196, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/tmp/t/venv/site-packages/pip/_internal/index.py", line 688, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for black

[asottile]

I'm ok with this being closed as "works as intended" though the option should probably be deprecated and/or removed if it has no effect

[cjerdonek]

Are you able to get the install to work by passing something for --python-version? My understanding is that --ignore-requires-python has a slightly different purpose (introduced in response to issue #3846).

[asottile]

nope:

$ pip install --ignore-requires-python --python-version 36 black
ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).

[asottile]

I even reverted back to when it was introduced -- it didn't work then either:

$ pip install git+https://github.com/pypa/pip@8df742e56db99c48f7ad85085501fe23456b419c
Collecting git+https://github.com/pypa/pip@8df742e56db99c48f7ad85085501fe23456b419c
  Cloning https://github.com/pypa/pip (to revision 8df742e56db99c48f7ad85085501fe23456b419c) to ./pip-req-build-mdzrw938
Building wheels for collected packages: pip
  Building wheel for pip (setup.py) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-3htnnj3r/wheels/f4/06/5d/1652f21a0ecf6bed88b63647ec859e0384f0f07fcae05c57f3
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-8.2.0.dev0
$ pip install --ignore-requires-python black
Collecting black
  Cache entry deserialization failed, entry ignored
  Could not find a version that satisfies the requirement black (from versions: )
No matching distribution found for black
You are using pip version 8.2.0.dev0, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[cjerdonek]

The ignore-requires option wasn’t meant to be used for the purpose you’re trying to use it. I was wondering if another option could help though.

[asottile]

my reading of #3846 especially these comments seems to indicate it is intended for this:

#3846 (comment)

#3846 (comment)

2df0ea9

The --python-version / abi / etc. parameters did not even exist at the time, my reading is that this argument was intended as an escape hatch for pip refusing to install incompatible requires_python (what else could it be for?)

[cjerdonek]

ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).

When you're trying --python-version, can you also try adding --only-binary=:all: like the error message says?

[cjerdonek]

(what else could it be for?)

When installing a package, Python performs a check of the Requires-Python metadata prior to allowing it:

pip/src/pip/_internal/resolve.py

Lines 298 to 304 in ac9010e

	try:
	check_dist_requires_python(dist)
	except UnsupportedPythonVersion as err:
	if self.ignore_requires_python:
	logger.warning(err.args[0])
	else:
	raise

The --ignore-requires-python option is an escape hatch to suppress that check.

In your case, it's rejecting the package before even getting to that point (before it's even downloaded), so the command option isn't coming into play. The --python-version and related options are for controlling whether a package is eligible for download.

[cjerdonek]

Okay, it looks like there is a second application of check_requires_python() that isn't taking ignore_requires_python into account:

pip/src/pip/_internal/index.py

Lines 852 to 863 in ac9010e

	try:
	support_this_python = check_requires_python(link.requires_python)
	except specifiers.InvalidSpecifier:
	logger.debug("Package %s has an invalid Requires-Python entry: %s",
	link.filename, link.requires_python)
	support_this_python = True
	if not support_this_python:
	logger.debug("The package %s is incompatible with the python "
	"version in use. Acceptable python versions are: %s",
	link, link.requires_python)
	return None

[cjerdonek]

PS - the most important part of the log messages in your OP are getting hidden by default because GitHub doesn't wrap the long lines when using the fixed-width formatting:

... (requires-python:>=3.6) is incompatible with the python version in use. Acceptable python versions are: >=3.6

I didn't see that until now.

[cjerdonek]

(Note that the wheels will still get rejected without using —python-version.)

[asottile]

😆 I'm glad you finally took the time to look instead of assuming I'm wrong

I've been following closely the --python-version / platform / abi / etc. options, they are a non-solution until the --target requirement is lifted: #5453 (comment)

[uranusjr]

Out of curiosity, what does it mean to pass --ignore-requires-python without --python-version? That the Python… does not have a version? That seems a bit pointless in this case. I would expect these two to be mutually exclusive: if --python-version is passed, the value is used to check against Requires-Python (both in metadata and on index), and if --ignore-requires-python is passed, checks against Requires-Python are disabled altogether (so all links are collected on the index).

[xavfernandez]

FWIW, --ignore-requires-python was added (and still has the expected effect) for escape hatching this kind of case:

$ docker run -ti --rm python:3.4 bash
root@b30693b8f33e:/# pip install https://files.pythonhosted.org/packages/54/85/0bef63668fb170888c1a2970ec897d4528d6072f32dee27653381a332642/Django-2.2-py3-none-any.whl
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Collecting Django==2.2 from https://files.pythonhosted.org/packages/54/85/0bef63668fb170888c1a2970ec897d4528d6072f32dee27653381a332642/Django-2.2-py3-none-any.whl
  Downloading https://files.pythonhosted.org/packages/54/85/0bef63668fb170888c1a2970ec897d4528d6072f32dee27653381a332642/Django-2.2-py3-none-any.whl (7.4MB)
    100% |████████████████████████████████| 7.5MB 3.6MB/s 
Django requires Python '>=3.5' but the running Python is 3.4.10
You are using pip version 19.0.3, however version 19.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@b30693b8f33e:/# echo $?
1
root@b30693b8f33e:/# pip install https://files.pythonhosted.org/packages/54/85/0bef63668fb170888c1a2970ec897d4528d6072f32dee27653381a332642/Django-2.2-py3-none-any.whl --ignore-requires-python
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Collecting Django==2.2 from https://files.pythonhosted.org/packages/54/85/0bef63668fb170888c1a2970ec897d4528d6072f32dee27653381a332642/Django-2.2-py3-none-any.whl
  Using cached https://files.pythonhosted.org/packages/54/85/0bef63668fb170888c1a2970ec897d4528d6072f32dee27653381a332642/Django-2.2-py3-none-any.whl
Django requires Python '>=3.5' but the running Python is 3.4.10
Collecting sqlparse (from Django==2.2)
  Downloading https://files.pythonhosted.org/packages/ef/53/900f7d2a54557c6a37886585a91336520e5539e3ae2423ff1102daf4f3a7/sqlparse-0.3.0-py2.py3-none-any.whl
Collecting pytz (from Django==2.2)
  Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 17.0MB/s 
Installing collected packages: sqlparse, pytz, Django
Successfully installed Django-2.2 pytz-2019.1 sqlparse-0.3.0
You are using pip version 19.0.3, however version 19.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@b30693b8f33e:/# echo $?
0

It might still be interesting to also use it in check_requires_python() though.

[cjerdonek]

I posted PR #6518 to address this.

[cjerdonek]

Out of curiosity, what does it mean to pass --ignore-requires-python without --python-version? That the Python… does not have a version? That seems a bit pointless in this case. I would expect these two to be mutually exclusive: if --python-version is passed, the value is used to check against Requires-Python (both in metadata and on index)

One issue that comes up when considering this is that pip's --python-version only permits a major-minor version (e.g. 3 or 34) for checking wheels, whereas the Requires-Python check uses major-minor-micro (e.g. (3, 6, 5)). How would pip's options work to support specifying both of these?

[cjerdonek]

How would pip's options work to support specifying both of these?

Incidentally, @xavfernandez, my comment / question above relates to the review comment you made less than a week ago on one of my PR's as to whether _py_version and _py_version_info should be provided separately or as a single argument.

[xavfernandez]

@cjerdonek Maybe we should migrate --python-version from 2, 27, 3, 38, ... values to dotted ones 2, 2.7, 3, 3.8, 3.6.4, ... allowing for the extra patch version.
During the transition phase, 2-digits numbers mn would be interpreted as m.n.

[cjerdonek]

@xavfernandez Something like that sounds good. However, the other issue is that because --python-version currently requires either "major" or "major-minor", it seems like we might still need a way to support saying whether, say, 3 or 36 should be used for the wheel-compatibility check if, say, 3.6.4 is given (for the Requires-Python check). Alternatively, maybe the Requires-Python code can be enhanced to support matching on a range of Python versions instead of just an exact major-minor-micro. It's not clear to me yet what's needed.

[xavfernandez]

it seems like we might still need a way to support saying whether, say, 3 or 36 should be used for the wheel-compatibility check if, say, 3.6.4 is given

Wheels selection is based on the most precise/matching first, so py36 wheel should be preferred over the py3 one (all other tags being equal).
If the user wants it the other way, providing --python-version 3.0.0 might work, and if it doesn't I'd argue it seems like a very special case that we might not want to cover.

[cjerdonek]

@cjerdonek Maybe we should migrate --python-version from 2, 27, 3, 38, ... values to dotted ones 2, 2.7, 3, 3.8, 3.6.4, ... allowing for the extra patch version.
During the transition phase, 2-digits numbers mn would be interpreted as m.n.

I just posted PR #6539 to work towards this.

[cjerdonek]

I would expect these two to be mutually exclusive: if --python-version is passed, the value is used to check against Requires-Python (both in metadata and on index),

FYI, I just filed PR #6577, which does the above,