[regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins #9243

webknjaz · 2020-12-07T17:45:44Z

What did you want to do?
Use pip-tools generated requirements.txt + constraints.txt pair with hashes via pip install -r requirements.txt -c constraints.txt. It seems to be unintentionally broken with the new resolvelib-based dependency resolver and works with the old one.

This seems to be related to #8792 but PR #8839 that claims to fix it doesn't add the specific case I'm presenting here to tests and so it is probably overlooked.

Please find the repro below. Note that I'm adding an unconstrainted direct env dep to myst.in and generate the concrete pins with hashes in myst.txt for simplicity but the case I've actually hit in the wild was with >= 0.10.0 (that shouldn't matter here).

STR + Output

$ python -m pip install pip-tools -U
Collecting pip-tools
  Downloading pip_tools-5.4.0-py2.py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 1.3 MB/s 
Requirement already satisfied, skipping upgrade: six in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (1.15.0)
Requirement already satisfied, skipping upgrade: click>=7 in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (7.1.2)
Requirement already satisfied, skipping upgrade: pip>=20.1 in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (20.2.3)
Installing collected packages: pip-tools
  Attempting uninstall: pip-tools
    Found existing installation: pip-tools 5.3.1
    Uninstalling pip-tools-5.3.1:
      Successfully uninstalled pip-tools-5.3.1
Successfully installed pip-tools-5.4.0
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.

$ echo myst-parser > myst.in

$ python -m piptools compile --generate-hashes --output-file myst.txt myst.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes --output-file=myst.txt myst.in
#
alabaster==0.7.12 \
    --hash=sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359 \
    --hash=sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02 \
    # via sphinx
attrs==20.3.0 \
    --hash=sha256:31b2eced602aa8423c2aea9c76a724617ed67cf9513173fd3a4f03e3a929c7e6 \
    --hash=sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700 \
    # via markdown-it-py
babel==2.9.0 \
    --hash=sha256:9d35c22fcc79893c3ecc85ac4a56cde1ecf3f19c540bba0922308a6c06ca6fa5 \
    --hash=sha256:da031ab54472314f210b0adcff1588ee5d1d1d0ba4dbd07b94dba82bde791e05 \
    # via sphinx
certifi==2020.12.5 \
    --hash=sha256:1a4995114262bffbc2413b159f2a1a480c969de6e6eb13ee966d470af86af59c \
    --hash=sha256:719a74fb9e33b9bd44cc7f3a8d94bc35e4049deebe19ba7d8e108280cfd59830 \
    # via requests
chardet==3.0.4 \
    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
    # via requests
docutils==0.16 \
    --hash=sha256:0c5b78adfbf7762415433f5515cd5c9e762339e23369dbe8000d84a4bf4ab3af \
    --hash=sha256:c2de3a60e9e7d07be26b7f2b00ca0309c207e06c100f9cc2a94931fc75a478fc \
    # via myst-parser, sphinx
idna==2.10 \
    --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6 \
    --hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 \
    # via requests
imagesize==1.2.0 \
    --hash=sha256:6965f19a6a2039c7d48bca7dba2473069ff854c36ae6f19d2cde309d998228a1 \
    --hash=sha256:b1f6b5a4eab1f73479a50fb79fcf729514a900c341d8503d62a62dbc4127a2b1 \
    # via sphinx
jinja2==2.11.2 \
    --hash=sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0 \
    --hash=sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035 \
    # via sphinx
markdown-it-py==0.5.6 \
    --hash=sha256:6143d11221495edbf71beb7e455821ae6c8f0156710a1b11812662ed6dbd165b \
    --hash=sha256:dcfe4a0c6bef711cb6c42494ebf23a3cfe6f249bf995556498497dd8193bfc22 \
    # via myst-parser
markupsafe==1.1.1 \
    --hash=sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473 \
    --hash=sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161 \
    --hash=sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235 \
    --hash=sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5 \
    --hash=sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42 \
    --hash=sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff \
    --hash=sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b \
    --hash=sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1 \
    --hash=sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e \
    --hash=sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183 \
    --hash=sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66 \
    --hash=sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b \
    --hash=sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1 \
    --hash=sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15 \
    --hash=sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1 \
    --hash=sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e \
    --hash=sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b \
    --hash=sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905 \
    --hash=sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735 \
    --hash=sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d \
    --hash=sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e \
    --hash=sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d \
    --hash=sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c \
    --hash=sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21 \
    --hash=sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2 \
    --hash=sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5 \
    --hash=sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b \
    --hash=sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6 \
    --hash=sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f \
    --hash=sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f \
    --hash=sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2 \
    --hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
    --hash=sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be \
    # via jinja2
myst-parser==0.12.10 \
    --hash=sha256:4612c46196e0344bb7e49dbc3deb288f9b9a88fcf6e9f210f7f3ea5bc9899bfc \
    --hash=sha256:a5311da4398869e596250d5a93b523735c3beb8bc9d3eba853223c705802043b \
    # via -r myst.in
packaging==20.7 \
    --hash=sha256:05af3bb85d320377db281cf254ab050e1a7ebcbf5410685a9a407e18a1f81236 \
    --hash=sha256:eb41423378682dadb7166144a4926e443093863024de508ca5c9737d6bc08376 \
    # via sphinx
pygments==2.7.3 \
    --hash=sha256:ccf3acacf3782cbed4a989426012f1c535c9a90d3a7fc3f16d231b9372d2b716 \
    --hash=sha256:f275b6c0909e5dafd2d6269a656aa90fa58ebf4a74f8fcf9053195d226b24a08 \
    # via sphinx
pyparsing==2.4.7 \
    --hash=sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1 \
    --hash=sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b \
    # via packaging
pytz==2020.4 \
    --hash=sha256:3e6b7dd2d1e0a59084bcee14a17af60c5c562cdc16d828e8eba2e683d3a7e268 \
    --hash=sha256:5c55e189b682d420be27c6995ba6edce0c0a77dd67bfbe2ae6607134d5851ffd \
    # via babel
pyyaml==5.3.1 \
    --hash=sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97 \
    --hash=sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76 \
    --hash=sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2 \
    --hash=sha256:6034f55dab5fea9e53f436aa68fa3ace2634918e8b5994d82f3621c04ff5ed2e \
    --hash=sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648 \
    --hash=sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf \
    --hash=sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f \
    --hash=sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2 \
    --hash=sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee \
    --hash=sha256:ad9c67312c84def58f3c04504727ca879cb0013b2517c85a9a253f0cb6380c0a \
    --hash=sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d \
    --hash=sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c \
    --hash=sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a \
    # via myst-parser
requests==2.25.0 \
    --hash=sha256:7f1a0b932f4a60a1a65caa4263921bb7d9ee911957e0ae4a23a6dd08185ad5f8 \
    --hash=sha256:e786fa28d8c9154e6a4de5d46a1d921b8749f8b74e28bde23768e5e16eece998 \
    # via sphinx
snowballstemmer==2.0.0 \
    --hash=sha256:209f257d7533fdb3cb73bdbd24f436239ca3b2fa67d56f6ff88e86be08cc5ef0 \
    --hash=sha256:df3bac3df4c2c01363f3dd2cfa78cce2840a79b9f1c2d2de9ce8d31683992f52 \
    # via sphinx
sphinx==3.3.1 \
    --hash=sha256:1e8d592225447104d1172be415bc2972bd1357e3e12fdc76edf2261105db4300 \
    --hash=sha256:d4e59ad4ea55efbb3c05cde3bfc83bfc14f0c95aa95c3d75346fcce186a47960 \
    # via myst-parser
sphinxcontrib-applehelp==1.0.2 \
    --hash=sha256:806111e5e962be97c29ec4c1e7fe277bfd19e9652fb1a4392105b43e01af885a \
    --hash=sha256:a072735ec80e7675e3f432fcae8610ecf509c5f1869d17e2eecff44389cdbc58 \
    # via sphinx
sphinxcontrib-devhelp==1.0.2 \
    --hash=sha256:8165223f9a335cc1af7ffe1ed31d2871f325254c0423bc0c4c7cd1c1e4734a2e \
    --hash=sha256:ff7f1afa7b9642e7060379360a67e9c41e8f3121f2ce9164266f61b9f4b338e4 \
    # via sphinx
sphinxcontrib-htmlhelp==1.0.3 \
    --hash=sha256:3c0bc24a2c41e340ac37c85ced6dafc879ab485c095b1d65d2461ac2f7cca86f \
    --hash=sha256:e8f5bb7e31b2dbb25b9cc435c8ab7a79787ebf7f906155729338f3156d93659b \
    # via sphinx
sphinxcontrib-jsmath==1.0.1 \
    --hash=sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178 \
    --hash=sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8 \
    # via sphinx
sphinxcontrib-qthelp==1.0.3 \
    --hash=sha256:4c33767ee058b70dba89a6fc5c1892c0d57a54be67ddd3e7875a18d14cba5a72 \
    --hash=sha256:bd9fc24bcb748a8d51fd4ecaade681350aa63009a347a8c14e637895444dfab6 \
    # via sphinx
sphinxcontrib-serializinghtml==1.1.4 \
    --hash=sha256:eaa0eccc86e982a9b939b2b82d12cc5d013385ba5eadcc7e4fed23f4405f77bc \
    --hash=sha256:f242a81d423f59617a8e5cf16f5d4d74e28ee9a66f9e5b637a18082991db5a9a \
    # via sphinx
urllib3==1.26.2 \
    --hash=sha256:19188f96923873c92ccb987120ec4acaa12f0461fa9ce5d3d0772bc965a39e08 \
    --hash=sha256:d8ff90d979214d7b4f8ce956e80f4028fc6860e4431f731ea4a8c08f23f99473 \
    # via requests

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools
The generated requirements file may be rejected by pip install. See # WARNING lines for details.

$ python -m pip -V                                                           
pip 20.2.3 from ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pip (python 3.9)

$ python -m pip --use-feature=2020-resolver install -r myst.in -c myst.txt 
Collecting myst-parser
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    myst-parser from https://files.pythonhosted.org/packages/2f/f0/bf66b8c5428dd7352cbc258981dce95fcd3ae63e66aa843f3b0f5986c3ba/myst_parser-0.12.10-py3-none-any.whl#sha256=a5311da4398869e596250d5a93b523735c3beb8bc9d3eba853223c705802043b (from -r myst.in (line 1))
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.

Additional information

Example crash: https://github.com/ansible/pylibssh/pull/157/checks?check_run_id=1511500384#step:9:15.

The text was updated successfully, but these errors were encountered:

webknjaz · 2020-12-07T17:51:47Z

Actually, here's a smaller repro, with just one dependency:

$ cat pyparsing-req.in                                                                
pyparsing

$ cat pyparsing-req.txt 
pyparsing==2.4.7 \
    --hash=sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1 \
    --hash=sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b

$ python -m pip --use-feature=2020-resolver install -r pyparsing-req.in -c pyparsing-req.txt   
Requirement already satisfied: pyparsing in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from -r pyparsing-req.in (line 1)) (2.4.7)
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.

$ python -m pip uninstall pyparsing                                       
Found existing installation: pyparsing 2.4.7
Uninstalling pyparsing-2.4.7:
  Would remove:
    ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pyparsing-2.4.7.dist-info/*
    ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pyparsing.py
Proceed (y/n)? y
  Successfully uninstalled pyparsing-2.4.7

$ python -m pip --use-feature=2020-resolver install -r pyparsing-req.in -c pyparsing-req.txt
Collecting pyparsing
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pyparsing from https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl#sha256=ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b (from -r pyparsing-req.in (line 1))
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.

Basically, pip "sees" the --hash on the pinned dep and then seems to "forget" that it was pinned.

uranusjr · 2020-12-07T17:55:00Z

Previous discussion: #9020

webknjaz · 2020-12-10T14:47:43Z

@uranusjr I saw that but I'm not sure it discusses exactly this case. Also, it doesn't look like there's an agreement among the pip maintainers on how this should actually work but nevertheless, I wanted to ensure this case is documented because it's important to me personally. It's a quite common paradigm among the pip-tools users that, among other things, showcases embedding -c constraints.txt in requirements.txt files in some cases, in README.

Ref: pypa/pip#9243

aawilson · 2021-01-06T21:29:05Z

We're running into this at work, we have formerly working constraints files paired with requirements that explicitly pin exactly the versions/hashes that pip output says is missing.

webknjaz · 2021-01-07T00:35:12Z

@pradyunsg is there a decision that needs to be made to get this going? It's not clear if it is accepted as a regression and a PR is welcome or not.

pfmoore · 2021-01-07T09:08:54Z

From a previous comment:

Previous discussion: #9020

That ticket was closed with a documentation update, so I guess that applies here as well.

uranusjr · 2021-01-07T09:51:33Z

I took a quick look at the code. The fix should not be too difficult, but requires some refactoring to not get messy. I don’t personally want to invest time doing that before the legacy resolver is removed.

pradyunsg · 2021-01-07T11:57:18Z

I'm on board for fixing the logic in the hash checking, to check that exactly 1 candidate is available, instead of checking that the requirement it got is a pin.

Beyond that, I don't have any thoughts.

Ref: pypa/pip#9243

This is necessary because the most recent versions of pip have a bug with using `-c` having `--require-hashes` or entries with extras. Ref: pypa/pip#9243

blueyed · 2021-10-15T16:40:56Z

I've quickly looked into this, and found that _iter_candidates_from_constraints is likely/maybe meant to help with that, but does not yield a candidate, since the constraint has no links.

Code ref:

pip/src/pip/_internal/resolution/resolvelib/factory.py

Lines 324 to 345 in 9d25d2a

    
               def _iter_candidates_from_constraints( 
        
                   self, 
        
                   identifier: str, 
        
                   constraint: Constraint, 
        
                   template: InstallRequirement, 
        
               ) -> Iterator[Candidate]: 
        
                   """Produce explicit candidates from constraints. 
        
                   This creates "fake" InstallRequirement objects that are basically clones 
        
                   of what "should" be the template, but with original_link set to link. 
        
                   """ 
        
                   for link in constraint.links: 
        
                       self._fail_if_link_is_unsupported_wheel(link) 
        
                       candidate = self._make_candidate_from_link( 
        
                           link, 
        
                           extras=frozenset(), 
        
                           template=install_req_from_link_and_ireq(link, template), 
        
                           name=canonicalize_name(identifier), 
        
                           version=None, 
        
                       ) 
        
                       if candidate: 
        
                           yield candidate

Additional info / clarifying my use case

My use case is having a requirements.txt file with hashes (produced by pdm, pip-tools/pip-compile etc), and then using this for constraints to install some dependency out of it:
req.txt:

pdm==1.9.0 \
    --hash=sha256:caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb \
    --hash=sha256:4de2f27bdce3a155fd56fb02f1c67d326ee24aa8083d8acaa2868e82d9c01999

pip install -c req.txt pdm then fails with ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. (apparently having hashes in a constraints file enables require-hashes mode).
With pip install -c req.txt pdm==1.9.0 (adding/copying the version from the constraint): ERROR: Hashes are required in --require-hashes mode, but they are missing from some requirements. … pdm==1.9.0 --hash=sha256:caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb).

uranusjr · 2021-10-16T02:26:17Z

_iter_candidates_from_constraints is for name @ url format constraints (this kind of specifiers constrain the requirement to only one possible candidate URL without additional version-URL resolution). For name == version constraints (which still requires additional resolution to look up that pinned version into something pip can download), you want to look at _iter_found_candidates (or find_candidates which populates the hashes argument for the function from constraints).

blueyed · 2021-10-16T08:20:53Z

Thanks for the clarification, I must have confused this.
(Part of) The problem appears to be a missing original_link, and not link also:
The error happens here:

[10] > …/Vcs/pip/src/pip/_vendor/resolvelib/resolvers.py(173)_add_to_criteria()

138         def _add_to_criteria(self, criteria, requirement, parent):
…
167
168             criterion = Criterion(
169                 candidates=build_iter_view(matches),
170                 information=information,
171                 incompatibilities=incompatibilities,
172             )
173  ->         if not criterion.candidates:
174                 raise RequirementsConflicted(criterion)
175             criteria[identifier] = criterion
HashUnpinned: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:\n    pdm from https://files.pythonhosted.org/packages/09/4f/c8efd28754f01618b876799e3114cf4d612365dc…
(Pdb++) bottom
378             # Unpinned packages are asking for trouble when a new version
379             # is uploaded.  This isn't a security check, but it saves users
380             # a surprising hash mismatch in the future.
381             # file:/// URLs aren't pinnable, so don't complain about them
382             # not being pinned.
383             if req.original_link is None and not req.is_pinned:
384  ->             raise HashUnpinned()
385
386             # If known-good hashes are missing for this requirement,
387             # shim it with a facade object that will provoke hash
388             # computation and then raise a HashMissing exception
389             # showing the user what the hash should be.
390             return req.hashes(trust_internet=False) or MissingHashes()
(Pdb++) bt
  [ 0] …/Vcs/pip/.venv/bin/pip(33)<module>()
       sys.exit(load_entry_point('pip', 'console_scripts', 'pip')())
  [ 1] …/Vcs/pip/src/pip/_internal/cli/main.py(70)main()
       return command.main(cmd_args)
  [ 2] …/Vcs/pip/src/pip/_internal/cli/base_command.py(98)main()
       return self._main(args)
  [ 3] …/Vcs/pip/src/pip/_internal/cli/base_command.py(212)_main()
       return run(options, args)
  [ 4] …/Vcs/pip/src/pip/_internal/cli/base_command.py(164)exc_logging_wrapper()
       status = run_func(*args)
  [ 5] …/Vcs/pip/src/pip/_internal/cli/req_command.py(205)wrapper()
       return func(self, options, args)
  [ 6] …/Vcs/pip/src/pip/_internal/commands/install.py(338)run()
       requirement_set = resolver.resolve(
  [ 7] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/resolver.py(92)resolve()
       result = self._result = resolver.resolve(
  [ 8] …/Vcs/pip/src/pip/_vendor/resolvelib/resolvers.py(482)resolve()
       state = resolution.resolve(requirements, max_rounds=max_rounds)
  [ 9] …/Vcs/pip/src/pip/_vendor/resolvelib/resolvers.py(349)resolve()
       self._add_to_criteria(self.state.criteria, r, parent=None)
  [10] …/Vcs/pip/src/pip/_vendor/resolvelib/resolvers.py(173)_add_to_criteria()
       if not criterion.candidates:
  [11] …/Vcs/pip/src/pip/_vendor/resolvelib/structs.py(151)__bool__()
       return bool(self._sequence)
  [12] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/found_candidates.py(155)__bool__()
       return any(self)
  [13] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/found_candidates.py(143)<genexpr>()
       return (c for c in iterator if id(c) not in self._incompatible_ids)
  [14] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/found_candidates.py(47)_iter_built()
       candidate = func()
  [15] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/factory.py(201)_make_candidate_from_link()
       self._link_candidate_cache[link] = LinkCandidate(
  [16] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/candidates.py(281)__init__()
       super().__init__(
  [17] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/candidates.py(156)__init__()
       self.dist = self._prepare()
  [18] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/candidates.py(225)_prepare()
       dist = self._prepare_distribution()
  [19] …/Vcs/pip/src/pip/_internal/resolution/resolvelib/candidates.py(292)_prepare_distribution()
       return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  [20] …/Vcs/pip/src/pip/_internal/operations/prepare.py(481)prepare_linked_requirement()
       return self._prepare_linked_requirement(req, parallel_builds)
  [21] …/Vcs/pip/src/pip/_internal/operations/prepare.py(520)_prepare_linked_requirement()
       hashes = self._get_linked_req_hashes(req)
> [22] …/Vcs/pip/src/pip/_internal/operations/prepare.py(384)_get_linked_req_hashes()
       raise HashUnpinned()
(Pdb++) req.original_link
(Pdb++) req.link
<Link https://files.pythonhosted.org/packages/09/4f/c8efd28754f01618b876799e3114cf4d612365dcfabb3a09ffc11d113ab8/pdm-1.9.0-py3-none-any.whl#sha256=caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb (from https://pypi.org/simple/pdm/) (requires-python:>=3.7)>

The LinkCandidate has a version, which could/should be used maybe?

[17] > …/Vcs/pip/src/pip/_internal/resolution/resolvelib/candidates.py(156)__init__()

141         def __init__(
142             self,
143             link: Link,
144             source_link: Link,
145             ireq: InstallRequirement,
146             factory: "Factory",
147             name: Optional[NormalizedName] = None,
148             version: Optional[CandidateVersion] = None,
149         ) -> None:
150             self._link = link
151             self._source_link = source_link
152             self._factory = factory
153             self._ireq = ireq
154             self._name = name
155             self._version = version
156  ->         self.dist = self._prepare()
(Pdb++) name
'pdm'
(Pdb++) version
<Version('1.9.0')>
(Pdb++) self
LinkCandidate('https://files.pythonhosted.org/packages/09/4f/c8efd28754f01618b876799e3114cf4d612365dcfabb3a09ffc11d113ab8/pdm-1.9.0-py3-none-any.whl#sha256=caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb (from https://pypi.org/simple/pdm/) (requires-python:>=3.7)')

uranusjr · 2021-11-17T09:52:11Z

Yes I think fixing original_link is the biggest part here. The version part is irralevant; what’s important is the Requirement needs a pinned specifier, not the Candidate having a pinned version.

uranusjr · 2021-11-17T09:53:48Z

I’d be happy to help if someone wants to work on a pull request for this.

q0w · 2022-03-09T22:16:26Z

Maybe because of this: #10625 (comment) ?

Also template in iter_found_candidates has neither specifier, nor hashes for make_install_req_from_link

+++template.req.specifier = specifier
+++template.hash_options = hashes._allowed
func = functools.partial(
    self._make_candidate_from_link,
    link=ican.link,
    extras=extras,
    template=template,
    name=name,
    version=ican.version,
)

It works after this changes.

pradyunsg · 2022-05-09T07:41:08Z

Reopening since #10962 was reverted. See 0c28452#r70774971 and the subsequent issue for the details.

Bumps [pip](https://github.com/pypa/pip) from 22.0.4 to 22.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>22.1 (2022-05-11)</h1> <h2>Process</h2> <ul> <li>Enable the <code>importlib.metadata</code> metadata implementation by default on Python 3.11 (or later). The environment variable <code>_PIP_USE_IMPORTLIB_METADATA</code> can still be used to enable the implementation on 3.10 and earlier, or disable it on 3.11 (by setting it to <code>0</code> or <code>false</code>).</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Revert <code>[#9243](pypa/pip#9243) <https://github.com/pypa/pip/issues/9243></code>_ since it introduced a regression in certain edge cases. (<code>[#10962](pypa/pip#10962) <https://github.com/pypa/pip/issues/10962></code>_)</li> <li>Fix missing <code>REQUESTED</code> metadata when using URL constraints. (<code>[#11079](pypa/pip#11079) <https://github.com/pypa/pip/issues/11079></code>_)</li> <li><code>pip config</code> now normalizes names by converting underscores into dashes. (<code>[#9330](pypa/pip#9330) <https://github.com/pypa/pip/issues/9330></code>_)</li> </ul> <h1>22.1b1 (2022-04-30)</h1> <h2>Process</h2> <ul> <li>Start migration of distribution metadata implementation from <code>pkg_resources</code> to <code>importlib.metadata</code>. The new implementation is currently not exposed in any user-facing way, but included in the code base for easier development.</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Drop <code>--use-deprecated=out-of-tree-build</code>, according to deprecation message. (<code>[#11001](pypa/pip#11001) <https://github.com/pypa/pip/issues/11001></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Add option to install and uninstall commands to opt-out from running-as-root warning. (<code>[#10556](pypa/pip#10556) <https://github.com/pypa/pip/issues/10556></code>_)</li> <li>Include Project-URLs in <code>pip show</code> output. (<code>[#10799](pypa/pip#10799) <https://github.com/pypa/pip/issues/10799></code>_)</li> <li>Improve error message when <code>pip config edit</code> is provided an editor that doesn't exist. (<code>[#10812](pypa/pip#10812) <https://github.com/pypa/pip/issues/10812></code>_)</li> <li>Add a user interface for supplying config settings to build backends. (<code>[#11059](pypa/pip#11059) <https://github.com/pypa/pip/issues/11059></code>_)</li> <li>Add support for Powershell autocompletion. (<code>[#9024](pypa/pip#9024) <https://github.com/pypa/pip/issues/9024></code>_)</li> <li>Explains why specified version cannot be retrieved when <em>Requires-Python</em> is not satisfied. (<code>[#9615](pypa/pip#9615) <https://github.com/pypa/pip/issues/9615></code>_)</li> <li>Validate build dependencies when using <code>--no-build-isolation</code>. (<code>[#9794](pypa/pip#9794) <https://github.com/pypa/pip/issues/9794></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Fix conditional checks to prevent <code>pip.exe</code> from trying to modify itself, on Windows. (<code>[#10560](pypa/pip#10560) <https://github.com/pypa/pip/issues/10560></code>_)</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/3c953322c6683b3f0f4d465d9fa361de55358462"><code>3c95332</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/bd54382b59b45975fd4ea00533fb92bd85e3b98d"><code>bd54382</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/c86f9f12594c0e05ed2de31a652bc1eaadb970d1"><code>c86f9f1</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11044">#11044</a> from uranusjr/importlib-metadata-backend-in-3.11</li> <li><a href="https://github.com/pypa/pip/commit/bd9bcef8b3262c513a0dd614731cfe8db8d64125"><code>bd9bcef</code></a> Enable importlib.metadata backend on Python 3.11</li> <li><a href="https://github.com/pypa/pip/commit/cb24fb4052ca8ab8009866b0de61980c81a7e13c"><code>cb24fb4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11085">#11085</a> from pypa/revert-10962-fix-hashes</li> <li><a href="https://github.com/pypa/pip/commit/6ad9a21a43b7fe5d436472de2492069c4541bf06"><code>6ad9a21</code></a> 📰</li> <li><a href="https://github.com/pypa/pip/commit/cf3696a81b341925f82f20cb527e656176987565"><code>cf3696a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11080">#11080</a> from sbidoul/requested-with-constraints</li> <li><a href="https://github.com/pypa/pip/commit/bab5bfce502be78318ab2a3b364b4923d657c854"><code>bab5bfc</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11073">#11073</a> from wimglenn/issue-9330</li> <li><a href="https://github.com/pypa/pip/commit/ae1c2e35e493d7a07cb0d300451625629ae07ce9"><code>ae1c2e3</code></a> Grammar fix in changelog</li> <li><a href="https://github.com/pypa/pip/commit/8d51b8365501132c4f9fe929aa40bd66c9eaa60e"><code>8d51b83</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11087">#11087</a> from mkniewallner/fix-version-changelog</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.0.4...22.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.0.4&new-version=22.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

22.1.2 (2022-05-31) =================== Bug Fixes --------- - Revert <pypa/pip#10979> since it introduced a regression in certain edge cases. - Fix an incorrect assertion in the logging logic, that prevented the upgrade prompt from being presented. 22.1.1 (2022-05-20) =================== Bug Fixes --------- - Properly filter out optional dependencies (i.e. extras) when checking build environment distributions. - Change the build environment dependency checking to be opt-in. - Allow using a pre-release version to satisfy a build requirement. This helps manually populated build environments to more accurately detect build-time requirement conflicts. 22.1 (2022-05-11) ================= Process ------- - Enable the ``importlib.metadata`` metadata implementation by default on Python 3.11 (or later). The environment variable ``_PIP_USE_IMPORTLIB_METADATA`` can still be used to enable the implementation on 3.10 and earlier, or disable it on 3.11 (by setting it to ``0`` or ``false``). Bug Fixes --------- - Revert <pypa/pip#9243> since it introduced a regression in certain edge cases. - Fix missing ``REQUESTED`` metadata when using URL constraints. - ``pip config`` now normalizes names by converting underscores into dashes.

At least, until pypa/pip#9243 is fixed.

…o devel This patch attempts to integrate a forgotten constraints file into the tox setup. The idea is to make PEP 517 builds reproducible. The python-build tool (PEP 517 front-end) parses the build requirements from `pyproject.toml` and `pip install`s them into an ephemeral temporary virtualenv. Unfortunately, this tool does not expose any interface to pin those requirements. But the underlying tool, pip, supports setting CLI options through env vars. So the `--constraint` option corresponds to `PIP_CONSTRAINT` env var which this change relies on. The constraints file can be regenerated as follows: $ python -c 'from pathlib import Path; from sys import argv; from tomli import loads; print("\n".join(loads(Path(argv[1]).read_text())["build-system"].get("requires", [])))' pyproject.toml | python3 -m piptools compile --allow-unsafe --generate-hashes --strip-extras --output-file requirements-build.txt - This change temporarily disables including hashes into the constraints file per pypa/pip#9243. It also sticks to generating the pins under the lowest-supported Python version which is Python 3.6 to address pypa/pip#11321.

Can't install with hashes with pipx 1.5.0, the workaround does not work anymore. upstream issue: pypa/pip#9243

webknjaz changed the title ~~[BUG] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins~~ [regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins Dec 7, 2020

webknjaz added a commit to ansible/pylibssh that referenced this issue Dec 10, 2020

Use -r instead of -c paradigm @ tox build-docs

f3c86fc

Ref: pypa/pip#9243

uranusjr added the C: new resolver label Dec 12, 2020

pradyunsg added C: constraint Dealing with "constraints" (the -c option) resolution: no action When the resolution is to not do anything labels Dec 12, 2020

aawilson mentioned this issue Jan 6, 2021

pip 20.2.4 with 2020 resolver does not use pinned version in the constraints file #9020

Closed

webknjaz added a commit to webknjaz/ansible-galaxy-collection-resolver that referenced this issue Feb 16, 2021

Use -r instead of -c because of pip bug 9243

302ebdd

Ref: pypa/pip#9243

webknjaz added a commit to ansible/pylibssh that referenced this issue Mar 1, 2021

Use -r in make-changelog tox env

0eb5a7f

This is necessary because the most recent versions of pip have a bug with using `-c` having `--require-hashes` or entries with extras. Ref: pypa/pip#9243

mayeut mentioned this issue Jul 4, 2021

pipx install fails when providing constraint file with hashes pypa/pipx#712

Closed

pradyunsg added C: dependency resolution About choosing which dependencies to install and removed C: new resolver labels Oct 12, 2021

uranusjr added state: awaiting PR Feature discussed, PR is needed and removed resolution: no action When the resolution is to not do anything labels Nov 17, 2021

pradyunsg mentioned this issue Nov 26, 2021

Poetry fails in CI/CD with ERROR: In --require-hashes mode, all requirements must have their versions pinned python-poetry/poetry#3472

Closed

3 tasks

graingert mentioned this issue Mar 7, 2022

Removing the legacy resolver #10946

Open

pradyunsg added this to the Drop the legacy resolver milestone Mar 8, 2022

q0w mentioned this issue Mar 14, 2022

Filter available distributions using hash declarations from constraints files #10962

Merged

pradyunsg closed this as completed in #10962 Mar 16, 2022

q0w mentioned this issue Mar 25, 2022

--require-hashes does not correctly handle pinned package with extras #9644

Open

github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2022

pypa unlocked this conversation May 3, 2022

pradyunsg mentioned this issue May 3, 2022

Revert "Filter available distributions using hash declarations from constraints files" #11085

Merged

pradyunsg reopened this May 9, 2022

pradyunsg added the type: enhancement Improvements to functionality label May 9, 2022

abend-arg mentioned this issue Jun 19, 2022

(aws_lambda_python): Poetry exported requirements not readable by pip aws/aws-cdk#14201

Closed

webknjaz added a commit to webknjaz/pylibssh that referenced this issue Jul 28, 2022

Stop pinning hashes in requirements-build.txt

b6e6863

At least, until pypa/pip#9243 is fixed.

sbidoul mentioned this issue Apr 17, 2023

Clarify which dependencies the hash-checking mode applies to #11974

Open

1 task

sleiner mentioned this issue May 24, 2023

Potential bug on install requirements mutation during dependency resolution #11019

Open

1 task

webknjaz mentioned this issue Jan 24, 2024

inlude hash for package when generating requirements.txt jazzband/pip-tools#2045

Closed

mayeut added a commit to pypa/manylinux that referenced this issue Apr 1, 2024

pin pipx<1.5.0

ca16100

Can't install with hashes with pipx 1.5.0, the workaround does not work anymore. upstream issue: pypa/pip#9243

danyeaw mentioned this issue Apr 2, 2024

Use Python3.12 on CI gaphor/gaphor#3233

Merged

8 tasks

mayeut mentioned this issue Aug 2, 2024

Allow omitting version pins from command line when version is given in hashed constraints file #12889

Closed

1 task

This was referenced Oct 29, 2024

generate hashes pypa/gh-action-pypi-publish#280

Closed

Require hashes and wheels for dependencies pypa/gh-action-pypi-publish#248

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins #9243

[regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins #9243

webknjaz commented Dec 7, 2020

webknjaz commented Dec 7, 2020 •

edited

Loading

uranusjr commented Dec 7, 2020

webknjaz commented Dec 10, 2020

aawilson commented Jan 6, 2021

webknjaz commented Jan 7, 2021

pfmoore commented Jan 7, 2021

uranusjr commented Jan 7, 2021

pradyunsg commented Jan 7, 2021

blueyed commented Oct 15, 2021

uranusjr commented Oct 16, 2021

blueyed commented Oct 16, 2021

uranusjr commented Nov 17, 2021

uranusjr commented Nov 17, 2021

q0w commented Mar 9, 2022 •

edited

Loading

pradyunsg commented May 9, 2022

[regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins #9243

[regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins #9243

Comments

webknjaz commented Dec 7, 2020

webknjaz commented Dec 7, 2020 • edited Loading

uranusjr commented Dec 7, 2020

webknjaz commented Dec 10, 2020

aawilson commented Jan 6, 2021

webknjaz commented Jan 7, 2021

pfmoore commented Jan 7, 2021

uranusjr commented Jan 7, 2021

pradyunsg commented Jan 7, 2021

blueyed commented Oct 15, 2021

Additional info / clarifying my use case

uranusjr commented Oct 16, 2021

blueyed commented Oct 16, 2021

uranusjr commented Nov 17, 2021

uranusjr commented Nov 17, 2021

q0w commented Mar 9, 2022 • edited Loading

pradyunsg commented May 9, 2022

webknjaz commented Dec 7, 2020 •

edited

Loading

q0w commented Mar 9, 2022 •

edited

Loading