18.0.1 files re-released on Jun 27 with new hashes

[parente]

Pipenv started complaining that the hashes for pyzmq 18.0.1 changed upstream on pypi from what we had in a lock file for one of our projects. Sure enough, two of the packages on pypi have today's date unlike the rest of them from March.

https://pypi.org/project/pyzmq/18.0.1/#files

I'm not sure if old packages were overwritten or if these are new flavors that Pipenv suddenly decided to pick up.

Just a small FYI.

[minrk]

Indeed, two manylinux2010 wheels were uploaded by accident when I published 18.0.2. No files have been removed or replaced, though. Can I see your pipenv to see how it matched an existing and available wheel, but decided there was an error because a different wheel also existed? I'm guessing there might be a bug (or possible improvement, at least) in pip or pipenv when validating hashes of packages, if it can download different files even when there is a hash match.

[pbudzon]

We're having the same issue. Package was added to Pipfile.lock some time ago and it has hashes for old files stored, so it refuses to install a file with a hash that doesn't match. I don't think it's a bug, it's intended behaviour in case files are replaced maliciously on the hosted server... Can you replace the packages with their previous versions? I imagine this will break for anyone who doesn't have the old packages cached (or clears their cache).

The contents of Pipfile.lock for this package:

"pyzmq": {
            "hashes": [
                "sha256:1651e52ed91f0736afd6d94ef9f3259b5534ce8beddb054f3d5ca989c4ef7c4f",
                "sha256:5ccb9b3d4cd20c000a9b75689d5add8cd3bce67fcbd0f8ae1b59345247d803af",
                "sha256:5e120c4cd3872e332fb35d255ad5998ebcee32ace4387b1b337416b6b90436c7",
                "sha256:5e2a3707c69a7281a9957f83718815fd74698cba31f6d69f9ed359921f662221",
                "sha256:63d51add9af8d0442dc90f916baf98fdc04e3b0a32afec4bfc83f8d85e72959f",
                "sha256:65c5a0bdc49e20f7d6b03a661f71e2fda7a99c51270cafe71598146d09810d0d",
                "sha256:66828fabe911aa545d919028441a585edb7c9c77969a5fea6722ef6e6ece38ab",
                "sha256:7d79427e82d9dad6e9b47c0b3e7ae5f9d489b1601e3a36ea629bb49501a4daf3",
                "sha256:824ee5d3078c4eae737ffc500fbf32f2b14e6ec89b26b435b7834febd70120cf",
                "sha256:89dc0a83cccec19ff3c62c091e43e66e0183d1e6b4658c16ee4e659518131494",
                "sha256:8b319805f6f7c907b101c864c3ca6cefc9db8ce0791356f180b1b644c7347e4c",
                "sha256:90facfb379ab47f94b19519c1ecc8ec8d10813b69d9c163117944948bdec5d15",
                "sha256:a0a178c7420021fc0730180a914a4b4b3092ce9696ceb8e72d0f60f8ce1655dd",
                "sha256:a7a89591ae315baccb8072f216614b3e59aed7385aef4393a6c741783d6ee9cf",
                "sha256:ba2578f0ae582452c02ed9fac2dc477b08e80ce05d2c0885becf5fff6651ccb0",
                "sha256:c69b0055c55702f5b0b6b354133e8325b9a56dbc80e1be2d240bead253fb9825",
                "sha256:ca434e1858fe222380221ddeb81e86f45522773344c9da63c311d17161df5e06",
                "sha256:d4b8ecfc3d92f114f04d5c40f60a65e5196198b827503341521dda12d8b14939",
                "sha256:d706025c47b09a54f005953ebe206f6d07a22516776faa4f509aaff681cc5468",
                "sha256:d8f27e958f8a2c0c8ffd4d8855c3ce8ac3fa1e105f0491ce31729aa2b3229740",
                "sha256:dbd264298f76b9060ce537008eb989317ca787c857e23cbd1b3ddf89f190a9b1",
                "sha256:e926d66f0df8fdbf03ba20583af0f215e475c667fb033d45fd031c66c63e34c9",
                "sha256:efc3bd48237f973a749f7312f68062f1b4ca5c2032a0673ca3ea8e46aa77187b",
                "sha256:f59bc782228777cbfe04555707a9c56d269c787ed25d6d28ed9d0fbb41cb1ad2",
                "sha256:f8da5322f4ff5f667a0d5a27e871b560c6637153c81e318b35cb012b2a98835c"
            ],
            "version": "==18.0.1"
        },

The file pipenv is trying to install for us:
https://files.pythonhosted.org/packages/ef/8f/dc96dc94efa4716dddd86ea346c2e40ef7a4b0fe9f1219bd3f32e3f1bef7/pyzmq-18.0.1-cp37-cp37m-manylinux1_x86_64.manylinux2010_x86_64.whl
currently has a hash 8af7cb6e788e41ad93a841d95ef1747638377a351f5740d10bd265e74d0774ca which doesn't match any on the list.

[parente]

Our Pipfile looks like the following with packages that have nothing to do with pyzmq ellided:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "==7.3.*"
notebook = "==5.7.*"
# packages here

[packages]
# packages here

[requires]
python_version = "3.7"

The relevant section of our Pipfile.lock generated off the above looks exactly like the one @pbudzon generated.

@minrk I agree it seems like Pipenv should also be tracking the "flavor" of wheel installed in the lock file so that it attempts to compare hashes for that same package, not a different one that appeared after the fact.

[minrk]

what version of pipenv are folks using? This sounds like a pipenv bug (shouldn't it just ignore new files that don't match the provided hashes), but I cannot reproduce it. What version of pipenv do you have?

[pbudzon]

I have pipenv, version 2018.11.26 with Python 3.7.3. Running this on Centos.

[minrk]

Ah, silly me. I was testing on macOS, where the new wheels aren't install candidates. Can reproduce on linux. I'll investigate this a little longer so I can report the bug to pyenv, then I'll pull the new wheels.

[wittenator]

This also broke my Docker builds for my Jupyter deployments. Is this really a pipenv bug though?

[minrk]

Is this really a pipenv bug though?

Not exactly. I tracked it down and it's not a bug in pipenv, but rather in pip.

Rather than doing the logical, safe thing when given a spec of version + hashes (only considering hash matches for install), pip ignores hashes when picking which artifact to install, and then balking if it chooses an artifact that wasn't in the approved list, despite plenty of matches being available. This makes it impossible to use pip + hashes to bless fewer than all artifacts on PyPI, and requires pipfiles to be kept up-to-date if additional artifacts are added, when they ought to be safely ignored, defeating a big part of what I thought was the point of pipfiles, which is freezing a collection of known-good artifacts.

The issue discusses some of the technical complications why this isn't implemented yet (mainly that picking a single artifact to download happens before hashes can be checked), so it will be nontrivial to do the right thing, but it looks like it'll get worked out.

For now, I've pulled the wheels since they don't help anything anyway, yet.

[minrk]

In fact, there's even a fix already in the works: pypa/pip#6464