Add hash-checking support to `install` and `sync` #2945

charliermarsh · 2024-04-09T20:15:42Z

Summary

This PR adds support for hash-checking mode in pip install and pip sync. It's a large change, both in terms of the size of the diff and the modifications in behavior, but it's also one that's hard to merge in pieces (at least, with any test coverage) since it needs to work end-to-end to be useful and testable.

Here are some of the most important highlights:

We store hashes in the cache. Where we previously stored pointers to unzipped wheels in the archives directory, we now store pointers with a set of known hashes. So every pointer to an unzipped wheel also includes its known hashes.
By default, we don't compute any hashes. If the user runs with --require-hashes, and the cache doesn't contain those hashes, we invalidate the cache, redownload the wheel, and compute the hashes as we go. For users that don't run with --require-hashes, there will be no change in performance. For users that do, the only change will be if they don't run with --generate-hashes -- then they may see some repeated work between resolution and installation, if they use pip compile then pip sync.
Many of the distribution types now include a hashes field, like CachedDist and LocalWheel.
Our behavior is similar to pip, in that we enforce hashes when pulling any remote distributions, and when pulling from our own cache. Like pip, though, we don't enforce hashes if a distribution is already installed.
Hash validity is enforced in a few different places:
1. During resolution, we enforce hash validity based on the hashes reported by the registry. If we need to access a source distribution, though, we then enforce hash validity at that point too, prior to running any untrusted code. (This is enforced in the distribution database.)
2. In the install plan, we only add cached distributions that have matching hashes. If a cached distribution is missing any hashes, or the hashes don't match, we don't return them from the install plan.
3. In the downloader, we only return distributions with matching hashes.
4. The final combination of "things we install" are: (1) the wheels from the cache, and (2) the downloaded wheels. So this ensures that we never install any mismatching distributions.
Like pip, if --require-hashes is provided, we require that all distributions are pinned with either == or a direct URL. We also require that all distributions have hashes.

There are a few notable TODOs:

We don't support hash-checking mode for unnamed requirements. These should be somewhat rare, though? Since pip compile never outputs unnamed requirements. I can fix this, it's just some additional work.
We don't automatically enable --require-hashes with a hash exists in the requirements file. We require --require-hashes.

Closes #474.

Test Plan

I'd like to add some tests for registries that report incorrect hashes, but otherwise: cargo test

charliermarsh · 2024-04-09T20:28:39Z

crates/distribution-types/src/prioritized_distribution.rs

            },
            Self::NoBinary => false,
+            Self::MismatchedHash => false,
+            Self::MissingHash => false,


@zanieb - I could use your help with this part. I'm not sure if I did the comparisons correctly here.

I think this is actually wrong, although I'm not sure if it matters. You're supposed to be enforcing an ordering but here you're saying that MissingHash and MismatchedHash are never more compatible than the other value. This means that if we see both MissingHash and MismatchedHash we would arbitrarily display the first one we saw instead of preferring to present one of them to the user.

I decided to change up the strategy in #2949, such that we treat distributions without hashes as compatible (but lower-priority). I can merge that into this PR if you agree with the change.

charliermarsh · 2024-04-09T20:29:23Z

crates/uv-extract/src/hash.rs

+            other => other,
+        }
+    }
+}


@BurntSushi - Do you mind reviewing this component? The basic idea is a wrapper around any reader that can compute a set of hashes as we go. It's then used to wrap the async streams as we unzip.

Aye. I think I have just one concern but otherwise this looks pretty reasonable to me.

charliermarsh · 2024-04-09T21:27:19Z

Open to advice on how best to test that if a registry tells us the wrong hashes, we still validate them at install time. I need a registry or --find-links entry that reports the wrong hashes for a given distribution.

charliermarsh · 2024-04-09T21:47:49Z

A few TODOs:

Add some tests for invalid indexes.
Run a basic benchmark (especially, to ensure that there's no regression for non---require-hashes).

I'd also like to refactor the dataflow from requirements file to RequireHashes, but this isn't required to merge.

charliermarsh · 2024-04-09T22:03:22Z

crates/uv/tests/pip_sync.rs

+    );
+
+    // Third, request the correct hash, that the registry _thinks_ is correct, but without the
+    // cache. We _should_ accept it, but we currently don't.


This is a TODO. We reject distributions without hashes when --require-hashes is provided, whereas in reality we should compute them.

I solved this in #2949.

konstin · 2024-04-10T12:28:17Z

Not exactly this PR (more #2909), but do we have an explanation for .http/.rev files in the docstrings?

konstin

Nice work!

konstin · 2024-04-10T12:03:08Z

Cargo.toml

@@ -93,6 +93,7 @@ indoc = { version = "2.0.4" }
 itertools = { version = "0.12.1" }
 junction = { version = "1.0.0" }
 mailparse = { version = "0.14.0" }
+md-5 = { version = "0.10.6" }


Do we want to support md5? It's cryptographically broken

crates/uv-client/src/cached_client.rs

konstin · 2024-04-10T12:17:12Z

crates/uv-distribution/src/archive.rs

+    /// The path to the archive entry in the wheel's archive bucket.
+    pub path: PathBuf,
+    /// The computed hashes of the archive.
+    pub hashes: Vec<HashDigest>,


nit: we have a constructor and getters

Suggested change

/// The path to the archive entry in the wheel's archive bucket.

pub path: PathBuf,

/// The computed hashes of the archive.

pub hashes: Vec<HashDigest>,

/// The path to the archive entry in the wheel's archive bucket.

path: PathBuf,

/// The computed hashes of the archive.

hashes: Vec<HashDigest>,

konstin · 2024-04-10T12:23:27Z

crates/uv-distribution/src/distribution_database.rs

+                    };
+                    let mut hashers = algorithms.into_iter().map(Hasher::from).collect::<Vec<_>>();
+                    let mut hasher = uv_extract::hash::HashReader::new(file, &mut hashers);
+                    uv_extract::stream::unzip(&mut hasher, temp_dir.path()).await?;


Why don't we spawn blocking here too?

Because it's async whereas the other version is sync. Does that make sense?

crates/uv-distribution/src/error.rs

crates/uv-extract/src/hash.rs

BurntSushi · 2024-04-10T15:09:23Z

crates/uv-extract/src/hash.rs

+
+    /// Exhaust the underlying reader.
+    pub async fn finish(&mut self) -> Result<(), std::io::Error> {
+        while self.read(&mut [0; 8192]).await? > 0 {}


This is potentially putting 8KB on the stack. Maybe not an issue, but it's big enough to stand out to me.

I think I'd probably just use vec![0; 8192] here instead. You could put it in a thread_local! to amortize the alloc if you're concerned about perf.

BurntSushi · 2024-04-10T15:10:10Z

crates/uv-extract/src/hash.rs

+            other => other,
+        }
+    }
+}


Aye. I think I have just one concern but otherwise this looks pretty reasonable to me.

## Summary This lets us remove circular dependencies (in the future, e.g., #2945) that arise from `FlatIndex` needing a bunch of resolver-specific abstractions (like incompatibilities, required hashes, etc.) that aren't necessary to _fetch_ the flat index entries.

Add basic hash More notes Looking at cache...

charliermarsh added the enhancement New feature or improvement to existing functionality label Apr 9, 2024

charliermarsh force-pushed the charlie/check-hashes-iii branch from 4a4838d to 706e330 Compare April 9, 2024 20:15

charliermarsh marked this pull request as ready for review April 9, 2024 20:15

charliermarsh force-pushed the charlie/check-hashes branch from b626db8 to 525ed8a Compare April 9, 2024 20:16

charliermarsh force-pushed the charlie/check-hashes-iii branch from 706e330 to 20f3aeb Compare April 9, 2024 20:16

charliermarsh marked this pull request as draft April 9, 2024 20:16

charliermarsh force-pushed the charlie/check-hashes-iii branch 9 times, most recently from 4fd10e2 to 1d8c657 Compare April 9, 2024 20:27

charliermarsh commented Apr 9, 2024

View reviewed changes

charliermarsh force-pushed the charlie/check-hashes-iii branch 6 times, most recently from 6ee405c to 4d5fbe0 Compare April 9, 2024 21:18

charliermarsh marked this pull request as ready for review April 9, 2024 21:18

charliermarsh force-pushed the charlie/check-hashes-iii branch 3 times, most recently from f3edd39 to 60019b0 Compare April 9, 2024 21:23

charliermarsh requested review from BurntSushi and zanieb April 9, 2024 21:27

charliermarsh requested review from konstin and removed request for BurntSushi April 9, 2024 21:27

charliermarsh force-pushed the charlie/check-hashes-iii branch 3 times, most recently from 7ba9862 to b1e9f80 Compare April 9, 2024 21:41

charliermarsh force-pushed the charlie/check-hashes-iii branch 3 times, most recently from 7f970bf to f66887c Compare April 9, 2024 22:02

charliermarsh commented Apr 9, 2024

View reviewed changes

charliermarsh force-pushed the charlie/check-hashes-iii branch 3 times, most recently from 32a2088 to d8f5d37 Compare April 9, 2024 23:56

konstin approved these changes Apr 10, 2024

View reviewed changes

BurntSushi reviewed Apr 10, 2024

View reviewed changes

Base automatically changed from charlie/check-hashes to main April 10, 2024 18:07

charliermarsh mentioned this pull request Apr 10, 2024

Move FlatIndex into the uv-resolver crate #2972

Merged

charliermarsh added 2 commits April 10, 2024 14:50

Filter out mismatching hashes in version map

33eb13e

Add notes about hash generation

5ee537f

Add basic hash More notes Looking at cache...

charliermarsh force-pushed the charlie/check-hashes-iii branch from d8f5d37 to 669384d Compare April 10, 2024 18:53

Rebase

e3f5242

charliermarsh force-pushed the charlie/check-hashes-iii branch from 669384d to e3f5242 Compare April 10, 2024 18:55

charliermarsh enabled auto-merge (squash) April 10, 2024 19:05

charliermarsh merged commit 1f3b5bb into main Apr 10, 2024
37 checks passed

charliermarsh deleted the charlie/check-hashes-iii branch April 10, 2024 19:09

BrewTestBot mentioned this pull request Apr 15, 2024

uv 0.1.32 Homebrew/homebrew-core#169127

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hash-checking support to `install` and `sync` #2945

Add hash-checking support to `install` and `sync` #2945

charliermarsh commented Apr 9, 2024 •

edited

Loading

charliermarsh Apr 9, 2024

zanieb Apr 9, 2024

charliermarsh Apr 9, 2024

charliermarsh Apr 9, 2024

BurntSushi Apr 10, 2024

charliermarsh Apr 10, 2024

charliermarsh commented Apr 9, 2024

charliermarsh commented Apr 9, 2024 •

edited

Loading

charliermarsh Apr 9, 2024

charliermarsh Apr 9, 2024

konstin commented Apr 10, 2024

konstin left a comment

konstin Apr 10, 2024

konstin Apr 10, 2024

konstin Apr 10, 2024

charliermarsh Apr 10, 2024

konstin Apr 10, 2024

BurntSushi Apr 10, 2024

BurntSushi Apr 10, 2024

Add hash-checking support to install and sync #2945

Add hash-checking support to install and sync #2945

Conversation

charliermarsh commented Apr 9, 2024 • edited Loading

Summary

Test Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charliermarsh commented Apr 9, 2024

charliermarsh commented Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin commented Apr 10, 2024

konstin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add hash-checking support to `install` and `sync` #2945

Add hash-checking support to `install` and `sync` #2945

charliermarsh commented Apr 9, 2024 •

edited

Loading

charliermarsh commented Apr 9, 2024 •

edited

Loading