use aggressive forking when encountering markers

[BurntSushi]

This PR completely rewrites forking in the resolver to follow the
approach to dealing with overlapping markers as described in #4732.

The resulting code is much simpler: forks are no longer constructed
by looking at package names and being limited to disjoint marker
expressions. Instead, we consider all dependencies and iteratively
construct a partitioning of the marker environment universe from the
marker expressions we find on dependencies.

This does generally result in more aggressive forking overall. And
importantly, because of that, puts more pressure on our marker routines
like is_disjoint and is_definitively_empty_set.

A weakness of this approach, particularly in the absence of perfect
marker simplification, is that marker expressions can get even
bigger than they were before. The snapshot updates will show some
potential low hanging fruit here. I decided not to spend time on marker
simplification given Ibraheem's ongoing work in that space.

There are a lot of updates to the snapshot tests. I did my best to
split "uninteresting but possibly unfortunate" updates into one commit.
And then any interesting updates into a commit-by-commit breakdown
where I looked at the specific results to see if it was correct or
not. There is at least one test update that I am unsure of (the one
involving cffi and pre-releases). So please review this commit by
commit.

Closes #4732, Fixes #4640, Fixes #4668

But notably does not seem to address #4959.

[BurntSushi]

Performance does regress with this change in at least one example. I tried @konstin's transformers test:

[project]
name = "i5344-transformers-without-markers"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"

requires-python = ">=3.9.0"
dependencies = [
  "datasets",
  "dill<0.3.5",
  "tensorboard",
  "tensorflow-text<2.16",
  "tensorflow>=2.6,<2.16",
  "torch",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

And a benchmark:

[andrew@duff i5344-transformers-without-markers]$ hyperfine -w10 -p 'rm -rf uv.lock' 'uv-main lock' 'uv-overlapping-markers lock'
Benchmark 1: uv-main lock
  Time (mean ± σ):      48.9 ms ±   3.3 ms    [User: 47.5 ms, System: 51.5 ms]
  Range (min … max):    41.6 ms …  54.9 ms    61 runs

Benchmark 2: uv-overlapping-markers lock
  Time (mean ± σ):     154.1 ms ±   4.7 ms    [User: 156.2 ms, System: 81.4 ms]
  Range (min … max):   145.6 ms … 165.0 ms    19 runs

Summary
  uv-main lock ran
    3.15 ± 0.23 times faster than uv-overlapping-markers lock

Performance on Home Assistant regresses too:

$ git remote -v
origin  [email protected]:home-assistant/core (fetch)
origin  [email protected]:home-assistant/core (push)

$ git rev-parse HEAD
7c5fcec062e1d2cfaa794a169fafa629a70bbc9e

$ hyperfine -w10 -p 'rm -rf uv.lock' 'uv-main lock' 'uv-overlapping-markers lock'
Benchmark 1: uv-main lock
  Time (mean ± σ):     165.2 ms ±   3.3 ms    [User: 136.9 ms, System: 91.9 ms]
  Range (min … max):   157.7 ms … 173.9 ms    17 runs

Benchmark 2: uv-overlapping-markers lock
  Time (mean ± σ):     672.5 ms ±  10.0 ms    [User: 603.8 ms, System: 157.0 ms]
  Range (min … max):   662.5 ms … 690.8 ms    10 runs

Summary
  uv-main lock ran
    4.07 ± 0.10 times faster than uv-overlapping-markers lock

I haven't done any investigation on this yet, but to a certain extent, some regression is expected since we are now more aggressively forking.

My hope is that there are some low hanging fruits here, but it may require better marker simplification/disjointness checks to achieve it. And it is indeed possible that there will be some regression here that is fundamentally tied to an increase in the number of forks.

[A5rocks]

is and () valid?

[konstin]

I'll add a proper review later, but we need to update some packse scenario descriptions with this, such as fork-marker-disjoint

[charliermarsh]

Should the lockfile be included here now that it succeeds?

[BurntSushi]

Yeah. Requires a packse update.

[konstin]

We need to remove this after we have better markers and before shipping to avoid failing in those cases.

[konstin]

We need to update the scenario descriptions.

[konstin]

This is major change from our previous design: We now always fork when we see a marker, not just when we see a package being used multiple times. Imo we should decide about this change independent from overlapping markers. This change makes fork-non-local-fork-marker-direct pass, but it's also responsible for the performance regression.

[konstin]

Haven't we guaranteed with the is_disjoint(&fork.markers, &markers) check that this can never be empty?

[konstin]

Suggested change

	for mut fork in std::mem::take(&mut self.forks) {
	for mut fork in std::mem::take(&mut self.forks) {
	// Invariant: The union of the markers of all forks is always the universe.
	// Let
	// m_f := markers of the fork
	// m_d := the remaining markers of the dependency
	// If `m_f ∩ m_d = ∅`, i.e. they are disjoint, we know that the dependency does not
	// occur in this fork, and we keep the fork as it was. Otherwise, `m_d` either
	// intersects with or covers `m_f`.
	// Should `m_f \ m_d` be empty, `m_d` covers `m_f`, and we add `m_f` verbatim and set
	// `m_d := m_d \ m_f`-
	// Otherwise, we split this fork into two new forks, `m_f \ m_d` and `m_f ∩ m_d`. This
	// is relevant if we have e.g. `python_version >= "3.9"` and `python_version >= "3.10"`,
	// so that we get forks `<3.9`, `>=3.9,<3.10` and `>=3.10`. It is a sound operation
	// since `(m_f \ m_d) ∪ (m_f ∩ m_d) = m_f` and since both new forks are either subsets
	// of `m_f` and `m_d`, meaning we miss no overlap.

[BurntSushi]

I opened a new PR with this same change rebased on latest main (not necessarily intended to merge): #6143.