Understanding pex lock sync

[benjyw]

I'm trying to fully understand the behavior of lock sync, I appear to be missing something fundamental after reading the help messages, the PR description in #2373, and parts of the code.

For specificity, with the following inputs (which happen to be Pants's current requirements, plus cowsay):

$ python -m pex.cli lock sync  --lock lockfile.json --style strict --pip-version latest --force-pep517 --indent 2 --pip-log pip.log "ansicolors==1.1.8" "chevron==0.14.0" "fasteners==0.16.3" "freezegun==1.2.1" "ijson==3.2.3" "libcst==1.4.0" "packaging==21.3" "pex==2.19.1" "psutil==5.9.8" "pytest>=6.2.4,<7.1.0" "python-lsp-jsonrpc==1.0.0" "PyYAML>=6.0,<7.0" "requests[security]>=2.28.1" "setproctitle==1.3.2" "setuptools>=63.1.0,<64.0" "toml==0.10.2" "types-freezegun==1.1.6" "types-PyYAML==6.0.3" "types-requests==2.28.1" "types-setuptools==62.6.1" "types-toml==0.10.8" "typing-extensions~=4.12" "mypy-typing-asserts==0.1.1" "node-semver==0.9.0" "urllib3<2" "debugpy==1.6.0" "pydevd-pycharm==203.5419.8" "fastapi==0.78.0" "starlette==0.19.1" "strawberry-graphql[fastapi]==0.114.0" "uvicorn[standard]==0.17.6" "beautifulsoup4==4.11.1" "python-gnupg==0.4.9" "PyGithub==2.4.0" "cowsay==6.1"

(In my real case I also set --complete-platforms to one specific local interpreter, which I've elided for brevity).

The initial run, when there is no lockfile.json (but with download cache primed), takes ~15 seconds on my machine (a macbook pro), which is short enough to not be annoying, but long enough to make it noticeable when more work than I expect is taking place. I am also verifying the "amount of work" by looking at the pip.log.

So:

• I run once, it takes 15 seconds, and creates lockfile.json.

• I run a second time with the exact same command line. It takes 15 seconds again, whereas I would have expected it to be more or less a no-op. But then I notice this comment so I figure that when there are no diffs at all pex refreshes the entire lockfile against the current state of PyPI.

• I remove cowsay from the list of requirements and rerun:

$ python -m pex.cli lock sync ...
Updates for lock generated by cp311-cp311-macosx_14_0_arm64:
  Deleted cowsay 6.1
Updates to lock input requirements:
  Deleted 'cowsay==6.1'

• The result is correct, but this takes 15 seconds again, and invokes pip (and the pip log shows that pip is fetching cowsay links and metadata). This I find odd because IIUC there is enough information in the lockfile to do this entirely locally without invoking pip. And even if pip is invoked, why is cowsay one of its inputs?

• I rerun without cowsay, and again, what seems like it should be a near no-op, is not.

And so on - what seem like small incremental updates or deletions appear to trigger basically a full resolve every time.

Note that if I set artifacts_can_change=False here then I get fast no-ops and deletions, which is more what I would expect from reading sync's help message. But I don't understand the full implications of that. Additions of new requirements still seem to take longer than I might expect, even with this edit, but I'd need to do more experiments to be sure.

Side note:

I have been removing pip.log between runs, but if I don't I can trigger an odd error. After removing cowsay rerunning and then rerunning again, I get

$ python -m pex.cli lock sync ...
Encountered 1 error updating lockfile.json:
1.) cp311-cp311-macosx_14_0_arm64: No distribution metadata found for cowsay==6.1.

And this error persists until I delete the pip.log file. So I suspect this is because when using a fixed pip.log, pex appends to it, and the pip log parser attempts to read old lines pertaining to cowsay which now don't match any input in the lockfile? Happy to file a bug for this one.

[jsirois]

I run a second time with the exact same command line. It takes 15 seconds again

and

And so on - what seem like small incremental updates or deletions appear to trigger basically a full resolve every time.

Exactly, That is "expected"! Every lock subcommand (create, update, sync) currently performs a complete fresh lock from scratch. The only difference is that update and sync apply constraints based on a prior lock. The pex3 lock {update,sync} commands get you incremental lock updates. They have 0 to do with speed of locking.

For speed, see #2044, but that bug was filed at least 6 months after discussion about this in Pants forums. It has been "well-known" for a long time now that the only way to speed up lock resolves with Pip is to use the trick detailed in that issue.

And know you're somehow not alone in this confusion: pantsbuild/pants#21223 (comment)

[benjyw]

Got it, so definitely my misunderstanding was from not being aware of the issues you linked to, and making a wrong assumption about what is going on under the hood. Will file the bug separately.

[jsirois]

Another way to think about it does not involve Pex at all. How would a Pants-compatible-tool use Pip to perform an incremental resolve? Normal Pip usage has the state of the venv it is housed in, but in Pants you can't really save state - at least not the state of a full venv! That was tried probably in 2020 IIRC and it proved too slow. Restoring the 10ks of files in a venv was incurring >500ms of overhead per venv restore. In that 2020 trial this overhead was not for locks - which weren't in the offing yet - but for creating per-test requirement PEXes. Pex has always been a stateless tool - it only has an append-only ~immutable cache, no mutable state. The new thing here will be simulating mutable state by creating a venv from the lock, then using that temporary mutable venv to incrementally update the lock.

[benjyw]

I guess the part that was missing in my brain was how pip is able to do fast incremental updates to add new requirements in a venv. Sounds like it relies on a lot of mutable state in the venv, but I'm still not sure what that state includes that isn't in a lockfile.

Anyway, I guess performance improvements need to come from pip implementing the tricks that make uv fast.

[jsirois]

Sounds like it relies on a lot of mutable state in the venv, but I'm still not sure what that state includes that isn't in a lockfile.

Pip doesn't know about the lockfile, right? It is just handed a set of input requirements by Pex and so all the Requires-Python and Requires-Dist info in the lock is lost.

Anyway, I guess performance improvements need to come from pip implementing the tricks that make uv fast.

They ~have! See that same Pants link for part 2 of my reply to @sureshjoshi. The current implementation of using range requests to just grab wheel metadata is flawed; so some fixes in-progress to that still need to land.

That said, the premise is a bit off too, there is an intermediate step Pex can take to improve speed even if Pip doesn't; that's #2044.

[jsirois]

In fact, Pip did range requests (experimentally - just like today - you must flag on) before uv existed. It's just never done them well.

[jsirois]

Although you threw me with an apparent lack of understanding that Pex always uses a full Pip resolve, your delete observation is correct. For that 1 case, an update with nothing but deletes, Pex could special case things and not call Pip at all: #2566

[benjyw]

Bug report: #2568

[jsirois]

Thanks! I have a fix PR out for that and the pure delete perf case. I'll get out a 2.20.4 release with those bug / perf fixes later this evening assuming you have time to review those fixes.

[benjyw]

Done! Thanks for the quick turnaround.

[jsirois]

Alright, that took longer than I'd hoped. This is all out now in 2.20.4: https://github.com/pex-tool/pex/releases/tag/v2.20.4

@benjyw are all your questions answered at this point or is there more to hash out?

[benjyw]

All answered, thanks!