Support PEX files as resolve repositories.

[jsirois]

This would allow using --pex-repository instead of --index / --find-links to attempt to resolve requirements from an already built PEX. This feature would support a Pants use case in pantsbuild/pants#11105 and the underlying implementation is also needed to support #1020 correctly. In that case, each interpreter found on a given host could be tried for compatibility with a PEX by attempting to resolve all its requirements from itself using that interpreter.

[jsirois]

A few notes on the resolve logic implementation:

• At buildtime to implement --pex-repository:
  1. Implementing custom logic to compare requirements to .deps/ wheel tags to interpreters and platforms recursing on wheel Requires-Dist metadata requirements.
  2. Leveraging pip:
     This requires the "wheels" we embed in .deps/ - which are not actually wheels but wheels installed in individual chroots - to be uninstalled wheels. This in turn requires the ability for the PEX .bootstrap/ runtime to contain wheel installing code. Currently it does not since this code lives in pip and we do not vendor Pip into the runtime .bootstrap/.
• At runtime to implement appropriate interpreter selection:
  Again, since the PEX .bootstrap/ does not currently contain Pip, which is large, we're pushed towards custom logic as sketched above.

Taking Pip as being too large to vendor in the PEX runtime .bootstrap/ as a given, we'll need to implement the custom resolve logic. With that in place we can support --pex-repository, but only when used as the exclusive source of dists. That covers the Pants use case.

If we did leverage Pip instead to do resolves / convert .deps/ to contain wheels instead of installed wheel chroots, we could support a mix of --pex-respository, --index and --find-links at build-time since, in that case, PEX files would contain wheels which could be exposed to pip directly via --find-links.

[Eric-Arellano]

Taking Pip as being too large to vendor in the PEX runtime .bootstrap/ as a given

The concern is bundle size, right? Vendoring pip at build time removed a lot of complexity from Pex, and I suspect that vendoring here would avoid adding lots of complexity/edge cases to runtime.

[jsirois]

The concern is bundle size, right?

Right. For example the Pex PEX itself would go from 2,683,442 bytes to 4,126,954 bytes.

Vendoring pip at build time removed a lot of complexity from Pex,

That is definitely not the case:

jsirois@gill ~/dev/pantsbuild/pex ((v1.6.12)) $ find pex -type f -name "*.py" ! -wholename "*/vendor/*" | xargs wc -l
   302 pex/package.py
    52 pex/sorter.py
   157 pex/platforms.py
    41 pex/iterator.py
   333 pex/pep425tags.py
    75 pex/fetcher.py
   308 pex/finders.py
   110 pex/requirements.py
    53 pex/archiver.py
   153 pex/installer.py
   240 pex/resolver_options.py
   181 pex/crawler.py
   307 pex/resolvable.py
   292 pex/http.py
   167 pex/translator.py
   586 pex/resolver.py
    93 pex/glibc.py
   135 pex/link.py
   ...
 (3585)
  9507 total

Vs:

jsirois@gill ~/dev/pantsbuild/pex (master) $ find pex -type f -name "*.py" ! -wholename "*/vendor/*" | xargs wc -l
    34 pex/dist_metadata.py
   124 pex/platforms.py
   370 pex/jobs.py
    58 pex/finders.py
    71 pex/requirements.py
   403 pex/pip.py
    55 pex/network_configuration.py
    84 pex/distribution_target.py
  1209 pex/resolver.py
  ...
 (2408)
 10231 total

What it did do though is make PEX resolution identical to the resolution people expected since Pip was and surely still is the dominant resolver / de-facto standard.

and I suspect that vendoring here would avoid adding lots of complexity/edge cases to runtime.

This could be true. Keep in mind, all the difficulty of a real resolve is completed by Pip and hidden behind the distributions inside the --pex-repository. No version needs to be chosen when resolving from a --pex-repository, only compatible wheels need be selected (matching tags). Assuming though vendoring Pip in the runtime .bootstrap/ saves us 10 bugs going forward, it will also undoubtedly be slower at performing the runtime resolve and this will be overhead on every run of a PEX. The Pip speed can be simulated with a pure --find-links resolve so I think I'd like to proceed with the custom logic far enough to time it and then compare to the Pip --find-links resolve.

[stuhood]

At buildtime to implement --pex-repository:

1. Implementing custom logic to compare requirements to `.deps/` wheel tags to interpreters and platforms recursing on wheel `Requires-Dist` metadata requirements.

2. Leveraging pip:
   This requires the "wheels" we embed in `.deps/` - which are not actually wheels but wheels installed in individual chroots - to be uninstalled wheels. This in turn requires the ability for the PEX `.bootstrap/` runtime to contain wheel installing code. Currently it does not since this code lives in pip and we do not vendor Pip into the runtime `.bootstrap/`.

@jsirois : My primary question to help decide between these is: "how quickly can pip resolve from a directory/collection of wheels like this?": if it's pretty quick (maybe, within 2x of a custom solution), then it seems like that would be the way to go, because there is potentially more logic involved in the "graph walk" than just literal lookups in a hashmap, and duplicating that logic in PEX to avoid talking to pip would probably be worth avoiding unless the performance difference is significant.

[jsirois]

@stuhood your conclusion sounds exactly equal to mine above. To do a time comparison the cheapest way is to implement the custom logic 1st since the pip side of the timing can just be emulated via a pip download command. To do the inverse is a much bigger change since getting pip to resolve from a PEX file requires converting PEX files to contain wheels (today they contain pre-installed wheels), and then installing them at runtime.