Use pip for resolving and building distributions.

[jsirois]

This changes Pex to vendor pip and defer all resolution and building to it.

As part of this change some ambiguous and / or broken features were changed and APIs adjusted:

• The egg distribution format is no longer supported.
• The deprecated --interpreter-cache-dir CLI option was removed.
• The --cache-ttl CLI option and cache_ttl resolver API argument were removed.
• The resolver API replaced fetchers with a list of indexes and a list of find_links repos.
• The resolver API removed (http) context which is now automatically handled.
• The resolver API removed precedence which is now pip default precedence - wheels when available and not ruled out via the --no-wheel CLI option or use_wheel=False API argument.
• The --platform CLI option and platform resolver API argument now must be full platform strings that include platform, implementation, version and abi; e.g.: --platform=macosx-10.13-x86_64-cp-36-m.
• The --manylinux CLI option and use_manylinux resolver API argument were removed. Instead, to resolve manylinux wheels for a foreign platform, specify the manylinux platform to use with an explicit --platform CLI flag or platform resolver API argument; e.g.: --platform=manylinux2010-x86_64-cp-36-m.

Fixes #781

Additionally:
Fixes #771
Fixes #763
Fixes #761
Fixes #735
Fixes #694
Fixes #660
Fixes #658
Fixes #642
Fixes #641
Fixes #628
Fixes #620
Fixes #614
Fixes #611
Fixes #608
Fixes #439
Fixes #415
Fixes #387
Fixes #315

[jsirois]

Reviewers - thanks in advance for taking a look at this large change. Commits are reviewable independently. Of these the 2nd from top - ffd5d75 - was automatically generated via tox -e vendor and can be ignored. Similarly, the 3rd from the top - 6bb0ebe - was mainly a copy over of files from pip with modifications all noted in comments as explained in the headers of the two updated files.

The last two commits add on feature kills proposed on #781 and since signed-off by interested parties fwict on Slack and in that issue itself.

There is ongoing discussion about breaking pex up into components including a resolve component, but I think that discussion is mainly orthoganal to this change. The resolve interface is tweaked in this change, but in a way that is much more neutral; ie less pex types like Fetcher, etc and more generic resolve configuration primitives that should be easily translateable to any given resolver should we move to add plug-in resolver support.

[jsirois]

NB: Fixed in commit caaf93d.

[jsirois]

NB: The old cross-product loop was very broken. When a platform is specified, the resolve is foreign and we should definitley not try to use one or more local interpreters to resolve the foreign platform wheels. One bit of brokenness: In the past for a resolve with a local python2.7 interpreter and python3.6 interpreter across (inexact) platforms of linux-x86_64 and macosx-10.13-x86_64 we'd accept any abi for the foreign platform per distribution which was broken in general and only worked when the indexes and find_links passed to resolve were carefully controlled to have uniform wheel platforms.

[Eric-Arellano]

Epic.

I hadn't pitched in on #781, but agree that this is a big win for the project. Beyond all the fixes this brings, the change has an added benefit of lowering the barrier to entry for new contributors.

--

Because this is a disruptive change, it would help to document the end-user API changes in the PR or some other documentation. I'm envisioning something like:

Removes these flags:

* --cache-ttl, Pex no longer does its resolves
* --manylinux, this was never safe. Instead, explicitly specify the platform.

Will this PR coincide with Pex 2.0?

[Eric-Arellano]

🎉

[kwlzn]

pretty cool John, thanks for doing this. I'd totally forgotten about the longstanding pex caching bug(s) this will also solve, so that's exciting as well.

I took a cursory look and kicked tires and things look good at a high level to me - but I'll try to carve out some more time to dig deeper if time permits. it would be good to have someone from the Build team vet this e2e with pants as well - as I'm long removed from the authoritative chain there.

btw, this may be significant enough to finally cut a 2.0.0?

[jsirois]

Will this PR coincide with Pex 2.0?

Yes.

btw, this may be significant enough to finally cut a 2.0.0?

And yes.

I'll be adding a few sentences in the release notes, but also added the major changes to the PR description.

[Eric-Arellano]

PR description looks great. Very informative. Thank you for adding that!

[jsirois]

What is [security] in this example, if it isn’t key? Is there any way to indicate that? If not, certainly let’s reserve that for a followup PR so that we can land this.

It's an "extra". It's a standard part of the requirement spec but irrelevant to VendorSpec; and I added it exactly for this reason - to emphasize the portions of a requirement the VendorSpec requires and cares about. The assumption here is that if any meaningful maintenance is to be done on this code, the maintainer will have to know about requirements as a pre-req. This is not dissimilar to my thought process on documenting things like python -s in a codebase where python environment isolation is a key thrust. I think where we may differ in approach here is we both refrain from explaining python language and stdlib features - those can be assumed to be understood by a maintainer. I tend to also include other bits of tightly coupled domain knowledge. Perhaps I should not.

[jsirois]

Thanks folks, I'm going to merge this and prepare a 2.0.0 release.