Rethink pip's PEP-517 mechanism

[xavfernandez]

What's the problem this feature will solve?

Currently, to honor PEP-517 and provide an isolated build environment, pip is calling itself.
Which means, spawning a new process (and its complexities), passing over a bunch of options but forgetting some other (and more).
And spawning processes also means that we have to keep track of them via our RequirementTracker.

In addition to those problems, since everything happens in a new process, the in-memory cache isn't shared and every process will reparse /simple/setuptools index page (or some other backend/dependency).

Describe the solution you'd like

Stop pip from calling itself to populate build environments.

Once pip has read the pyproject.toml file and a PEP-517 build is needed, it could (thanks to the new resolver):

• create a new resolver,
• resolve this new set of dependencies (while using/conforming to the same options provided by the user and taking advantage of the in-memory cache),
• install those resolved dependencies in the build environment (possibly building wheels & invoking a new PEP517 process; yes, this will still require a tracker for build dependencies requiring themselves)
• invoke PEP517 backend in the build environment (like in the "pip calling itself" case)

Additional context

We might want to drop the legacy resolver first (or only support this behavior with the new resolver).

[pradyunsg]

100% in board. I thought there's an issue for this already (if not, well, no worries).

IIRC This was mentioned in one of the chats that @pfmoore, @uranusjr and I had, as a "cleaning up the resolver situation will let us do that" and we all reacted in some form of "oh, yea, that'd be nice".

[uranusjr]

yes, this will still require a tracker for build dependencies requiring themselves)

Does anyone remember what the resolution was in PEP 517 Backend bootstrapping on this? I seems to remember there are some requirements imposed on backend so they can bootstrap themselves with backend-path.

[pradyunsg]

That's covered in https://www.python.org/dev/peps/pep-0517/#in-tree-build-backends.

[pfmoore]

I like this idea too. It would improve performance a lot, fix a load of bugs around passing the right arguments to the subprocess, and probably let us report errors in the setup of the isolated environment better.