Review and fix pruning of installed packages.

[jenstroeger]

The prune goal in our Makefile is intended to remove those installed packages that are not direct or indirect dependencies of the repo package itself. Because pip doesn’t provide a “prune” or related command itself, we prune the installed packages manually.

The assumption of our custom pruning is that all required packages are already installed (and then some, those which will be pruned) and at no time during pruning would we need to reach out to an external package repository to download additional dependent packages — not even to update installed packages.

However, that assumption seems to be false: in this run the

python -m pip wheel --wheel-dir build/wheelhouse/ --requirement build/prune-requirements.txt

command downloads an update to pip although that exact same version was already installed. In other instances, it’s not the pip package that’s being updated and, worse, such an update during pip wheel can cause a significant slowdown because all possible packages are being downloaded to resolve and already resolved dependency.

I think we need to consider the following:

• We currently build the wheels for all installed packages such that we can re-install only required packages later during pruning without pulling from an external PyPI server. Is there another way to create a local package repo of source packages?
• Why does pip wheel need to resolve a dependency using external packages, if the dependency has already been resolved and all dependent packages are installed?
• Could we use a tool like pipdeptree to create a set of packages that need to be pruned away? Ironically, though, that would require a new, pruneable dependency.

I raised a question Trying to understand the details of pip wheel, hopefully the community can shed a little more light on the details here.