Support redirection to the system installer for system packages

[ncoghlan]

Virtual environments and --user installations are entirely pip's domain. The same is also true for a custom install of Python. However, for the system Python (provided by the OS vendor), it is desirable for pip to be able to redirect an install request to the system package manager, and only proceed with a direct install from PyPI if no system package is available. Installing arbitrary packages from PyPI can have unpredictable effects on system stability, especially on operating systems like RHEL which deliberately avoid backwards incompatible upgrades to system packages.

What I am thinking here is to rely on the fact that OS vendors are able to patch their system pip install, so rather than a complex plugin system, just have a snippet somewhere in pip like the following:

def install_platform_dists(*installspecs):
    # Linux distro vendors are expected to patch this appropriately
    # for their own packaging policies
    # The passed in dist specifiers are installation specifiers of the form
    # described in PEP 426
    # It should return an iterable of all Python dists made available as a result of
    # handling the request
    return ()

(Note: to be clear, this isn't a prerequisite for bundling pip with Python 3.4 - it's just something I think we should be looking at long term to reduce the scope for conflict between pip and system package management tools. If pip transparently redirects to yum or apt-get when appropriate, then Pythonistas can use the pip commands regardless of distro, but automatically get yum and apt packages when they meet the constraints)

[dstufft]

On the surface this proposal sounds like it might be a good idea but I think it's a case of getting entirely too clever for our own good. I think it's not entirely uncommon for, at least in the web developer world, to not use the system package tools to install libraries at all. I would be very unhappy if I got locked out of installing something system wide because apt-get or yum decided I shouldn't.

There's also the problem of versioning. Linux Distributions often patch things to make them integrate better with the environment. However that isn't always done in a way that doesn't introduce subtle bugs. I feel very strongly that if I ask for pip install Django==1.5.1 I should get the same bits no matter what machine I'm on (modulo different wheels for different platforms). If I got a slightly patches one from the Linux Distribution and that caused a bug in my software I'd be pretty upset, especially when I likely couldn't reproduce it locally on my Mac.

There's also a problem with people upgrading pip itself with pip. They upgrade beyond something that the distro has packaged, so pip dutifully gets yum to install that package (can it uninstall it? What else will it uninstall?) and then installs the new pip. This new pip hasn't been patched by the vendors and all hell breaks loose as pip in places upgrades or removes things that apt/yum etc believe to be installed.

I do believe that distros should not be installing into site-packages (but pip should be) and should instead be installing into a dist-packages or something similar. However that still doesn't solve the problem of what to do with conflicts between dist-packages and site-packages. Maybe pip can warn (and require confirmation) before touching files that were installed by the OS system? I think the OS's would need to gain the same sort of ability as well though and warn if they are going to disrupt packages installed by pip.

I do however, understand where you're coming from here and what you're trying to achieve. I think reducing the scope of conflict here is a good goal. I'm not quite sure what the answer is here.

[dstufft]

That's not to say I wouldn't be ok with something like this either but it'd need a really solid proposal that handled all the dark corners in differences between OS packages and pip installed packages.

[ncoghlan]

Agreed, this is a "conversation starter", rather than a solution (or even something admitting to an easy solution). However, it's an explicit goal of mine that the idea of using pip to install things system wide will eventually become anathema - pip installs should either be for a specific user, or for a specific virtual env, with the system Python left entirely to run system utilities and ad hoc user scripts. Installing to dist-packages does nothing to solve the problem, because the version in site-packages can end up shadowing the version in dist-packages and causing all the same stability problems it would if it overwrote the system version (only now without any warnings of the conflict from the system package manager).

In the long run, the path from PyPI to the system Python should run through the system package manager wherever possible (which is why one of the goals of PEP 426 is to provide sufficiently rich metadata to make it feasible to automate the step of creating a policy compliant system package for most PyPI packages).

However, I also agree that any such mechanism should be optional (whether opt-in or opt-out), and there are other pieces we need to get in place (e.g. to support package sharing between virtual environments in a cross platform manner) before the idea of not using pip to directly manipulate the system Python package set becomes reasonable.

There's another tracker issue asking for the package listing to distinguish between pip installed and OS installed files. So maybe that's a better place to start.

[dstufft]

I think distinguishing between them is a really good idea. I do think pip should at least provide some warning if it's going to be mucking about with something installed by the OS tooling as that tends to lead towards very broken things. So at the very least we should get that type of warning in place.

I actually think that most OS tooling will yell and refuse to install if something is already there (is that right?). So maybe keeping them both installing into site-packages is the best path. Pip can determine (via that other issue) if a preexisting package was installed by the OS and not go fiddling with it otherwise (without some sort of a no really do this flag) and apt/yum/whatever will already blow up and refuse to install say Django overtop of a pip installed Django.

Maybe? Hard problems!

[ncoghlan]

yum complains if there's a direct file conflict, but unless you've seen the problem before, good luck figuring it out from the error message :)

It gets even more confusing if the conflict is module vs package or if it is a Debian/Ubuntu style case of incompatible shadowing the system package with an incompatible upgrade.

So making this issue depend on #949 is probably a good idea - identifying the origins of installed packages would be a useful first step towards saying "that was installed by the system package manager, better to use that to upgrade it".

[dstufft]

I'm going to close this, I think the proposal in this issue is ultimately a bad idea, and that pushing people away from using pip to install globally by defaulting to --user is a better idea (#1668) as well as maybe some other enhancements we can figure out later! Either way I don't think we're likely to forget that pip and the OS packages aren't exactly best friends and there's nothing actionable on this ticket.