pip 10 no longer uninstalls distutils packages

[ianw]

I know, I know, we're back having the same discussion ...

fabb739 removed disutils package uninstallation. It was marked with a pip10 deprecation error so this was not unreasonable.

This was first done in 8.0 but reverted; see issue #3389. We can reminisce about what broke and the discussions we had at the time in [1] & [2] and further discussion in Issue #3384.

The problems discussed there are still essentially the same :/ I guess it comes down to -- since the last time it was removed, has the packaging landscape for major distros like Debuntu, CentOS and Fedora changed sufficiently that users can be expected to sufficiently work around this?

I'd probably have to suggest the answer is no. AFAICT the external issues discussed in Issue #3384 just haven't been resolved. People who are over-installing system deb/rpm python packages without the right egg info are going to hit this at the 10.0 release and I imagine start filing the same issue again (it already happened in openstack with [3]).

---

One point raised was [4]

I think Fedora is planning (or is) on using pip to install things so they would, Debuntu side they do not seem to interested in it

Has this changed? Are we at a point that from Fedora XYZ/Ubuntu AA etc we can confidently say that packages come with correct egg-info so that pip can uninstall them?

For mine, the best course would be to determine this, revert but with a deprecation time-frame that's fact based on when users can reasonably be expected to not require it (p.s. I proposed the original revert, and I think "10" was just an arbitrary point that seemed far away, rather than something considered carefully)

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-20.log.html#t2016-01-20T01:18:19
[2] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-20.log.html#t2016-01-20T05:19:24
[3] http://lists.openstack.org/pipermail/openstack-dev/2017-October/123836.html
[4] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-20.log.html#t2016-01-20T05:25:11

[ianw]

@dstufft provided excellent context on this (again) today

So this is a distutils induced problem. When you remove a distutils based install, all you remove is the .egg file. A pretty good example is

apt-get update && apt-get install python-pip python-yaml && python -m pip install -U pip && python -c "import yaml" && python -m pip uninstall pyyaml -y && python -c "import yaml"

Where you can see that removing the pyyaml (provided by debian's python-yaml) didn't actually remove the library. This is because, as you can see here pyyaml is distutils based, and the package is made by running "python setup.py install" here. The file PyYAML-3.12-py2.7.egg-info, say, actually has no info in it that the files reside in the yaml directory, so pip can't remove it.

The other thing is, if you use pip to install a pure distutils based package, pip will actually override it to use setuptools behind the scenes, and write out top_level.txt etc so i can make a stab at removal. So the problem is isolated to distutils only packages that are being installed via "setup.py install"

I think actually a reasonable representation of this problem is finding .egg-info FILES on a regular system

$ find /usr/lib64/python2.7/site-packages/ -type f -name '*.egg-info'
/usr/lib64/python2.7/site-packages/pwquality-1.3.0-py2.7.egg-info
/usr/lib64/python2.7/site-packages/mercurial-4.2.3-py2.7.egg-info
/usr/lib64/python2.7/site-packages/rpm_python-4.13.0.1-py2.7.egg-info
/usr/lib64/python2.7/site-packages/gpg-1.8.0-py2.7.egg-info
/usr/lib64/python2.7/site-packages/python_nss-1.0.1-py2.7.egg-info
/usr/lib64/python2.7/site-packages/pycrypto-2.6.1-py2.7.egg-info
/usr/lib64/python2.7/site-packages/pyparted-3.10.7-py2.7.egg-info
/usr/lib64/python2.7/site-packages/systemd_python-234-py2.7.egg-info
/usr/lib64/python2.7/site-packages/pycurl-7.43.0-py2.7.egg-info
/usr/lib64/python2.7/site-packages/bzr-2.7.0-py2.7.egg-info
/usr/lib64/python2.7/site-packages/pycups-1.9.72-py2.7.egg-info
/usr/lib64/python2.7/site-packages/PyYAML-3.12-py2.7.egg-info
/usr/lib64/python2.7/site-packages/async-0.6.1-py2.7.egg-info
/usr/lib64/python2.7/site-packages/pygpgme-0.3-py2.7.egg-info
/usr/lib64/python2.7/site-packages/yum_metadata_parser-1.1.4-py2.7.egg-info
/usr/lib64/python2.7/site-packages/pygobject-3.24.1-py2.7-linux-x86_64.egg-info
/usr/lib64/python2.7/site-packages/python_dmidecode-3.12.2-py2.7.egg-info

So the problem is largely isolated to a subset of packages ... but enough to cause a bit of pain and more than you could reasonably say "just convert them to setuptools".

That said, also maybe these are less likely to be updated? maybe pyyaml is a bit of an outlier

Firstly, distutils [1] itself says

Most Python users will not want to use this module directly, but instead use the cross-version tools maintained by the Python Packaging Authority

so I don't have stats, but you'd think that new code should be avoiding this?

But, there is no "death" of distutils planned. PEP518 [2] ("Specifying Minimum Build System Requirements for Python Projects") is an approach to satisfying chicken-egg issue some packages have when your installer depends on things which might help transitions.

I'm not arguing that overwriting these packages is the right thing to do. It overwrites files managed by the package manager and we know that. But ... it has been working for a long time, and just switching it off is a big pain point. People are warned that "you get what you get and don't get upset" with it. It's not really terribly invasive, just a couple of lines. Maybe, while distutils lives, this can too, behind the big warnings it has.

[1] https://docs.python.org/3/library/distutils.html#module-distutils
[2] https://www.python.org/dev/peps/pep-0518

[ghost]

We're working on removing distutils from Cpython but it's going to take some time.

[fungi]

As it looks like we're getting ever closer to a pip 10 release and this situation still seems to be the same, is there any possibility for an escape hatch option to allow pip to continue installing even if it's unable to guarantee proper uninstallation of the distro-provided versions of the same packages? While I'm no fan of sudo pip install ... and would prefer to see pip always install into isolation from the system context, I help maintain a lot of systems which still rely on that unfortunate pattern where I expect we'll be stuck figuring out a means of pinning pip<10 for a long time otherwise.

[pfmoore]

@fungi If someone provides a PR, we can review it. But I'm not aware of anyone working on such a PR.

[fungi]

Thanks, I meant the question to read as, "if one of us were to implement and propose an option... would it be too late now?" I think many have been holding out hope that the break-when-you-encounter-distutils feature would be reverted again before pip 10 releases, but at this point it's looking less and less likely so we probably need to propose a workaround instead.

[dstufft]

I think many have been holding out hope that the break-when-you-encounter-distutils feature would be reverted again before pip 10 releases, but at this point it's looking less and less likely so we probably need to propose a workaround instead.

I think I've said it before, but just to be sure, my opinion on this is basically I'm not against reverting, if reverting comes with a plan and a timeline for how we're going to make sure we can actually land that change in the future. This isn't the first time we tried to land this change (so that we can stop lying to users and pretending we're uninstalling something we are't) and at the time we reverted without a plan other than "well we'll wait some more time and hopefully more people will listen to the warnings we're outputting between now and then".

It's now later, and quite obviously by the fact folks are asking for it, people didn't listen to the warnings and get their systems fixed so that packages are installed with the appropriate metadata (or they're using something isolated from places where incorrect metadata has been installed). That tells me that it's unlikely going to be fixed by kicking the can further down the road and just saying that waiting some more time without any other action is going to allow us to merge this in the future.

Basically, I think landing this is an important change for pushing the ecosystem forward and I am loathe to revert it without a solid plan to make us able to land it in N releases without simply having a repeat of "this breaks, can we revert?". If we can't come up with a solid plan, then I think that breaking things for people is the best way we have to drag the ecosystem forward (and is something we've done previously for the same reason).

So No, I don't think it's too late, but any proposal would need to include the above plan for finally being able to remove this.

[pfmoore]

My point about a PR was intended to be similar - I don't want to just revert this without any way forward, so if someone offers a solution that moves forward (which in my mind meant a PR, but could be something along the lines @dstufft suggests) then we can look at that.

To be precise:

1. We've proved that a deprecation error doesn't work (we've tried that).
2. You're claiming that just getting on with it and making the change doesn't work.
3. Never making this change isn't an option (it leaves pip lying about what it did).

So what other option are you suggesting? The default as things stand will be to try (2) - the error will go into pip 10, and we see if people can survive (if they can't, we always have the option of a pip 10.1 reverting this change, but we'd still need someone to offer an alternative.

has the packaging landscape for major distros like Debuntu, CentOS and Fedora changed sufficiently that users can be expected to sufficiently work around this?
I'd probably have to suggest the answer is no.

As a pip maintainer, the question I'd have to ask here is how much pressure have users put on the major distros for them to solve this issue? Is it also an issue in paid distributions like Red Hat, and if so, have their users (who are paying RH for support) lobbied Red Hat to develop or fund a solution?

[fungi]

I figured not reverting but simply adding a --sorry-yes-i-know sort of flag to just ignore the inability to uninstall and let the chips fall where they may would be preferable to an indefinite pip<10 pin all over the place where people are sadly mixing sudo pip install ... and latest pip with distro-packaged Python libraries on longer-lived LTS distros. I or one of my colleagues could probably whip that up fairly quickly.

The unfortunate fact is that, pressure or no, you're talking about enterprise Linux distributions with ~decade-long support timeframes where they won't modernize within those release series. Even for a relatively recent source package, Ubuntu 16.04 (their latest LTS) ships a python-psutil package which will break attempts to pip install anything into the system context with a transitive dependency on the psutil package when using (future) pip 10. I agree pip developers shouldn't be hamstrung by distro package maintainers, but it also seems like a bit of a gamble making pip 10 basically incompatible with long-lived distros in hopes it will encourage users put pressure on them to change their packaging. I'm pretty sure the answer from most distros will be, "We ship a (much older) packaged version of pip which doesn't exhibit this problem, use that instead."

[pradyunsg]

@fungi Thanks for the PR but I think even for that, I don't want pip to have another option like --ignore-installed unless we have a plan to remove the new option.

[fungi]

No worries. I thought it might be a way out of this, but if the primary concern from pip maintainers is how quickly they'll be able to rip it back out then it's not going to help for any actually useful (~7-year) timeframe. Some distro package maintainers I've checked with also seem to not see a problem with this, as they'd rather users didn't use pip/PyPI in the first place. Given the opinions on both sides it's probably better if pip simply stopped claiming to support installation outside virtualenvs at all, but that's obviously not my call.

[pfmoore]

if pip simply stopped claiming to support installation outside virtualenvs at all

We definitely wouldn't recommend (and never have, to my knowledge) using pip to manage a Python installation that's managed by another package manager. Specifically, that means that if you're using something like a Linux distribution, where the system Python comes from a distribution specific manager like apt or yum, you shouldn't be using pip to manage files in that installation. That's just common sense - expecting 2 different package managers not to argue over who owns which files is pretty risky, at best. But that's not the same as not claiming to support use outside or a virtualenv. There's all the Windows users, for whom there's no distro package manager. Also on Unix there's pyenv, and Python built from source.

It's essentially a user choice how they manage their Python setup. We're not trying to dictate that, nor are the distros. But we're trying to clarify for users that they have to make that choice - using both and hoping things will be OK isn't really a viable option.

[fungi]

For what it's worth, I completely agree one shouldn't mix package managers in the same context, certainly at least for production scenarios. Most of the concern I've seen raised has been related to development environments for people working on complex system daemons written in Python (where dependencies of the software being developed overlap with other packaged applications provided by the distribution) and/or for ephemeral test systems where nobody cares if they're able to effectively upgrade packages in the future because they'll be deleting the server again in an hour.

The counterargument has been "well it used to sort of work" and people are always going to have knee-jerk reactions to perceived regressions [insert obligatory "overheating spacebar" XKCD link]. The actual breakage from pip not being able to uninstall distro-packaged libraries was mostly mitigated on Debian derivatives because they didn't share the same file tree and so the location where pip installed modules was hit earlier in the import search path than those provided by distro packages, forming a sieve of sorts (granted that did still pose some issues, e.g. modules renamed between different versions of a package continuing to be importable).

[pradyunsg]

We definitely wouldn't recommend (and never have, to my knowledge) using pip to manage a Python installation that's managed by another package manager.

To that end, we want to avoid "fighting" with the package managers by default -- see #1668 -- that should help in general from pip's end.

[jharshman]

Just ran into this issue with PyOpenSSL. Workaround for me right now is to run in a container. Is there a fix in the pipe for this? Or if there is a tried and true workaround is it documented somewhere?

[fungi]

It's entirely intentional behavior, and as discussed previously any attempt to provide a workaround in pip seems to need to come with a timeline for the pip maintainers removing said workaround (I have provided one at #5066 though I did not provide a recommended deprecation timeframe, and it may also need minor rebasing at this point).

[jharshman]

@fungi alright thanks. Just so I'm absolutely clear on this, Pip isn't going to handle this case anytime soon right? Which means I'm kinda locked in to the PyOpenSSL version that came along with distutils.

[ianw]

@jharshman I don't think pip is going to change on this.

It's very brute force, but if you have an existing system where things are working, you can basically do an "rm" of the egg files and pip10 will reinstall. Of course, this is dangerous for all the reasons which caused pip to stop doing this. However, it might get you out of a jam depending on the situation (e.g. https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=b423292cd0e5b3bd266e083e0a4a46f383b0960c)

[fungi]

Oh, there are lots of ways to get around it. You could use alternate means of uninstalling PyOpenSSL (is it installed from a package provided by your Linux distribution? then use its package manager to uninstall first) before asking pip to install a newer version. You could also manually delete the PyOpenSSL files out of the site-packages tree before calling pip. Or you could use a non-global context such as --user or a virtualenv rather than relying on system-wide PyOpenSSL. Or you could downgrade to an older version of pip.

[jharshman]

@ianw @fungi cool thanks for the information and the super quick replies!

[rezroo]

I'm trying to install openstack-ocata and I get this error:

2018-05-18 01:00:26.276 |   Found existing installation: Jinja2 2.8
2018-05-18 01:00:26.280 |     Uninstalling Jinja2-2.8:
2018-05-18 01:00:26.280 |       Successfully uninstalled Jinja2-2.8
2018-05-18 01:00:26.839 |   Found existing installation: PyYAML 3.11
2018-05-18 01:00:26.969 | Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I'm trying to remove the distutils installed PyYAML but I can't figure it out. So I have:

$ pip show PyYAML
Name: PyYAML
Version: 3.12
Summary: YAML parser and emitter for Python
Home-page: http://pyyaml.org/wiki/PyYAML
Author: Kirill Simonov
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python2.7/dist-packages

$ ls -l /usr/local/lib/python2.7/dist-packages/*yaml*
-rwxr-xr-x 1 root staff 854216 May 18 00:21 /usr/local/lib/python2.7/dist-packages/_yaml.so

/usr/local/lib/python2.7/dist-packages/yaml:
total 448
-rw-r--r-- 1 root staff  4921 May 18 00:21 composer.py
-rw-r--r-- 1 root staff  4456 May 18 00:21 composer.pyc
-rw-r--r-- 1 root staff 25145 May 18 00:21 constructor.py

Do I just delete the /usr/local/lib/python2.7/dist-packages/yaml?

[rezroo]

I also tried:

$ sudo pip uninstall PyYAML
The directory '/home/stack/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Uninstalling PyYAML-3.12:
  /usr/local/lib/python2.7/dist-packages/PyYAML-3.12.dist-info/INSTALLER
  /usr/local/lib/python2.7/dist-packages/PyYAML-3.12.dist-info/METADATA
  /usr/local/lib/python2.7/dist-packages/PyYAML-3.12.dist-info/RECORD
  /usr/local/lib/python2.7/dist-packages/PyYAML-3.12.dist-info/WHEEL
  /usr/local/lib/python2.7/dist-packages/PyYAML-3.12.dist-info/top_level.txt
  /usr/local/lib/python2.7/dist-packages/_yaml.so
Proceed (y/n)? y
  Successfully uninstalled PyYAML-3.12

But the error persists.

[ianw]

@rezroo Honestly whatever you're doing, it's not going to work out well if you're getting these errors. It's not clear if you're running via devstack or other ways to install openstack.

This PR won't be a good place to get help, I'd suggest possibly a mail to http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[rezroo]

I'm installing via devstack - ocata. I pip uninstall PyYAML and install it using pip install, then run devstack and I still get the message that it's a distutils install. So is the conclusion that some devstack component overwriting my preinstalled PyYAML and doing its own installation?

I'm happy to go to devstack and openstack community, but I just want to get a feel for how in the hell the packaging could be so misaligned.

[fungi]

Let's please have OpenStack troubleshooting discussions in appropriate forums, and not here.

[tony-caffe]

Just downgrade to version 8 and your golden. Sad to have to revert to an old version but this fixed it.

[rsilvery]

Running into this installing AirBnb Superset. Really frustrating as I can't remove PyYAML.

Installing collected packages: pyyaml, contextlib2, sqlalchemy-utils, vine, amqp, kombu, celery, urllib3, chardet, certifi, requests, geographiclib, geopy, gunicorn, webencodings, html5lib, bleach, polyline, unidecode, sasl, thrift, thrift-sasl, simplejson, sqlparse, flower, parsedatetime, pydruid, unicodecsv, humanize, jmespath, docutils, botocore, s3transfer, boto3, superset
Found existing installation: PyYAML 3.10
Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Even removing the site-packages/yaml directory doesn't do it. Downgrading.

[pradyunsg]

Downgrading to an older version of pip isn't going to help since pip has never been able to uninstall such packages.

The only difference is that older versions of pip will delete the metadata that makes the package discoverable to pip itself, but not Python.

[pfmoore]

@rsilvery You removed site-packages/yaml (which is the package itself) but you didn't remove site-packages/pyyaml-*.dist-info (I think the name is something like that, it's the dist-info directory for the PyYAML project). You need to remove that as well, so that pip doesn't see the metadata that tells it PyYAML is still present.

Sorry it's messy doing this by hand - that's basically why pip can't do it for you, though. The information you need to know about what to delete, and the information pip needs to know, are the same, and distutils doesn't save that information for you.

[newsgrep]

Could you please add an option to "pip list" and "pip freeze" to mark or filter distutils-packages somehow maybe by adding a "*" to there name or add an option to display distutils-packages (or not distutils-packages) only. So that it is more easy to find out which ones are pip recognized distutils-packages and which are not.

Also when the process of installing / updating a package begins could you please check early if it is a distutils-packages or not, so that it does break before the uninstall process is triggered?

[joba-1]

Hi, I'm a python beginner, and since this issue just hit me and cost me ~1h of investigating now: maybe you could change the wording of the error message to be more helpful and less misleading?

I did this on my opensuse leap 42.3:
pip install --upgrade http://192.168.1.4/hg/chordsplay/archive/tip.tar.gz
It spit out:
Cannot uninstall 'chordsplay'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
It is likely I do something wrong here, but I'm sure it is not a distribution utilities installed project, since I wrote it myself and never produced an opensuse rpm for it. I always upgrade it this way, no idea why it does not work anymore now.

Finally I found fungis tip above how to handle this, hope this helps resolving it for me.

For the error message, how about something like: "Could not find pip meta data of package 'chordsplay' '/here' or '/there'. Please uninstall with the same tool you installed it." (which might have been python setup.py in my case)

[chrahunt]

Since it's been a few years with no sign of reverting this change, I will close this issue.

@joba-1, thanks for bringing this up. We know in general that we have some improvement to do in our error reporting and appreciate any feedback! To help get focused attention on this, please feel free to file a separate issue with a proposed wording.