Default to --user

[dstufft]

When pip is installed on a system that has an OS Python install there is currently a problem where pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python. This creates an issue where people are using pip to manage system level packages when they should likely be using the system package manager.

So my intention is that pip should default to --user however there are a few sticking points with this:

• How does this interact with Windows? Does this make sense there?
• How does this interact with altinstall'd Pythons? Specially such as are installed with tools like https://github.com/yyuu/pyenv
• What do we do for when people do invoke pip as root? Installing into /root/.local/ does not seem very useful.
• What does this mean for get-pip and the pypa install instructions?
• ~/.local/bin is not on many people's $PATH, is there anything that can be done about this?
• --user installs lack precedence to global easy_install'd packages, which can be quite unexpected.

There are a number of issues that are relevant here: #624 #1443 #1153 #1500 #1051

/cc @ncoghlan

[qwcode]

What does this mean for the pypa install instructions?

currently the mention of root or Administrator access is in a footnote, so the instructions would fundamentally stay the same (assuming get-pip automatically defaulted to --user as well). the change would be to explain that they will end up with installs that only apply to their current user.

regardless of the --user change, the current instructions need to explain that some distributions like debian, are disabling ensurepip, so they shouldn't go looking for it as a way to get pip.

[dstufft]

Long term I'm pretty sure they are going to keep it enabled, they are just figuring out how to do it.

[qwcode]

although this issue is described as being about where pip should manage packages by default, almost more important (to me at least) is that this is indirectly about where get-pip will place pip by default.

[qwcode]

How does this interact with altinstall'd Pythons?
Specially such as are installed with tools like https://github.com/yyuu/pyenv

the user scheme is .local/lib/pythonX.Y/site-packages, so if you're managing multiple pythons with the same major/minor version, you could end up with a mess I guess.

for tools like that, that manage real pythons, pinning pip to global mode would make sense for those.

[dstufft]

A relevant issue on the redhat/fedora bug tracker https://bugzilla.redhat.com/show_bug.cgi?id=662034#c10

[dstufft]

It looks like for Fedora/RedHat PATH=$PATH:$HOME/.local/bin:$HOME/bin is already in their /etc/skel/.bash_profile. This would make stuff installed with --user still show up by default, at least for bash users (default shell). Maybe other distros already have this?

[pfmoore]

There's nothing specially different about Windows that I know of. But we've only just got the standard Python directory onto people's PATH, I don't expect that getting a per-user scripts directory on is going to be particularly easy - and there may be technical difficulties as well, I'm not sure putting a user-level environment variable like %LOCALAPPDATA% into the system-level %PATH% even works :-(

I'm against rushing this. I would prefer waiting till we've had more experience with user installs and have managed to iron out the issues first. I know that's a chicken and egg problem, but I don't see rushing into a change as helping.

Also, with my Windows hat on, I'd have to say that making it harder for Unix users who haven't yet worked out that careless use of sudo is bad to do stupid things, isn't really that good a justification...

I'm assuming that this would only apply when running pip from a system Python.

[dstufft]

Well the different thing on Windows is there isn't a system provided Python like there is on *nix that also comes with system provided Python packages :)

I'm not particularly wanting to rush this either. I just wanted to start the discussion.

This doesn't really affect people who type sudo pip install as I think installing something as root should still go to the system site packages by default. This would be to guide people towards using --user when running as an unprivileged user. Right now what you get is that pip install as a regular user just bombs out with a permission error. Even if we improve that message to bomb out and suggest --user that's still not the greatest UX.

In talking to one of the Fedora people, I think the way it may make to do this is to implement a permissions check. If I have permission to write to the site-packages directory then I am a privledged user and pip installs to the global Python, If I do not then I am an unprivileged user and it installs to --user.

[ncoghlan]

As far as Windows goes, we usually duck the problem because we install Python to a non-privileged directory by default, so pip doesn't trigger UAC when installing globally. If someone changes that to install into Program Files instead, then UAC will trigger when they run pip (which is also OK). I'm not sure what happens if they select a per-user install in the installer.

As Paul notes, PATH on Windows doesn't include the user scripts directory yet (just the global one), so that's definitely worth taking into account.

I don't see any major barriers on the POSIX side though. How does this UX idea sound: for wheel based installs, check for permissions on the installation target directory, and if the user doesn't have write permissions, put up a prompt asking if they would like to do a --user install instead? A new "--global" or "--system" flag would force the "no" answer.

And then at some point in the future, we could skip the prompt and simply assume --user if the permissions weren't right for a global install.

[dstufft]

A prompt (except in the case of --no-input) probably makes sense for the transition step. I think checking permissions should make things work for Windows too since by default the location is in a spot where people have permissions on and so the only time they'd hit this would be if they are using a systems admin provided Python that explicitly didn't give them permission.

Essentially this makes our failure mode much better.

[qwcode]

making it harder for Unix users who haven't yet worked out that careless use of sudo
is bad to do stupid things, isn't really that good a justification..

sudo and security is really a secondary issue IMO.
this is primarily about preventing the conflict between OS packaging and pip in the system python.
Currently, linux users are often placed in an impossible situation.

1. OS Mandate: 'Use the OS pkg mgr; Don't infect your system with roque pip-installed packages (including a get-pip'd pip)'
2. Reality: "The OS packages (including pip) are too old, and I can't upgrade to the experimental release of the distro, just to get upgrades of some package. I'm going rogue...."

[qwcode]

another baby step is to document (assuming we test it) that you can get-pip.py --user, and have the PUG mention --user as a possibility for the install of virtualenv (and wheel, and twine too, if not in a virtualenv)

[ncoghlan]

FWIW, distro vendors also consider the status quo a problem and are looking at various ways to improve the tools for doing selective upgrades without impacting core OS components. Still worth us tackling from the upstream side though, since those efforts are at various stages of maturity and we want a consistent cross-platform solution for end users.

[pfmoore]

I have no problem with changing things to help co-operate with distros on Unix; if defaulting to --user on Unix and not on Windows is acceptable, that's fine with me. I just don't see switching from a bad experience on Unix to a bad experience on Windows as a good trade-off. And I'm yet to be convinced that --user on Windows is ready for prime time.

Note that triggering UAC when running pip is bad on Windows, because what it actually means is that pip won't run except in an elevated console window. It does not mean that pip users get a nice prompt which they can say "OK" to, unfortunately, that's only how GUI apps work...

[dstufft]

@pfmoore What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

[dstufft]

FWIW I'm totally OK with making this optional depending on Windows or not. I'm just wondering if the check of "Do I have permissions to write to the site-packages folder" check won't achieve that anyways in 99% of installs, and if installing to --user would be a better experience in the fraction of cases where you don't have write permissions to the site-packages directory.

In other words, if we check permissions the proposal isn't replacing global install with a user install, it's replacing a permission error with a --user install.

[qwcode]

the proposal isn't replacing global install with a user install
it's replacing a permission error with a --user install

ok, but recall that the permissions check idea (#1048) won't the be the easiest thing

[dstufft]

It'll be pretty easy in the Wheel case, and we can get the 99.9% case covered with sdists, it should only be a problem in setup.py's that do really funky things.

[pfmoore]

What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

Well, being able to write to the site-packages directory is so rare on Windows that I'm not sure how relevant the question is in practice. But if it happened, I would say that an error saying "system site-packages is not writeable - did you mean to use --user?" would be the sensible approach.

Actually, I think that would be better on Unix as well. Switching the behaviour depending on writeability is bizarre, and it's not clear to me if using --user for a personally-built Python in a writeable directory is the right approach (but I have precisely zero experience of such a setup, so I don't have anything to back that opinion up).

Suggestion: Start with an error suggesting --user as above. Once people are using --user more commonly, and we have more experience with it, then let's consider switching the default.

[ncoghlan]

The approach Paul suggests is roughly what I had in mind, but with a yes/no prompt rather than bailing out with an error message. There are pros & cons to those two approaches (mostly relating to how they fail when invoked from another script).

[rkuska]

Would it be possible to add to Python something like sys.localprefix (I see python-dev here)?
Sys.localprefix would be default to sys.prefix. Distros could define it same way as they define prefix (e.g. prefix = '/usr', localprefix='/usr/local'). Pip and easy_install would use sys.localprefix as a location to install to if used as sudo, if they don't have sufficient rights fall back to --user.

[bkabrda]

We've done something like this with "gem install" in Fedora 17 and Ruby devels were generally very satisfied about it, so I thought I'd share:

• "gem install foo" installs "foo" into ~/somedir if uid != 0
• "gem install foo" installs "foo" into /usr/local/somedir if uid == 0
• "yum install rubygem-foo" installs RPM-packaged "foo" into /usr/somedir

I think that choosing install dir based on user privileges would be very confusing for people; "pip install foo" should work the same way for all non-root users. If you consider majority of users, they'll want to "pip install" into their home and "sudo pip install" into a system-wide dir. Therefore I think it's the best approach to do this based on uid as shown above - and for those users who are not root but want to install into a system-wide dir, there is always the --target option.

[Ivoz]

@bkabrda that's an easy choice to make as long as you're not worried about adding scripts to $PATH. What have you done there?

[bkabrda]

@Ivoz as noted by @dstufft, Fedora already has $HOME/.local/bin on PATH, so everything worked out of the box (speaking of scripts).

[ncoghlan]

Slavek's suggestion sounds good to me for POSIX systems. It should also work nicely with containers, since stuff running in a container can be told to think it is uid 0, even though it's something else entirely from outside the container.

For Windows, I'm less sure what the right answer is. We don't have quite the same problem with getting into an argument with the system package manager, although it does exist to some degree (pip install vs MSI installers), and have the additional complication that even in Python 3.4, the installer's optional PATH modifications only add the global Scripts directory, not the per-user one.

Windows also has weirdness based on whether Python is installed to the default location (uncontrolled) or into Program Files (UAC privilege escalation needed to make changes).

[felixonmars]

The "localprefix" idea suggested by @rkuska looks nice to me, simple and easy. Also I really like the approach @bkabrda suggests, especially for distros like Arch that make python 3.x the default python.

One more note is, Arch has disabled ensurepip in the 3.4.0 release just like Debian (mostly because we want to provide latest versions of setuptools and pip, while the bundled versions may be outdated for ages), so it would be nice to have a tip for it.

@LVoz
For Arch we didn't add any path under $HOME to PATH, but I still think it fine. We already have wiki entry for Ruby that tells the user to add it.

[ncoghlan]

It would be nice if rather than just disabling it, the Arch and Debian developers would help Slavek work on his patch to have ensurepip reconstruct a wheel from the system versions and install that into virtual environments.

[bkabrda]

Just FYI, we already have working Python 3.4 RPM builds in Fedora's Copr build system (testing, not advisable to install them if you don't want to break anything). If you want to know more about how we approach this, have a look at the code etc, see my discussion with Barry Warsaw [2] at debian-python ML.
(Our patches still need some love before we propose them upstream, but everything works ok for us downstream ATM)

[1] http://copr-fe.cloud.fedoraproject.org/coprs/bkabrda/python-3.4/
[2] https://lists.debian.org/debian-python/2014/03/msg00046.html

[felixonmars]

Sorry, but AFAIK Arch never supported wheel, nor do we want to make ensurepip invoke package manager (correct me if I understand this approach wrong, though).

For ensurepip, it would be a nice idea to warn the user not to install pip using ensurepip though, and that's the most I can think of for now.

[takluyver]

It looks like the issue there is that PATH (for finding commands) and sys.path (for finding Python imports) have opposite priorities, so the launcher script from one installation tries to load the package from the other. It's presumably not specific to pip - it could affect any package that installs a script.

I'm not sure what, if anything, we can do about that, but I agree it's worth thinking about.

[pfmoore]

Yep, that's the main "complication of USER_SITE" I had in mind. IMO, deprecating wrapper scripts and only supporting python -m pip is the most effective solution. (I'm not against wrappers per se, but the problems we see with them are usually people accidentally running the wrong wrapper, so fixing wrapper script issues is usually not a pip issue as such, more of a "diagnosing and fixing user environment misconfiguration" issue)

[takluyver]

You mean deprecating the pip wrapper script, or deprecating wrapper scripts as a thing pip can install?

I'm certainly -1 on the latter - the ability to install commands is very useful, even if it causes problems.

Deprecating pip in favour of python -m pip might be more reasonable. As pip installs into the Python environment it's running in, it makes sense to be clear which one that is. But it would still be a loss of convenience, and a big break in what people are used to.

[pfmoore]

I did mean specifically the wrapper scripts for pip itself. Agreed it's a loss of convenience, and I wouldn't be against retaining them in some form (#3164 suggests having a new pip-cli project that just supplies the wrappers) but the key point would be for python -m pip to be the supported means of running pip.

[ErikBjare]

Couldn't calling the pip wrapper script show a warning if the current python (in the PATH or whatever would be used with python -m pip) is not the python used by pip? Maybe even delegate to python -m pip if called this way?

Similarly, pipenv warns if the user is already in a virtualenv, and if so uses that virtualenv instead of managing it's own, after emitting a warning.

[pradyunsg]

Okay, noting here that #7002 has been approved by 3 of 6 pip maintainers at this point.

It does make pip switch from non-user installs to user installs by default, in most cases where we really should be doing user installs anyway. I have two broader concerns with making this change, that we should figure out before making the release that makes that enhancement.

• Do we want a mechanism to explicitly opt-out of user installs once we switch to it by default? If so, how would it be different from --no-user? (I'm on board for renaming --no-user to --global as an explicit opt-out).
• What is our "transition" strategy?
  • How do we want to be communicate this change?
    • tell users to be careful when making this upgrade?
  • What kinds of issues do we expect users to raise, when this change is implemented + released?

[takluyver]

I'd say --global is misnamed if you're inside an environment. For flit install, the opposite of --user is --env, which I loosely meant to include the "system Python environment", but I don't think that's totally clear either. Maybe --no-user is still the best choice - it's clearly the opposite of --user.

I think --no-user (or the equivalent environment variable or config entry) should get you back the existing behaviour, since the default at present is effectively user = False.

The only scenario I've come up with where the change would be confusing is if you think you have permission to install into an environment (with user site enabled, e.g. conda) when actually you don't: a failure is clearer than doing a user install, even if there's a log message about it. I haven't thought of any good way to improve that.

It's also possible that the writability detection could go wrong. A false positive just gives you the existing behaviour (try and fail to do a normal install). A false negative would do a user install when it could have done a normal one.

[pfmoore]

It does make pip switch from non-user installs to user installs by default.

Not (for example) on Windows, where (by default) the Python install is a per-user install and site-packages is writeable by default. That's something I prefer in general about #7002, because my experience since this issue (#1668) was opened is that it's way too easy to get in a mess with "mixed" environments where you have packages like pip installed both in site-packages and user-site. So I'm now in favour of doing system installs if at all possible, and only doing user installs in cases like Linux system managed Pythons, where site-packages can't be written to.

Having said that:

1. There seems like no point to a --no-user option, as Default to --user install in certain conditions #7002 means you get a user install only if a system install would fail anyway.
2. The biggest thing I think we need to communicate regarding transition is that people need to be very careful about multiple installs, and that's something that can only be dealt with by user understanding and education, not by any workaround in pip. It's basically a core Python issue. We could extend pip check to report when there's a site and a user install of a package and the user one is shadowing the site one, I guess...

[takluyver]

There seems like no point to a --no-user option

To be clear, this already exists, although it's probably not often useful. If you've got user=true in a config file, --no-user should override that.

[pfmoore]

To be clear, this already exists

Sorry, I should have been clearer. I see no point to having "a mechanism to explicitly opt-out of user installs once we switch to it by default", at least in the sense that I assume @pradyunsg meant, which is about opting out of the #7002 behaviour, which is not the same as "switching to user installs by default", as I said.

Actually, I'm not sure that is any clearer ;-) Maybe all I want to say is "we don't need any more options than the ones we currently have"...

[pradyunsg]

Hmm... @pfmoore After #7002, what is needed to call this issue resolved?

PS: I'd written that earlier comment in a hurry and it was accidentally posted. The bullets are OK; I was still drafting the paragraph before it, when it got posted. Sorry about any confusion that might've caused.

[pfmoore]

@pradyunsg Given that I'm now much less inclined to feel that --user as a default is a good idea unless it's absolutely necessary (the cases #7002 addresses) I'd rather say that we can abandon this issue as no longer a good idea now that #7002 is done.

Taking your 2 bullet points as things we might need to count #7002 as complete, my comments above cover that.

[Aricg]

I followed the release notes here.
https://pip.pypa.io/en/stable/news/#id3
"Default to doing a user install (as if --user was passed) when the main site-packages directory is not writeable and user site-packages are enabled. (#1668)"

My builds are now failing with

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

This seems fixed by removing the --user from my pip commands.

Was this expected behavior? The release notes to not seem to reflect this.

link to the script that now fails on pip 20.0.1 (removing the --user fixes the hard fail)
https://github.com/lfit/releng-global-jjb/blob/master/shell/python-tools-install.sh

[takluyver]

If it is correct that user site packages aren't importable from that environment, then that is the expected behaviour, but it's orthogonal to this PR. That check was introduced years ago (#567), but it looks like it was broken with venv until recently (#7155). There's a release note for that:

Correctly handle system site-packages, in virtual environments created with venv (PEP 405).

[Aricg]

Ah yes, thank you for your comments. I figured this out...
my script was populating .local/bin/
when I ran

python3 -m venv ~/.local

and then in a ubuntu login shell (#!/bin/bash -l) .local/bin gets added to path (so calling python3 now calls the .local/bin/python3 executable) Which we don't want.
I never needed to run python3 -m venv ~/.local
or call my scripts from a login shell, and removing both or either of these problems fixes it.

#fresh ubuntu docker
root@7d8107816f64:/# python3 -m pip install  --user --upgrade pip
Successfully installed pip-20.0.1
root@7d8107816f64:/# python3 -m pip --version
pip 20.0.1 from /root/.local/lib/python3.6/site-packages/pip (python 3.6)
root@7d8107816f64:/# ls root/.local/bin/pip
pip     pip3    pip3.6
#Now we populate ~/.local/bin/
root@7d8107816f64:/# python3 -m venv ~/.local
root@7d8107816f64:/# ls root/.local/bin/
activate          activate.fish     easy_install-3.6  pip3              python
activate.csh      easy_install      pip               pip3.6            python3
root@7d8107816f64:/# ./root/.local/bin/python --version
Python 3.6.9
root@7d8107816f64:/# ./root/.local/bin/python -m pip install  --user --upgrade pip
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

So if we ever run the executable .local/bin/python we can not pip install with --user

[takluyver]

Oh, I'd missed that you were doing python3 -m venv ~/.local . That's likely to confuse tools like pip, because ~/.local is the prefix for --user installs (on Linux). venvs and --user are two different ways you can install packages without making systemwide changes: you should use one or the other. Creating a venv in ~/.local makes some weird hybrid of the two.

[tgs]

The new behavior is presumably convenient for many, but 'user' installs have caused me some grief as a web developer - I often need to prepare virtualenvs that will be deployed to our production machines, and if any package is installed in my user library, then by default pip will refuse to install it to the virtualenv I'm preparing, and will still exit with success. The notification that it decided not to install the package is buried inside 30 or 40 pages of output from the script I use to prepare the virtualenv. The end effect is that once the virtualenv is deployed, the packages from my home dir are no longer available, so some random packages are missing from my production web app, and it falls over dead.

I'm willing to admit that my use case is not the typical one, so maybe this was still a good idea from a general usability point of view.

For others who have had similar problems, there are a few ways around it:

• Add --force-reinstall to all your pip install command lines in scripts etc
• Delete ~/.local/lib/python* and edit ~/.pip/pip.conf to add user = no in the global section:

[global]
user = no

• Do something more drastic, like change ~/.local/lib to a file rather than a directory 😏

As always, dear packaging maintainers, thank you for all you do - I'm sure it's virtually impossible to change anything without breaking someone's stuff, and the fact that you forge on anyway is actually great, because it lets us make progress.

[tgs]

Ah, user installs can also cause huge problems in continuous integration environments, where lots of different stuff is run as the same user account, but people still expect builds to be isolated from each other. We solved this by making ~/.local read-only, but the solutions above would probably also work.

[takluyver]

When you create a virtualenv with the default options, it's isolated: it can't see your user or system site-packages, and consequently pip should only work with packages installed in the env. This became the default many years ago, because of exactly the kinds of issues you describe.

If you create virtualenvs with the --system-site-packages option (or very old versions of virtualenv where that was the default), they act as an overlay on the packages you'd otherwise have installed, both system-wide and in your home directory. If you want to have system packages visible but not user packages, you can run Python with the -s option or the PYTHONNOUSERSITE environment variable.

[tgs]

Oh! I'm very sorry, you're totally right. My organization's weird system-site-packages policy strikes again. Thanks for pointing out some solutions.

[takluyver]

No problem. System site packages was more common a few years ago when we often relied on system package managers for things like numpy, so maybe it's worth revisiting that policy. But there may still be sensible reasons to use it today.

[pradyunsg]

Closing in line with #1668 (comment).

Many thanks @takluyver! ^>^