python: stop shadowing system python.

[MikeMcQuaid]

Use the python2 etc. binaries instead and provide a link to python in a directory not in the PATH by default. With Homebrew/brew#2897 this requires no formula changes and just bumps for those that hardcode python's full path.

[tdsmith]

I appreciate there's a certain consistency to this change but I think this is going to prove more frustrating to users than the status quo, which I think is manageable. Can you explain the rationale for this change? I think it'll help engage and explain this idea to the community.

My strongest feeling remains that the majority of python installs come from an intentional decision to brew install python and I don't think this change serves those people well.

For example, brew install python is a good and widely-recommended way for a beginner who's just starting out with Python to get a reasonable Python environment set up on their Mac. Many of those people still want Python 2 and don't know what a PATH is yet. I would hesitate to recommend a Homebrew solution to those people if python@2 were keg-only because it increases the conceptual and practical burden of getting to a point where they can pip install requests. I think it's painful for more experienced Homebrew users as well; the keg-only user experience isn't very good, most Homebrew users haven't encountered it before for software they actually want to use directly, and I think many won't realize they have to add something to PATH in order to use the formula they just installed.

Some nits:

• Maybe the least consequential question: when we use a Homebrew python internally, do you want to use python@2 or python? It feels weird to prefer a versioned formula but the python formula will break them more often because new 3.x releases come out regularly.
• Does python install a python executable? It would be weird if it only installed a python3 since the name of the formula is python. But if it installs a python, then the potential for confusion in the several-installed-Pythons case is even greater than it is now since print "foo" stops working.
• I don't generally think we should be beholden to the way folks use Homebrew in CI but I do think this feels like something that will break a lot of Homebrew-using CI.

I don't think we should make this change, but if we do, some way of signposting this to users and enabling a transition seems like a helpful thing to do. Maybe that takes the form of adding a keg-only python@2 formula without doing anything else for a while.

What's your timeline for making a decision on this? FWIW I'm traveling this weekend.

[MikeMcQuaid]

Can you explain the rationale for this change?

brew install python should install the latest version of Python. Python 2 is both a system duplicate and a versioned formula and should be keg only for both of those reasons.

My strongest feeling remains that the majority of python installs come from an intentional decision to brew install python and I don't think this change serves those people well.

The data disagrees. In the last 30 days:

install events in the last 30 days for python
================================================================================
 1 | python                                                   | 94,603 |  99.27%
 2 | python --with-tcl-tk                                     |    610 |   0.64%
 3 | python --with-tcl-tk --with-sphinx-doc                   |     13 |   0.01%
 4 | python --HEAD                                            |     11 |   0.01%
 5 | python --with-sphinx-doc                                 |     11 |   0.01%
 6 | python --with-quicktest --with-tcl-tk --with-poll --with |     10 |   0.01%
 7 | python --with-poll                                       |      6 |   0.01%
 8 | python --with-tcl-tk --with-sphinx-doc --with-berkeley-d |      6 |   0.01%
 9 | python --with-quicktest --with-tcl-tk --with-poll --with |      5 |   0.01%
10 | python --with-berkeley-db@4                              |      4 |   0.00%
11 | python --with-tcl-tk --with-poll --with-sphinx-doc --wit |      4 |   0.00%
12 | python --with-tcl-tk --with-poll --with-berkeley-db@4    |      2 |   0.00%
13 | python --without-readline --without-sqlite --without-gdb |      2 |   0.00%
14 | python --HEAD --with-quicktest --with-tcl-tk --with-poll |      1 |   0.00%
15 | python --HEAD --with-sphinx-doc --with-berkeley-db@4     |      1 |   0.00%
16 | python --HEAD --with-tcl-tk                              |      1 |   0.00%
17 | python --HEAD --with-tcl-tk --with-poll --with-berkeley- |      1 |   0.00%
18 | python --universal                                       |      1 |   0.00%
19 | python --with-poll --with-berkeley-db@4                  |      1 |   0.00%
20 | python --with-poll --with-sphinx-doc                     |      1 |   0.00%
21 | python --with-poll --with-sphinx-doc --with-berkeley-db@ |      1 |   0.00%
22 | python --with-quicktest --with-tcl-tk --with-sphinx-doc  |      1 |   0.00%
23 | python --with-tcl-tk --with-poll                         |      1 |   0.00%
24 | python --with-tcl-tk --with-poll --with-sphinx-doc --wit |      1 |   0.00%
25 | python --without-sqlite                                  |      1 |   0.00%
================================================================================
Total                                                         | 95,299 |    100%
================================================================================
install_on_request events in the last 30 days for python
================================================================================
 1 | python                                                   | 44,056 |  99.09%
 2 | python --with-tcl-tk                                     |    330 |   0.74%
 3 | python --with-tcl-tk --with-sphinx-doc                   |     13 |   0.03%
 4 | python --HEAD                                            |     11 |   0.02%
 5 | python --with-sphinx-doc                                 |     11 |   0.02%
 6 | python --with-poll                                       |      6 |   0.01%
 7 | python --with-tcl-tk --with-sphinx-doc --with-berkeley-d |      6 |   0.01%
 8 | python --with-quicktest --with-tcl-tk --with-poll --with |      5 |   0.01%
 9 | python --with-berkeley-db@4                              |      4 |   0.01%
10 | python --with-tcl-tk --with-poll --with-sphinx-doc --wit |      4 |   0.01%
11 | python --with-tcl-tk --with-poll --with-berkeley-db@4    |      2 |   0.00%
12 | python --without-readline --without-sqlite --without-gdb |      2 |   0.00%
13 | python --HEAD --with-quicktest --with-tcl-tk --with-poll |      1 |   0.00%
14 | python --HEAD --with-sphinx-doc --with-berkeley-db@4     |      1 |   0.00%
15 | python --HEAD --with-tcl-tk                              |      1 |   0.00%
16 | python --HEAD --with-tcl-tk --with-poll --with-berkeley- |      1 |   0.00%
17 | python --with-poll --with-berkeley-db@4                  |      1 |   0.00%
18 | python --with-poll --with-sphinx-doc                     |      1 |   0.00%
19 | python --with-poll --with-sphinx-doc --with-berkeley-db@ |      1 |   0.00%
20 | python --with-quicktest --with-tcl-tk --with-poll --with |      1 |   0.00%
21 | python --with-quicktest --with-tcl-tk --with-sphinx-doc  |      1 |   0.00%
22 | python --with-tcl-tk --with-poll --with-sphinx-doc --wit |      1 |   0.00%
23 | python --without-sqlite                                  |      1 |   0.00%
================================================================================
Total                                                         | 44,461 |    100%
================================================================================

More than twice the number of Python installs came from being a dependency.

For example, brew install python is a good and widely-recommended way for a beginner who's just starting out with Python to get a reasonable Python environment set up on their Mac. Many of those people still want Python 2 and don't know what a PATH is yet.

They have a /usr/bin/python that is a reasonable Python environment on their Mac.

Maybe that takes the form of adding a keg-only python@2 formula without doing anything else for a while.

I'm not sure duplication on this is the way to go and I also don't think we'll reach anywhere near the majority of Homebrew users no matter what we do. That said, I'm not opposed to a blog post and tweet about this.

What's your timeline for making a decision on this? FWIW I'm traveling this weekend.

I don't plan on merging this over the weekend but I would like to have this merged before e.g. August.

[tdsmith]

Thanks for pulling the data. That's still 45k intentional installs—I apologize for my rhetorical flourish of "the majority" but the point I wanted to make was that there's a huge mass of people who actually do want to use Python 2.

They have a /usr/bin/python that is a reasonable Python environment on their Mac.

It's an arguable point but, empirically, this is not at all the opinion of the OS X-using Python community. There are a lot of good reasons to brew install python, which is why it's widely recommended. This change makes that less compelling.

[mithrandi]

I don't think /usr/bin/python is a reasonable Python environment on macOS for any purpose other than running Apple's system software written in Python. It is built in an idiosyncratic way (LibreSSL etc.), and has some very ancient system copies of some things (eg. distribute, Twisted) which make life hard on anyone trying to use it for the usual purposes.

As far as the formula itself goes, regardless of how many people are doing brew install python, of those that do it how many are going to expect Python 3 to be installed? Expectations may change over time, especially if this change is implemented, but I'm not sure what benefit there is to pushing people in that direction.

[ilovezfs]

I don't think /usr/bin/python is a reasonable Python environment on macOS for any purpose other than running Apple's system software written in Python.

This is belied by the fact that about 170 formulae in homebrew/core use system Python.

[alex]

Is there a way that existing python users could be migrated to python@2, instead of just the latest version of python (Python 3)? For me that would significantly mitigate my fear of breakages.

[ilovezfs]

@alex not really. One reason is that the migration would put a symlink in the Cellar python -> python@2 which would be in the way of python 3's Cellar/python directory.

[dreid]

Overall I think this is clearly the right long-term change.

1. Beginners should generally be using the latest version of Python.
2. python == python3 is clearly the intent of the Python developers.
3. The system python is wildly inappropriate for anyone and will hopefully one day be private if not removed completely.

However, it's not clear to me that the data supports this change at this time or that the migration path is well defined.

To the point of data, I'd be most interested if more people intentionally install python3 than python. This to me would strongly indicate a preference of the homebrew users for python3.

And as to migration, I'd expect some warning period at least if not as @alex suggested a migration from python->python@2 for existing python users, this is roughly the equivalent of upgrading making the openssl formula install libressl, a fork that is subtly incompatible (Python3 is a fork by the same authors as Python but it's subtly incompatible none the less).

Finally to the point about the system python… it only ever gets updated because the only 2.7 releases that can happen are "security" releases. Using it, and attempting to update software used by it has historically led to more confusion for beginners than anything else, almost every Twisted meetup/sprint for the last 10 years or so has seen people with broken system python installations because of it's idiosyncratic build and rather arbitrary set of pre-installed packages.

Just because some people have managed to use it without breakage doesn't mean we should encourage more people to use it, it is fragile to say the very least.

[ilovezfs]

To the point of data, I'd be most interested if more people intentionally install python3 than python. This to me would strongly indicate a preference of the homebrew users for python3.

In the last 30 days, 46,645 python 3, 44,461 python 2.

[ilovezfs]

However, it's not clear to me [...] that the migration path is well defined.

@dreid you can take a look at #11083 as an example of a similar migration.

[dstufft]

Like @dreid I think this is the correct long term change, but I'm not sure if it's the correct change to make now. For a large part of the Python ecosystem, "python" is still Python 2. I also agree that the system Python is not a very good Python to use, but I personally generally recommend people use pyenv to get their Python on macOS rather than Homebrew or the macOS system (or even the Python.org installers), though I do say brew is a good second choice.

My other question is what would happen with the existing python3 formula? Would it start failing or somehow redirect to the new 3.x version of the python formula?

[ilovezfs]

My other question is what would happen with the existing python3 formula? Would it start failing or somehow redirect to the new 3.x version of the python formula?

python3 would become an alias for python and existing python3 installations would be migrated from Cellar/python3 to Cellar/python.

[ilovezfs]

In terms of consistency with other formulae, I'm wondering if we go down this road, whether we'd need to make the main python formula (i.e. python 3) keg only as well.

If we don't, it seems like it will be confusing to brew install python but then have to invoke it as python3 not python. If you just run python, you'd be running the system python assuming no other modification to your profile.

How many users will run into that gotcha and be confused if we make python non-keg-only but have it installing suffixed binaries?

[FranklinYu]

@ilovezfs

If we don't, it seems like it will be confusing to brew install python but then have to invoke it as python3 not python.

How about having python symlinked to python3, so that user can invoke it as either python or python3 (while keeping system Python shadowed)?

[ilovezfs]

@FranklinYu that is the problem with packaging system duplicates. It wouldn't be acceptable to replace python 2 with python 3 without requiring manual PATH tweaking by the user since third-party scripts will expect python to mean python 2 on macOS.

[MikeMcQuaid]

@durin42 @DomT4 Just for what it's worth: 👎ing my PRs is a feature GitHub allows but it's also something that I find, particularly without other comment elaborating, to be negative, demotivating and rude.

Thanks for pulling the data. That's still 45k intentional installs—I apologize for my rhetorical flourish of "the majority" but the point I wanted to make was that there's a huge mass of people who actually do want to use Python 2.

Then again: more people who install python are not having it installed intentionally. Add that to the fact that Python is being treated neither consistently as a latest version, system duplicate or versioned formula. Add to that the argument about PATH can be flipped to be those users who don't know about their PATH are getting a python they don't want without choosing to do so and the argument for the status quo looks less persuasive.

It's an arguable point but, empirically, this is not at all the opinion of the OS X-using Python community. There are a lot of good reasons to brew install python, which is why it's widely recommended. This change makes that less compelling.

Please don't say things like "empirically" when you can't back that up with data. The last claim you made was falsified by the data. What you mean is: that's the opinion of you and others you speak to. Similarly, my opinion and that of many others I've met at conferences is that which python being overridden by installing a formula that depends on it is a terrible user experience.

I don't think /usr/bin/python is a reasonable Python environment on macOS for any purpose other than running Apple's system software written in Python. It is built in an idiosyncratic way (LibreSSL etc.), and has some very ancient system copies of some things (eg. distribute, Twisted) which make life hard on anyone trying to use it for the usual purposes.

Like @ilovezfs I disagree with this. I've been paid for Python development while using the system Python.

In the last 30 days, 46,645 python 3, 44,461 python 2.

So, more people want Python 3 than Python 2 (just).

Overall I think this is clearly the right long-term change.

That multiple people are disagreeing with this change but stating this makes me feel far more strongly that we should be doing this sooner rather than later. I've been watching the Python ecosystem for years at this point wanting to make a similar change and I don't see the Python ecosystem changing sufficiently. There's an argument about being the change you want to see; Homebrew has affected change on other areas of package management and I think this is another one where we can make a positive change.

In terms of consistency with other formulae, I'm wondering if we go down this road, whether we'd need to make the main python formula (i.e. python 3) keg only as well.

Yeh, I can see an argument for that.

If we don't, it seems like it will be confusing to brew install python but then have to invoke it as python3 not python. If you just run python, you'd be running the system python assuming no other modification to your profile.
How many users will run into that gotcha and be confused if we make python non-keg-only but have it installing suffixed binaries?

The main argument I see is that python (as mentioned above) could be reasonably expected to by Python 2. Then again, this is something we can decide to change in future (although I agree getting it right first time would be ideal). If we make it keg-only I don't see people being less confused but just more people will be told to run brew link python3.

How about having python symlinked to python3, so that user can invoke it as either python or python3 (while keeping system Python shadowed)?

Unless we're going to have no packages depend on Python or make this keg-only (which seems worse to me): we end up with the same problem that which python can change based on dependencies.

[DomT4]

Just for what it's worth: 👎ing my PRs is a feature GitHub allows but it's also something that I find, particularly without other comment elaborating, to be negative, demotivating and rude.

Apologies. I thought about this and decided a 👎 was simply easiest way to note that I find this change to be likely confusing, complex & quite frustrating for users, perhaps even more so than the status quo.

Given me & you have talked about this specific subject maybe, what, 10-15+ times over the last couple years, I wasn't sure what else I could add that I hadn't already, so I simply left the thumb as a sort of you know I think this is not worth the hassle it will be. As ever it wasn't a personal ding on you, your willingness to do the work or your ideas in general.

[DomT4]

On a purely technical note, given that you are taking the /usr/local/opt/python link for python3 in this PR, at least some of the brew uses python will need revision bumps. Thinking especially of vim, macvim, mercurial, etc which either link to the Python Framework or have hardcoded hashbangs for /usr/local/opt/python/bin/python2.7.

[MikeMcQuaid]

so I simply left the thumb as a sort of you know I think this is not worth the hassle it will be. As ever it wasn't a personal ding on you, your willingness to do the work or your ideas in general.

It's taken as a personal ding, intended or not.

Apologies.

Thanks, I appreciate that.

at least some of the brew uses python will need revision bumps. Thinking especially of vim, macvim, mercurial, etc which either link to the Python Framework or have hardcoded hashbangs for /usr/local/opt/python/bin/python2.7.

Cheers, yeh, there will need to be loads of revision bumps and maybe some formulae changes. Was holding off on them until this is a little further along.

[dstufft]

I don't think /usr/bin/python is a reasonable Python environment on macOS for any purpose other than running Apple's system software written in Python. It is built in an idiosyncratic way (LibreSSL etc.), and has some very ancient system copies of some things (eg. distribute, Twisted) which make life hard on anyone trying to use it for the usual purposes.

Like @ilovezfs I disagree with this. I've been paid for Python development while using the system Python.

With my pip maintainer hat on, we see a large number of errors/issues coming from people who use the system Python on macOS. Between SIP, sys.path being broken until relatively recently, and the preinstalled packages, it is generally more trouble than it's worth to use it. My number one recommendation to people who run into issues with it has up been not to use it, and instead use (in order of preference): pyenv, Homebrew, Python.org installers.

I've been watching the Python ecosystem for years at this point wanting to make a similar change and I don't see the Python ecosystem changing sufficiently.

I'm not sure what "changing sufficiently" actually means to you, but FWIW here is some data I have from installs from PyPI that originate from macOS:

Python Version	Downloads	Percent
2.7	20192004	78%
3.6	3124456	12%
3.5	1938864	8%
3.4	294802	1%
Unknown	157776	0.6%
2.6	39177	0.2%
3.7	3553	0.01%
3.2	113	0.0004%

Things are trending towards Python 3 (combined it has ~21% of the share of macOS installs) but Python 2.7 is still the vast majority at ~78%. Is that enough adoption for Homebrew to decide it's time to make a change like this? Well I don't know, I'm not a Homebrew maintainer and I don't use Homebrew Python myself (I use pyenv), but if it were my project I wouldn't feel comfortable doing that yet.

[MikeMcQuaid]

Is that enough adoption for Homebrew to decide it's time to make a change like this? Well I don't know, I'm not a Homebrew maintainer and I don't use Homebrew Python myself (I use pyenv), but if it were my project I wouldn't feel comfortable doing that yet.

The underlying question on this is: does Homebrew wish to follow existing user behaviour or seek to shape it in a way that both upstream Python and many members of the Python community agree is the desired future. Personally, I think we should be seeking to shape user and ecosystem behaviour towards a better future (and that's my view even when we ignore the fact that the status quo in Homebrew is problematic and inconsistent in multiple ways I've articulated above).

[MikeMcQuaid]

Very relevant to this discussion:

Effective Tuesday, June 20th, 2017, new Python applications pushed to Heroku will use the python-3.6.1 runtime by default (instead of python-2.7.13).
https://devcenter.heroku.com/changelog-items/1169

[DomT4]

Given your above comment, I guess I'll ask the question I didn't ask yesterday that was prompted by this wording:

Cheers, yeh, there will need to be loads of revision bumps and maybe some formulae changes. Was holding off on them until this is a little further along.

Are you proposing to migrate things more aggressively from python to python3, for example, vim poured from bottle would ship with python3 support instead of retaining python2 support by default?

[iMichka]

Just to chime in, I am +1 on making that change. For homebrew-science, I would like to slowly get rid of legacy Python over the next months/years. This will allow to bottle more stuff with Python 3 by default, and get rid of all the Python 2 / Python 3 option mess we are currently carrying around.

I also tend to think that the Python 2 vs Python 3 download values are slightly biased towards Python 2 because this is the default. Just because some people just historically run brew install python. Giving this a slight kick in the right direction, and I bet the download stats will look different. But I have no proof for that, it's just the impression I have :)

BTW I bet Apple has Python 2 as default in High Sierra? :(

[FranklinYu]

@iMichka I think Apple will stay on Python 2 for several future macOS. Not even the latest Python 2: Sierra comes with Python 2.7.10, which is released two years ago, even before El Capitan. I don't remember the Python version in El Capitan though.

[MikeMcQuaid]

Are you proposing to migrate things more aggressively from python to python3, for example, vim poured from bottle would ship with python3 support instead of retaining python2 support by default?

vim is a special snowflake where we'd likely keep things as similar as possible. In general: we prefer system Python unless there's network calls in which case we'd use Python 3 if possible or python@2 if not.

I also tend to think that the Python 2 vs Python 3 download values are slightly biased towards Python 2 because this is the default.

This is a good point and I agree.

[methane]

/usr/bin/python is still Python 2 on Ubuntu, Debian and Fedora, while it's not installed by default.

Instead of making this aggressive change now, how about just recommend :python3 dependency, instead of :python?
Currently, says:

If a library supports both Python 2.x and Python 3.x, the :python dependency should be :recommended (i.e. built by default) and the :python3 dependency should be :optional.

In Ubuntu, vim package is linked with Python 3 and vim-py2 package is linked with Python 2.

[methane]

I also tend to think that the Python 2 vs Python 3 download values are slightly biased towards Python 2 because this is the default.

This is a good point and I agree.

I agree. This is a chicken-egg problem.
Most people use Python 2 because it's default. Maintainer keep Python 2 as default because most people use it.

Clearly, Python 3 is recommended for new users.
And I think "recommended for new user" should be default.

[pmarques]

Can you please revert this please?

So, now no one knows when we are using system AKA /usr/bin or a version from brew AKA /usr/local/bin

$ which python
/usr/bin/python
$ which python2
/usr/local/bin/python2
$ which python2.6
/usr/bin/python2.6
$ which python2.7
/usr/local/bin/python2.7

[MikeMcQuaid]

@pmarques No. Please read our Code of Conduct and adjust future communication accordingly.

[pmarques]

@MikeMcQuaid updated, but this is closed, so I will need to open a new issue

[MikeMcQuaid]

@pmarques Do not open a new issue. We are not going to revert this. We are willing to accept pull requests to improve documentation or bugs in our implementation.

[FranklinYu]

@pmariano I thought scripts that are still maintained have already been using python2. If not, I think the maintainer deserves a notification.

[pmarques]

@FranklinYu you are seeing just brew, there are much more than that and the things are not consistent now from my POV.

• This broke packages and tools that I had installed: This tools are now using a different python with no modules installed
• Some other tools just installing a bunch of module inside my system... because pip is gone too
• This seems to be changed without notice in a short period of time
• etc

Since my system are already managed using some tools I end up making the link myself....

[FranklinYu]

@pmariano That's really sad. Any example of affected package and tools?

I think we may need some way to notify end user about breaking upgrade (because they do happen). However a message for every invocation of brew seems too verbose.

[MikeMcQuaid]

This seems to be changed without notice in a short period of time

There was notice in several forms:

• a pull request here
• brew outdated and brew upgrade combined with brew log python
• the caveats

I don't want to go around on this yet again.