Macvim does not find standard macOS python3

[dlejay]

Steps to reproduce

1. Install python with xcode-select --install
2. Install Macvim via the .dmg file
3. Launch Macvim and type :python3 --version
4. Launch Terminal.app and type python3 --version

Expected behaviour

Macvim should find python, since it is installed by macOS.
Terminal.app replies Python 3.9.6

Macvim looks for the wrong default path. I guess it's really looking for
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Python3

Version of Vim and architecture

9.0, macOS, arm64

Environment

macOS Ventura 13.1

How MacVim was installed

Downloaded from GitHub

Logs and stack traces

E370: Could not load library /opt/homebrew/Frameworks/Python.framework/Versions/
3.10/Python: dlopen(/opt/homebrew/Frameworks/Python.framework/Versions/3.10/Pyth
on, 0x0009): tried: '/opt/homebrew/Frameworks/Python.framework/Versions/3.10/Pyt
hon' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Framewor
ks/Python.framework/Versions/3.10/Python' (no such file), '/opt/homebrew/Framewo
rks/Python.framework/Versions/3.10/Python' (no such file), '/Library/Frameworks/
Python.framework/Versions/3.10/Python' (no such file), '/System/Library/Framewor
ks/Python.framework/Versions/3.10/Python' (no such file, not in dyld cache)

Vim configuration where issue is reproducable

No response

Issue has been tested with given configuration

• by running MacVim.app from GUI macOS interface
• by running vim/gvim/etc installed by MacVim
• by running other versions of vim (e.g. /usr/bin/vim)

Issue has been tested with no configuration

• by running mvim --clean (or gvim, supplied by MacVim distribution)
• by running vim --clean (in terminal, supplied by MacVim distribution)
• by running vim --clean (in terminal, other suppliers, e.g. /usr/bin/vim)

Other conditions

• The both Homebrew packages "vim" and "macvim" are installed

[ychin]

Hmm, currently MacVim is configured to look for Homebrew Python, as that's the most popular version. I guess I can add a fallback to look for the Xcode Command-Line Tools one as well as that's another possible way people would have Python installed.

I wouldn't say the default is wrong though. Homebrew is the most popular package manager for macOS and for installing the most-up-to-date Python version. One issue with the Xcode command-line tools version is that as usual, Apple bundles an old version of Python. The one you have is 3.9, but MacVim binary was linked against 3.10 (the next version that will be released soon will likely be linked against 3.11). Usually things will still work but there may be minor inconsistency that could happen if you have a mismatched Python version.

For now, you can add this line to your vimrc which will make it use the version you want. Ultimately it's impossible for MacVim to know which one you want so it never hurts to explicitly state it:

set pythonthreedll=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/Current/Python3

I will think about whether to add that to the list of fallback paths we search for in MacVim's default search paths for Python 3.

[dlejay]

Great answer, everything makes sense to me.

When I wrote that the default path is wrong, I meant that it looks for /Library/Frameworks/ Python.framework/Versions/3.10/Python which I thought was the folder in which it expected xcode python.
Homebrew is the most popular package manager, there's no doubt about that.

By the way, the same python problem arises if Macvim is installed via Homebrew but without having installed python before. It also happens when python3.10 is installed by Macports but not set as default python.

Now the main problem as you explained is that Macvim is linked to a particular version of python (a typical package manager problem). I see two possible solutions to this:

1. Macvim could always link with the default python on macOS. The default python on macOS is a bit old but only a bit (it gets updated with every new version of macOS). Also, every user who installed homebrew will have macOS python installed since the first thing that homebrew does is run xcode-select --install. This command is also the first one when installing Macports by hand. The website from Macvim could tell the user to run xcode-select --install before installing Macvim.
2. Macvim warns the user that it depends on python3.10 (otherwise, how am I supposed to know which version Macvim expects ?) and that the path to python3.10 should be manually supplied to Macvim (since it's a GUI app that is not supposed to read PATH). Before posting I looked at :help macvim and I didn't find information on python.

In both cases, I would suggest adding some documentation to the Macvim help (and to the macvim website) on this subject.

Is Python an isolated case, or are there other things that Macvim links to that expect the Homebrew version ?

[ychin]

Macvim could always link with the default python on macOS

I would much rather have MacVim link with the Homebrew Python because as I said that's the more popular version as far as I understand. It's also much newer (it now has 3.11, which was released recently), which I prefer. I also want to point out that the problem with xcode-select is that it's not "macOS" Python per se, but it's Xcode Python. Also, different macOS versions support different versions of Xcode, which come with different versions of Python. MacVim currently supports down to 10.9 (in later release it will be split into separate 10.9-10.12 and 10.13+ binaries) and so we need to link against something that most people can install independent of macOS versions. (Side Note: I can see this problem being exacerbated in near future as Apple is going crazy aggressive in removing support for older Macs from new macOS releases, so there will be more and more capable / usable Macs left on older macOS versions not by choice)

Macvim warns the user that it depends on python3.10 (otherwise, how am I supposed to know which version Macvim expects ?) and that the path to python3.10 should be manually supplied to Macvim (since it's a GUI app that is not supposed to read PATH). Before posting I looked at :help macvim and I didn't find information on python

Yeah I have some plans to make the docs / error message clearer. I do have to point out that this is not a MacVim-specific problem. Python support is built in to Vim, and the pythonthreedll setting is built in to any Vim built with +python3. It's just less of an issue for say Vim installed as part of a package manager (e.g. Homebrew for macOS / apt for Ubuntu) because the version of Vim built this way would have made sure to link Vim against the Python version provided by the package manager. It's only an issue for something like MacVim where we release a pre-compiled binary that needs to "just work".

Is Python an isolated case, or are there other things that Macvim links to that expect the Homebrew version ?

The other scripting languages are also all dynamically linked. They are usually less annoying because their library paths aren't explicitly versioned like in Python. In Vim, scripting languages can be either statically or dynamically linked, and MacVim is using the dynamic route to allow using the pythonthreedll / etc settings to let the user change it to fit their needs.

[eirnym]

Python automatic detection is here https://github.com/macvim-dev/macvim/blob/master/src/MacVim/vimrc.

If MacVim haven't found homebrew's python, It tries to detect it from MacPorts and from official installation.
Currently, on the master branch it tries to detect Python 3.10.

[ychin]

Btw @dlejay I'm currently working on better/more flexible Python integration for Vim. Do you mind telling me what type of plugins you are using that require Python? I'm doing some casual testing and curious what plugins people are using that actually require Python. In particular, some of the previously popular plugins like denite and deoplete have migrated to newer plugins (in this instance ddu.vim) which don't use builtin Python anymore, so I just want to make sure I'm actually testing plugins people are using.

[eirnym]

@ychin

I'll answer from my side:

I use Python from MacPorts mainly 3.11, sometimes 3.10. Homebrew is not installed

Plugins I use:

• Yggdroot/LeaderF
• editorconfig/editorconfig-vim
• Valloric/YouCompleteMe
• vim-python/python-syntax
• psf/black

[dlejay]

@ychin That would be :

• SirVer/ultisnips
• psf/black

[ychin]

Oh cool that's helpful. I knew about some of them but not others. FWIW with the change I'm working on you would be able to freely mix Python versions in the future so in this case for example it would be totally fine to use the Xcode Python 3.9 for a MacVim build (right now it kind of works but you really shouldn't, which is why I don't want to add the code to auto-detect Xcode's Python installation which is still on 3.9).

[holybit]

@ychin Another thing to think about is those who use a homebrew installed python but use it in conjunction with virtualenv. For example, on my Apple Silicon mac laptop I have homebrew with python3:

❯ /opt/homebrew/bin/python3 --version
Python 3.11.2
❯ echo $VIRTUAL_ENV
$HOME/virtualenv
❯ $HOME/virtualenv/bin/python3 --version
Python 3.11.2

# symlink hack to get python3 support in MacVim
❯ pwd
/Library/Frameworks/Python.framework/Versions/3.10
❯ ls -alrth Python
lrwxr-xr-x root wheel 39 B Mon Mar 20 13:40:49 2023 Python ⇒ $HOME/virtualenv/bin/python

# Here is where the Xcode python is installed
❯ /Applications/Xcode.app/Contents/Developer/usr/bin/python3 --version
Python 3.9.6

[ychin]

Right. In your case though, couldn't you use Homebrew to install the exact version of Python to use? (Instead of python3, Homebrew allows you to install [email protected] or [email protected]). Just tell MacVim to locate that specific version and you can use other Python versions for other uses. Maybe I'm missing how the virtualenv aspect matters.

I'm also not understanding why you need to symlink Python. You just directly set pythonthreedll to whatever Python lib you are using and you shouldn't need to symlink anything.

Either way, once we get Python 3 stable ABI support I think we can make the detection smarter as we will be guaranteed to work across Python versions. I happen to have a PR on Vim (vim/vim#12032) to solve this issue once and for all but it's been not getting much traction. You should go voice your support :)

Edit: I edited the comment because I forgot what this issue was about.

[eirnym]

@ychin Nice feature! Thank you for an inclusion