virtualenv_create sets include-system-site-packages = true

[jiasli]

By installing https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/azure-cli.rb, the virtual environment created by virtualenv_create has

$ cat /usr/local/Cellar/azure-cli/2.27.1/libexec/pyvenv.cfg
home = /usr/local/opt/[email protected]/bin
include-system-site-packages = true
version = 3.9.6

This causes conflicts with libraries installed to system site-packages (/usr/local/lib/python3.9/site-packages): Azure/azure-cli#19027

According to Homebrew/brew#1790, this should not be the expected behavior. Am I doing anything wrong?

--system-site-packages from venv module defaults to false.

[carlocab]

Yes, we changed this in Homebrew/brew#11611. I do think having access to the system site packages can be useful, though admittedly it can cause issues at times (like the one you're seeing).

Both settings for --system-site-packages has its disadvantages: when set to true, you could break things with a pip install. When set to false, you can't use Homebrew numpy in Homebrew jupyterlab, for example. Hard to say which one is worse, but I do think there are workarounds for the pip install issue.

[jiasli]

Thanks @carlocab, I wish I can mark your answer, but there seems to be some issue with GitHub that I can't click the "Got it!" or "Mark as answer" button 😥:

[jiasli]

Well. Did a quick test and it seems --user won't work. The user site-packages (/Users/runner/Library/Python/3.9/lib/python/site-packages) is still included in sys.path:

$ pip3 install --user requests --force-reinstall

$ pip3 list -vv
Package            Version   Location                                                  Installer
------------------ --------- --------------------------------------------------------- ---------
certifi            2021.5.30 /Users/runner/Library/Python/3.9/lib/python/site-packages pip
charset-normalizer 2.0.4     /Users/runner/Library/Python/3.9/lib/python/site-packages pip
idna               3.2       /Users/runner/Library/Python/3.9/lib/python/site-packages pip
pip                21.1.3    /usr/local/lib/python3.9/site-packages                    pip
requests           2.26.0    /Users/runner/Library/Python/3.9/lib/python/site-packages pip
setuptools         57.0.0    /usr/local/lib/python3.9/site-packages                    pip
urllib3            1.26.6    /Users/runner/Library/Python/3.9/lib/python/site-packages pip
wheel              0.36.2    /usr/local/lib/python3.9/site-packages                    pip

$ /usr/local/Cellar/azure-cli/2.27.1/libexec/bin/python -c "import sys; print(sys.path)"
[ ... '/Users/runner/Library/Python/3.9/lib/python/site-packages', '/usr/local/lib/python3.9/site-packages']

[carlocab]

Huh, I always thought --system-site-packages should exclude the user site-packages directory. However, I also see that azure-cli's site-packages directory precedes both the user and the system site-packages:

❯ /usr/local/opt/azure-cli/libexec/bin/python -c 'import sys; print(sys.path)'
['', '/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/Cellar/azure-cli/2.27.1/libexec/lib/python3.9/site-packages', '/Users/carlocab/Library/Python/3.9/lib/python/site-packages', '/usr/local/lib/python3.9/site-packages']

sys.path should be searched in the order it appears, so I now don't understand why doing a pip install can break a Homebrew-installed azure-cli.

[jiasli]

Per PEP 405 Isolation from system site-packages:

PEP 370 user-level site-packages are considered part of the system site-packages for venv purposes: they are not available from an isolated venv, but are available from an include-system-site-packages = true venv.

sys.path should be searched in the order it appears, so I now don't understand why doing a pip install can break a Homebrew-installed azure-cli.

Well, this is a little bit complicated, as we changed the packaging model from regular package to namespace package, I provided more info in Azure/azure-cli#19027 (comment).

[carlocab]

Ah, I see. That is a little complicated. However, your suggestion in Homebrew/brew#11885 is a good solution to this.