Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use posix_prefix for Apple's system Python #10256

Merged

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Aug 1, 2021

For #10151 (comment)

This adds additional logic to NOT use Apple’s osx_framework_library for pip install --prefix, matching distutils’s behaviour. Using a framework scheme under a custom prefix doesn’t really work, so this makes sense.

cc @stiewie33 for testing.

@uranusjr uranusjr added this to the 21.2.3 milestone Aug 1, 2021
@stiewie33
Copy link

Thanks! I'd love to test :) haven't done this before though, how do I upgrade my pip to this branch?

@uranusjr
Copy link
Member Author

uranusjr commented Aug 1, 2021

Try

pip install https://github.com/uranusjr/pip/archive/refs/heads/locations-osx-framework-not-default-prefix.zip

@stiewie33
Copy link

Cool, I've installed it, tested by un/reinstalling the Meraki python package but it's still coming up:

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8/site-packages
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages
WARNING: Value for scheme.purelib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8/site-packages
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8/include/UNKNOWN
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/UNKNOWN
WARNING: Value for scheme.scripts does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /usr/local/bin
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin
WARNING: Value for scheme.data does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Defaulting to user installation because normal site-packages is not writeable
Collecting meraki
  Using cached meraki-1.10.0-py3-none-any.whl (406 kB)
Requirement already satisfied: requests in ./Library/Python/3.8/lib/python/site-packages (from meraki) (2.26.0)
Requirement already satisfied: aiohttp in ./Library/Python/3.8/lib/python/site-packages (from meraki) (3.7.4.post0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (3.0.1)
Requirement already satisfied: attrs>=17.3.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (21.2.0)
Requirement already satisfied: typing-extensions>=3.6.5 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (3.10.0.0)
Requirement already satisfied: multidict<7.0,>=4.5 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (5.1.0)
Requirement already satisfied: yarl<2.0,>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (1.6.3)
Requirement already satisfied: chardet<5.0,>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (4.0.0)
Requirement already satisfied: idna>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from yarl<2.0,>=1.0->aiohttp->meraki) (3.2)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (1.26.6)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (2021.5.30)
Installing collected packages: meraki
Successfully installed meraki-1.10.0
➜  ~ pip3 debug
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 21.3.dev0 from /Users/stefan.coetzer/Library/Python/3.8/lib/python/site-packages/pip (python 3.8)
sys.version: 3.8.2 (default, Jun  8 2021, 11:59:35)
[Clang 12.0.5 (clang-1205.0.22.11)]
sys.executable: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: darwin
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /Users/stefan.coetzer/Library/Python/3.8/lib/python/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.4
  CacheControl==0.12.6
  colorama==0.4.4
  distlib==0.3.2
  distro==1.5.0 (Unable to locate actual module version, using vendor.txt specified version)
  html5lib==1.1
  msgpack==1.0.2 (Unable to locate actual module version, using vendor.txt specified version)
  packaging==21.0
  pep517==0.11.0
  progress==1.5
  pyparsing==2.4.7
  requests==2.26.0
  certifi==2021.05.30
  chardet==4.0.0
  idna==3.2
  urllib3==1.26.6
  resolvelib==0.7.1
  setuptools==44.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.16.0
  tenacity==8.0.1 (Unable to locate actual module version, using vendor.txt specified version)
  tomli==1.0.3
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 1607
  cp38-cp38-macosx_11_0_x86_64
  cp38-cp38-macosx_11_0_intel
  cp38-cp38-macosx_11_0_fat64
  cp38-cp38-macosx_11_0_fat32
  cp38-cp38-macosx_11_0_universal2
  cp38-cp38-macosx_11_0_universal
  cp38-cp38-macosx_10_16_x86_64
  cp38-cp38-macosx_10_16_intel
  cp38-cp38-macosx_10_16_fat64
  cp38-cp38-macosx_10_16_fat32
  ...
  [First 10 tags shown. Pass --verbose to show all.]

I'm available if you want to do live testing via Skype/Zoom

@uranusjr
Copy link
Member Author

uranusjr commented Aug 1, 2021

Hmm OkK, so we made progress (the warning is different) but are not quite there. Thanks, I'll continue investigating.

Copy link
Member Author

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stupid mistake

src/pip/_internal/locations/_sysconfig.py Outdated Show resolved Hide resolved
@uranusjr
Copy link
Member Author

uranusjr commented Aug 1, 2021

Caused by a stupid mistake 🤦 Could you install the latest change? (same command)

@stiewie33
Copy link

Nice catch, here's the output now:

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8/site-packages
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages
WARNING: Value for scheme.purelib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8/site-packages
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8/include/UNKNOWN
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/UNKNOWN
WARNING: Value for scheme.scripts does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /usr/local/bin
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin
WARNING: Value for scheme.data does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
distutils: /Library/Python/3.8
sysconfig: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Defaulting to user installation because normal site-packages is not writeable
Collecting meraki
  Using cached meraki-1.10.0-py3-none-any.whl (406 kB)
Requirement already satisfied: requests in ./Library/Python/3.8/lib/python/site-packages (from meraki) (2.26.0)
Requirement already satisfied: aiohttp in ./Library/Python/3.8/lib/python/site-packages (from meraki) (3.7.4.post0)
Requirement already satisfied: chardet<5.0,>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (4.0.0)
Requirement already satisfied: attrs>=17.3.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (21.2.0)
Requirement already satisfied: typing-extensions>=3.6.5 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (3.10.0.0)
Requirement already satisfied: multidict<7.0,>=4.5 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (5.1.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (3.0.1)
Requirement already satisfied: yarl<2.0,>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (1.6.3)
Requirement already satisfied: idna>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from yarl<2.0,>=1.0->aiohttp->meraki) (3.2)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (1.26.6)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (2021.5.30)
Installing collected packages: meraki
Successfully installed meraki-1.10.0
➜  ~ pip3 debug
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 21.3.dev0 from /Users/stefan.coetzer/Library/Python/3.8/lib/python/site-packages/pip (python 3.8)
sys.version: 3.8.2 (default, Jun  8 2021, 11:59:35)
[Clang 12.0.5 (clang-1205.0.22.11)]
sys.executable: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: darwin
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /Users/stefan.coetzer/Library/Python/3.8/lib/python/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.4
  CacheControl==0.12.6
  colorama==0.4.4
  distlib==0.3.2
  distro==1.5.0 (Unable to locate actual module version, using vendor.txt specified version)
  html5lib==1.1
  msgpack==1.0.2 (Unable to locate actual module version, using vendor.txt specified version)
  packaging==21.0
  pep517==0.11.0
  progress==1.5
  pyparsing==2.4.7
  requests==2.26.0
  certifi==2021.05.30
  chardet==4.0.0
  idna==3.2
  urllib3==1.26.6
  resolvelib==0.7.1
  setuptools==44.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.16.0
  tenacity==8.0.1 (Unable to locate actual module version, using vendor.txt specified version)
  tomli==1.0.3
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 1607
  cp38-cp38-macosx_11_0_x86_64
  cp38-cp38-macosx_11_0_intel
  cp38-cp38-macosx_11_0_fat64
  cp38-cp38-macosx_11_0_fat32
  cp38-cp38-macosx_11_0_universal2
  cp38-cp38-macosx_11_0_universal
  cp38-cp38-macosx_10_16_x86_64
  cp38-cp38-macosx_10_16_intel
  cp38-cp38-macosx_10_16_fat64
  cp38-cp38-macosx_10_16_fat32
  ...
  [First 10 tags shown. Pass --verbose to show all.]
➜  ~```

@uranusjr
Copy link
Member Author

uranusjr commented Aug 1, 2021

Hmm, the output didn’t change? That doesn’t sound right. Could you try adding --force-reinstall when applying the fix? Updating a package without changing the URL can be a bit wonky (since it breaks all sorts of HTTP caching assumptions).

@stiewie33
Copy link

It looks like that worked!

@stiewie33
Copy link

➜  ~ pip3 install meraki
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
Collecting meraki
  Using cached meraki-1.10.0-py3-none-any.whl (406 kB)
Requirement already satisfied: aiohttp in ./Library/Python/3.8/lib/python/site-packages (from meraki) (3.7.4.post0)
Requirement already satisfied: requests in ./Library/Python/3.8/lib/python/site-packages (from meraki) (2.26.0)
Requirement already satisfied: typing-extensions>=3.6.5 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (3.10.0.0)
Requirement already satisfied: attrs>=17.3.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (21.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (5.1.0)
Requirement already satisfied: chardet<5.0,>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (4.0.0)
Requirement already satisfied: yarl<2.0,>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (1.6.3)
Requirement already satisfied: async-timeout<4.0,>=3.0 in ./Library/Python/3.8/lib/python/site-packages (from aiohttp->meraki) (3.0.1)
Requirement already satisfied: idna>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from yarl<2.0,>=1.0->aiohttp->meraki) (3.2)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (2021.5.30)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./Library/Python/3.8/lib/python/site-packages (from requests->meraki) (1.26.6)
Installing collected packages: meraki
Successfully installed meraki-1.10.0```

@uranusjr uranusjr force-pushed the locations-osx-framework-not-default-prefix branch from d9bf6e1 to 051de2f Compare August 1, 2021 19:15
Copy link
Contributor

@SylvainDe SylvainDe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclaimer: I have a very poor understanding of the logic here. I just happened to be looking at it a few hours ago for a completely unrelated problem. Hence my suggestions may be completely irrelevant: feel free to ignore.

@uranusjr
Copy link
Member Author

uranusjr commented Aug 2, 2021

@SylvainDe I’d still be happy to know what you think! Although it seems you didn’t post any comments?

Copy link
Contributor

@SylvainDe SylvainDe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Sorry for the inconvenience, my initial comments were not sent for some reason)

@@ -43,8 +69,7 @@ def _infer_prefix() -> str:
"""
if _PREFERRED_SCHEME_API:
return _PREFERRED_SCHEME_API("prefix")
os_framework_global = is_osx_framework() and not running_under_virtualenv()
if os_framework_global and "osx_framework_library" in _AVAILABLE_SCHEMES:
if _should_use_osx_framework_prefix():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the feeling that this somehow hides the:

if val1 in _AVAILABLE_SCHEMES:
    return val1
if val2 in _AVAILABLE_SCHEMES:
    return val2

In fact, I'd even be tempted to rewrite the code in such a way that the check if performed in a single place.

    def yield_candidates():  # or a better name :)
        if is_osx_framework() and not running_under_virtualenv():  # or _should_use_osx_framework_prefix()
            yield "osx_framework_library"
        yield f"{sys.implementation.name}_{os.name}"
        yield sys.implementation.name
        yield f"{os.name}_prefix"
        yield os.name  # On Windows, prefx is just called "nt".
    for val in yield_candidates:
        if val in _AVAILABLE_SCHEMES:
            return val
    return "posix_prefix"

This is probably going out of scope for a code review comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The choice to not always call _AVAILABLE_SCHEMES is intentional; those that aren’t checked are from upstream CPython, and a sysconfig without those values is seriously broken, so it’s better to panic-abort than silently trying something else IMO.

# Special case: When installing into a custom prefix, use posix_prefix
# instead of osx_framework_library. See _should_use_osx_framework_prefix()
# docstring for details.
if prefix is not None and scheme_name == "osx_framework_library":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If my understanding is correct, this can only correspond to the situation where we've gone into the scheme_name = _infer_prefix() case and that particular function is only called from here.

Hence, I wonder if it'd make sense to have that particular piece of logic directly in the _infer_prefix rather than as step performed afterward.

This could correspond to doing something like:

  • adding a param to the _infer_prefix function about the prefix being provided or not
  • writing return "osx_framework_library" if provided_prefix is None else "posix_prefix" in the _infer_prefix function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intention behind the current approach is to make those_infer_* functions act similarly to sysconfig.get_preferred_scheme (because eventually we’re going to replace the ad-hoc inference with it). If we put the logic in _infer_prefix, when we remove _infer_user and _infer_home, we’ll still be left with one short _infer_prefix function. I don’t think either structure is better though, this is just how I mentally structure this logic (first I ask the macOS Python what its preferred scheme is, and if it says osx_framework_library, oh I’m going to ignore it).

@uranusjr uranusjr merged commit 8ed2178 into pypa:main Aug 6, 2021
@uranusjr uranusjr deleted the locations-osx-framework-not-default-prefix branch August 6, 2021 09:56
@uranusjr uranusjr mentioned this pull request Aug 6, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants