Implement sysconfig-based location inference #9626

uranusjr · 2021-02-19T09:54:37Z

This detects location differences between distutils and sysconfig, and tell users to comment in #9617.

pypa-bot · 2021-02-19T09:54:39Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

BrownTruck · 2021-02-20T10:15:04Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

Module-level logic is bad.

This also helps catch bugs in the logic, which are also fixed in this commit.

uranusjr · 2021-02-23T13:14:01Z

Okay, I think I’ve caught all the pip-specific edge cases. The only remaining failures are all PyPy. Its purelib and platlib values differ:

distutils: {sys.prefix}/site-packages
sysconfig: {sys.prefix}/lib/pypy3.6/site-packages

and this is arguably exactly the kind of mismatches we want to catch and help implementations fix. Who should we reach out to discuss this with PyPy folks?

pradyunsg

This is really great and looks good to me. The only thought I have is we should probably add more "proper unit tests" for the newly introduced sysconfig-related code, instead of only patching the existing ones.

But, eh.

pradyunsg · 2021-02-23T13:22:43Z

/cc @mattip for help with PyPy's sysconfig vs distutils.sysconfig stuff. :)

mattip · 2021-02-23T13:51:14Z

Thanks for pinging me. Please test the latest PyPy3.6 and 3.7, which is available on github actions via the actions/setup-python@v2 action, this will also help you move yet another few builds off travis.

As for distutils.sysconfig vs. sysconfig: where is there an _INSTALL_SCHEME in distutils.sysconfig? PyPy intentionally sets up site-packages to be in a different location than CPython, to prevent confusion, so the _INSTALL_SCHEME in sysconfig is changed on purpose. Could you help me understand what is actually failing in the tests?

uranusjr · 2021-02-23T14:18:09Z

The distutils-based implementation uses distutils.command.install.INSTALL_SCHEME to get the site-packages directory.

I checked PyPy’s implementation, it seems like the mismatch happens because our sysconfig implementation does not use the pypy key in sysconfig._INSTALL_SCHEMES, but posix_home when the “home” scheme is requested. Should we use pypy here instead?

This also seems to surface another issue in sysconfig: There is no way or a tools to ask the interpreter what are their preferred scheme names. We have _get_default_scheme() for the “prefix” schemes, but no equivalents for “user” and “home”, so pip can only guess the scheme names when it needs “the current platform/implementation’s user scheme name” for pip install --user (and likewise for “home” on --target), and implementations like PyPy has no way to communicate to tools messages like “please use pypy instead of posix_home”. This should probably be raised in Discourse for CPython core devs to consider.

mattip · 2021-02-23T16:33:26Z

Hmm. It seems there are a few problems. One is that PyPy never bothered to sync up distutils.command.install with the changes it made to sysconfig. Another is the difference in schemes, that it never backported to CPython. But +- bugs, there should be no need for pip to adjust posix_name or posix_user, this should be provided by the sysconfig implementation, that each implementation can modify to fit its needs.

If we want a single source for sysconfig across implementations, I think the best course would be to modify sysconfig's scheme layout and add a implementation key, that could get its value from platform.python_implementation(). This would look something like what PyPy did. Note that this also impacts site.py since they copy out parts of sysconfig.

mattip · 2021-02-23T16:56:05Z

We have _get_default_scheme() for the “prefix” schemes ...

It seems that is a private interface, the public one is sysconfig.get_path()

mattip · 2021-02-23T17:42:00Z

Yeah, this is all very emperical work, and on top of what the implementations do, the OS distributors do their thing too. Anyhow, thanks for pointing out the distutils install command, I have already made a stab at fixing that in PyPy. Could you point me to the tests that discovered the discrepancy?

pradyunsg · 2021-02-23T18:33:04Z

I mean, part of why I want distutils out of the standard library is this stuff, but I’m glad that @uranusjr filed this PR, because this has always felt like a field of thorns - it touches legacy defacto stuff + new stuff and alternate Python implementations. :)

uranusjr · 2021-02-24T09:29:17Z

I’ve raised this in bpo-43312.

src/pip/_internal/locations/base.py

BrownTruck · 2021-02-26T11:15:05Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

pradyunsg · 2021-02-27T14:19:06Z

@uranusjr Let's get this up to date, mark the tests as xfail(strict=True) on PyPy, and merge? The longer we let this linger, the slower our progress would be here. :)

uranusjr · 2021-02-27T22:54:46Z

Let me try fixing PyPy instead. I’ll do the xfails if this does not work…

uranusjr · 2021-02-28T01:23:16Z

🔔 THIS WORKS 🔔

pradyunsg · 2021-02-28T07:21:53Z

AHAHAHAHA. Let's do this! :)

pradyunsg · 2021-04-02T00:12:15Z

FWIW, I'm seeing that this fails with GitHub's pypy3:

https://github.com/pradyunsg/pip/pull/9/checks?check_run_id=2250267986#step:6:4862

mattip · 2021-04-02T06:08:16Z

I don't see any failures there. Could you copy-paste the failing test or otherwise point out what is failing on pypy3 ?

pradyunsg · 2021-04-02T07:14:37Z

https://github.com/pradyunsg/pip/runs/2250246469?check_suite_focus=true

Whoops, I think what was the wrong tab's URL. Sorry!

mattip · 2021-04-02T08:53:39Z

If I understand the log correctly, it seems there is a mismatch between distutils and sysconfig?

WARNING: Value for scheme.platlib does not match.  \
    Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: <workspace>/venv/user/lib/python3.6/site-packages
sysconfig: <workspace>/venv/user/lib/pypy3.6/site-packages
 WARNING: Value for scheme.purelib does not match. \
    Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: <workspace>/venv/user/lib/python3.6/site-packages
sysconfig: <workspace>/venv/user/lib/pypy3.6/site-packages
WARNING: Additional context:
        user = True
        home = None
        root = None
        prefix = None

... and going over to issue #9617 I see you already reached the same conclusion :). I will continue there.

pypa-bot added the needs rebase or merge PR has conflicts with current master label Feb 19, 2021

uranusjr force-pushed the sysconfig branch 2 times, most recently from 755de34 to 9da8b15 Compare February 19, 2021 10:00

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Feb 19, 2021

BrownTruck added the needs rebase or merge PR has conflicts with current master label Feb 20, 2021

uranusjr added 7 commits February 23, 2021 01:57

Move distuitls location logic into subpackage

738e600

Implement sysconfig locations and warn on mismatch

7662c59

Abstract out get_python_lib() usages

edbda25

Provide issue URL to report

686734f

News

b4545a0

Blackify

08c2829

Type fixes

790bd02

uranusjr force-pushed the sysconfig branch from b6dd2a4 to 790bd02 Compare February 22, 2021 17:58

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Feb 22, 2021

uranusjr added 3 commits February 23, 2021 02:04

Goodness

7563a9c

User-site special case fixes

1e1289e

Split bin_user and bin_prefix implementations

b7068f6

Module-level logic is bad.

uranusjr force-pushed the sysconfig branch from 58b5026 to b7068f6 Compare February 22, 2021 20:29

uranusjr added 3 commits February 23, 2021 04:38

Better erroring

60a82e7

Stray name

0c4bd55

Split scheme inference functions for clarity

a0a3bde

This also helps catch bugs in the logic, which are also fixed in this commit.

pradyunsg approved these changes Feb 23, 2021

View reviewed changes

Additional logging to get context

e0d6028

uranusjr force-pushed the sysconfig branch from 33aa6dd to e0d6028 Compare February 24, 2021 09:53

uranusjr mentioned this pull request Feb 24, 2021

Migrate PyPy Jobs from Travis to GitHub Actions #9651

Closed

uranusjr commented Feb 25, 2021

View reviewed changes

src/pip/_internal/locations/base.py Outdated Show resolved Hide resolved

BrownTruck added the needs rebase or merge PR has conflicts with current master label Feb 26, 2021

uranusjr mentioned this pull request Feb 26, 2021

PIP doesn't read setup.cfg in UTF-8, which causes UnicodeDecodeError #8931

Closed

uranusjr added 2 commits February 28, 2021 06:43

Add a couple of special cases for PyPy

b4ce289

Merge branch 'master' into sysconfig

09513b8

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Feb 27, 2021

New style type hints

826234e

uranusjr force-pushed the sysconfig branch 2 times, most recently from 78fcbeb to a51faf4 Compare February 27, 2021 23:56

Special-case PyPy's lib location differences

6d018bf

uranusjr force-pushed the sysconfig branch from a51faf4 to 6d018bf Compare February 28, 2021 00:31

pradyunsg merged commit f4f664b into pypa:master Feb 28, 2021

pradyunsg added the type: maintenance Related to Development and Maintenance Processes label Feb 28, 2021

uranusjr deleted the sysconfig branch February 28, 2021 08:28

github-actions bot locked as resolved and limited conversation to collaborators Sep 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement sysconfig-based location inference #9626

Implement sysconfig-based location inference #9626

uranusjr commented Feb 19, 2021

pypa-bot commented Feb 19, 2021

BrownTruck commented Feb 20, 2021

uranusjr commented Feb 23, 2021 •

edited

Loading

pradyunsg left a comment

pradyunsg commented Feb 23, 2021 •

edited

Loading

mattip commented Feb 23, 2021

uranusjr commented Feb 23, 2021 •

edited

Loading

mattip commented Feb 23, 2021

mattip commented Feb 23, 2021

mattip commented Feb 23, 2021

pradyunsg commented Feb 23, 2021

uranusjr commented Feb 24, 2021

BrownTruck commented Feb 26, 2021

pradyunsg commented Feb 27, 2021

uranusjr commented Feb 27, 2021

uranusjr commented Feb 28, 2021

pradyunsg commented Feb 28, 2021

pradyunsg commented Apr 2, 2021

mattip commented Apr 2, 2021

pradyunsg commented Apr 2, 2021

mattip commented Apr 2, 2021

Implement sysconfig-based location inference #9626

Implement sysconfig-based location inference #9626

Conversation

uranusjr commented Feb 19, 2021

pypa-bot commented Feb 19, 2021

BrownTruck commented Feb 20, 2021

uranusjr commented Feb 23, 2021 • edited Loading

pradyunsg left a comment

Choose a reason for hiding this comment

pradyunsg commented Feb 23, 2021 • edited Loading

mattip commented Feb 23, 2021

uranusjr commented Feb 23, 2021 • edited Loading

mattip commented Feb 23, 2021

mattip commented Feb 23, 2021

mattip commented Feb 23, 2021

pradyunsg commented Feb 23, 2021

uranusjr commented Feb 24, 2021

BrownTruck commented Feb 26, 2021

pradyunsg commented Feb 27, 2021

uranusjr commented Feb 27, 2021

uranusjr commented Feb 28, 2021

pradyunsg commented Feb 28, 2021

pradyunsg commented Apr 2, 2021

mattip commented Apr 2, 2021

pradyunsg commented Apr 2, 2021

mattip commented Apr 2, 2021

uranusjr commented Feb 23, 2021 •

edited

Loading

pradyunsg commented Feb 23, 2021 •

edited

Loading

uranusjr commented Feb 23, 2021 •

edited

Loading