Incorrect sysconfig in uv-managed Python installation

[akhilles]

uv platform: macOS arm64
uv version: 0.4.9 (77d278f68 2024-09-10)

The sysconfig module is used by pyo3 (and other tools) to discover information about the Python distribution. For example, https://pyo3.rs/v0.14.2/building_and_distribution.html?highlight=pyo3_python#configuring-the-python-version:

Once the Python interpreter is located, pyo3-build-config executes it to query the information in the sysconfig module which is needed to configure the rest of the compilation.

In a system-managed Python installation, LIBDIR in sysconfig correctly points to the installation's lib dir:

$ uv venv --python-preference only-system
$ source .venv/bin/activate
$ python3 -m sysconfig | rg "\sLIBDIR ="
        LIBDIR = "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/lib"

In a uv-managed Python installation, LIBDIR is set to a non-existent directory:

$ uv venv --python-preference managed
$ source .venv/bin/activate
$ python3 -m sysconfig | rg "\sLIBDIR ="
        LIBDIR = "/install/lib"

It should probably point to something like $HOME/Library/Application Support/uv/python/cpython-3.12.3-macos-aarch64-none/install/lib.

The incorrect LIBDIR causes some issues when using pyo3 + cargo test in a uv virtual environment. I can work around it by messing around with PYTHONPATH, but it would be good to have consistent behavior across system-managed and uv-managed Python installations.

Related: https://github.com/bluss/sysconfigpatcher

[bluss]

Relatively simple patching go a long way and fixes many packages (as a start, patch CC and LIBDIR), but there is no obvious or perfect solution

Missing knowledge:

• Windows aspects of this issue?
• In which cases does the sysconfigpatcher approach not work, it's not sufficient? It apparently helps in many simple cases, but what "coverage" does it give?
• It depends on how the package works. Numpy doesn't need anything like this, it seems to build from source with uv's managed pythons anyway (it depends on Cython which does not build, but that's a different package)

Because there is no perfect solution, it's probably good go an experimental path:

1. The first step of the experimental approach is already done, sysconfigpatcher is that step - users can use that external tool - or other implementations of the same idea - and report their results
2. Could uv introduce opt-in patching of its managed python installs?
3. User feedback on the uv patching feature guides development - is different patching needed, is the method viable, should it be continued, should it be enabled by default

[bluss]

This proposed PEP is a little bit relevant for this issue

• https://peps.python.org/pep-0739/
• https://discuss.python.org/t/pep-739-static-description-file-for-build-details-of-python-installations/44572

[zanieb]

Going to track improvements to this in #8429