INCLUDEPY is incorrect

[ngoldbaum]

I'm experimenting with using a python-build-standalone python executable in CI on some projects like PyO3 and NumPy and am running into some issues.

The biggest one so far is that INCLUDEPY points to a non-existent path:

$  python -msysconfig | grep INCLUDEPY
	CONFINCLUDEPY = "/install/include/python3.13t"
	INCLUDEPY = "/install/include/python3.13t"

This makes it difficult to build C extensions against this version of Python, since it looks like libraries tend to check INCLUDEPY.

Interestingly, include is fine:

$ python -msysconfig | grep include
	include = "/Users/goldbaum/.local/share/uv/python/cpython-3.13.0+freethreaded-macos-aarch64-none/include/python3.13t"
	platinclude = "/Users/goldbaum/.local/share/uv/python/cpython-3.13.0+freethreaded-macos-aarch64-none/include/python3.13t"

I see that there is prior discussion about these /install paths being hard to update, although maybe since one path is updated we can update INCLUDEPY?

[ngoldbaum]

We could also update the libraries to get the include path from 'include', but it would be nice if libraries that introspect Python to try to ask questions like this didn't have to get updates to work with python-build-standalone builds.

[ngoldbaum]

PyO3 uses INCLUDEPY here: https://github.com/PyO3/pyo3/blob/face5936ac275d386096b44ab42d1bca4089ef34/pyo3-ffi-check/build.rs#L8

NumPy uses it here: https://github.com/numpy/numpy/blob/dc862b43e3bca09dcf5c235c46b2c7573e2e4b80/numpy/testing/_private/extbuild.py#L109

I don't know the history here or why this particular variable gets picked up so much.

[zanieb]

Thanks for the report. I wonder how include gets derived? It's not like the build knows it'll be unpacked there. I worry this is an upstream CPython behavior?

[zanieb]

Maybe relevant

• get_python_inc returns wrong include directory for bundled Pythons pypa/distutils#178

[ngoldbaum]

I stepped through the standard library in a debugger. It looks like sys.get_path('include') ends up going into _expand_vars here, which then substitutes the correct path from the templates in the _INSTALL_SCHEMES dictionary.

> /Users/goldbaum/.local/share/uv/python/cpython-3.13.0+freethreaded-macos-aarch64-none/lib/python3.13t/sysconfig/__init__.py(277)_expand_vars()
-> value = os.path.expanduser(value)
(Pdb) p _INSTALL_SCHEMES[scheme]['include']
'{installed_base}/include/{implementation_lower}{py_version_short}{abiflags}'
(Pdb) p _subst_vars('{installed_base}/include/{implementation_lower}{py_version_short}{abiflags}', vars)
'/Users/goldbaum/.local/share/uv/python/cpython-3.13.0+freethreaded-macos-aarch64-none/include/python3.13t'

INCLUDEPY on the other hand it looks like is directly written to the binary and comes from get_config_vars(). So it looks like this is the intended behavior and maybe the downstream libraries are doing the incorrect thing by checking INCLUDEPY?

I can certainly make a PyO3 patch that checks sysconfig.get_path('include') if INCLUDEPY isn't a valid path. I already did that for NumPy in numpy/numpy#27581.

[zanieb]

Thank you! That makes sense to me. The alternative is to change the INCLUDEPY value when the standalone distributions are installed — but I don't think we can do anything at build time.

[ngoldbaum]

If it can be changed at install time I think that would be best, there's a lot of code out there checking for INCLUDEPY: https://github.com/search?q=INCLUDEPY&type=code

[zanieb]

Related astral-sh/uv#7369

[zanieb]

Opened astral-sh/uv#8429 to track this

[charliermarsh]

If we rewrote INCLUDEPY at install time, would it be such that we point to an absolute path, referencing the destination to which we unpacked the Python?

[zanieb]

I think so, yes.

[charliermarsh]

This is now fixed at install-time in uv -- should we close out?

[zanieb]

I think we may want to make it easier to apply the patches that uv does outside of uv, but perhaps we should track that separately.