Python dev headers missing in venv #9347

arogozhnikov · 2024-11-22T05:41:20Z

I think python dev headers are now missing when uv installs venv; repro (start from ubuntu 22.04):

confirmed on uv==0.5.0 and 0.5.4, not tested on other versions

export $VIRTUAL_ENV=/tmp/.venv
curl -LsSf https://astral.sh/uv/0.5.0/install.sh | sh
. $HOME/.local/bin/env
uv venv --python 3.10 $VIRTUAL_ENV 
. $VIRTUAL_ENV/bin/activate
uv pip install ihm

output:

[stderr]
src/ihm_format_wrap_1.7.c:203:11: fatal error: Python.h: No such file or directory
  203 | # include <Python.h>
      |           ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  Caused by: This error likely indicates that you need to install a library that provides "Python.h" for [email protected]

Annoyingly, it used to work - I was able to install this package, but a couple of weeks ago binaries shipped with uv were swapped for some other builds, this affected even previous versions of uv.

The text was updated successfully, but these errors were encountered:

samypr100 · 2024-11-23T02:06:28Z

Similar to #8966 (comment)

TL;DR Try sysconfigpatcher to adjust the uv python downloads to have the proper pkgconfig and sysconfig metadata for your system.

zanieb · 2024-12-02T22:30:33Z

but a couple of weeks ago binaries shipped with uv were swapped for some other builds, this affected even previous versions of uv.

This doesn't quite make sense, we didn't change previous versions of uv and the Python distributions we use are hard-coded for each release and cannot change.

zanieb · 2024-12-02T22:38:24Z

If you're using a system interpreter, you should install the headers, e.g., the following works:

docker run --platform linux/x86_64 -it ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install python3-dev && uv venv && uv pip install ihm"

I can't reproduce this failure using a managed interpreter:

docker run --platform linux/x86_64 -it ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang && uv venv -p 3.10 && uv pip install ihm"

arogozhnikov · 2024-12-04T03:46:34Z

this works:

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.10 && uv pip install ihm"

this doesn't

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 && uv pip install ihm"

wjakob · 2024-12-08T09:26:56Z

I am also having this issue, and using Docker is not a feasible workaround. Could UV not be modified so that installs the Python headers when creating a venv?

samypr100 · 2024-12-08T14:32:34Z

👋 @wjakob, the docker examples here are to provide an MRE, you can apply the same steps outside of docker.

wjakob · 2024-12-08T14:35:30Z

This is on a machine where I don't have root access, hence installing the headers via apt is not an option (also, that would be for the OS Python version, which is often quite outdated). With uv not installing them as part of the virtual environment, it is impossible to build C extensions within the environment.

zanieb · 2024-12-08T16:14:45Z

@arogozhnikov your failing example uses the system interpreter (3.11 is installed in the image already)

root@c0d1cea75e52:/# uv python list
cpython-3.13.0+freethreaded-linux-x86_64-gnu    <download available>
cpython-3.12.7-linux-x86_64-gnu                 <download available>
cpython-3.11.10-linux-x86_64-gnu                <download available>
cpython-3.11.2-linux-x86_64-gnu                 /usr/bin/python3.11
cpython-3.11.2-linux-x86_64-gnu                 /usr/bin/python3 -> python3.11
cpython-3.11.2-linux-x86_64-gnu                 /bin/python3.11
cpython-3.11.2-linux-x86_64-gnu                 /bin/python3 -> python3.11
cpython-3.10.15-linux-x86_64-gnu                <download available>
cpython-3.9.20-linux-x86_64-gnu                 <download available>
cpython-3.8.20-linux-x86_64-gnu                 <download available>
cpython-3.7.9-linux-x86_64-gnu                  <download available>
pypy-3.10.14-linux-x86_64-gnu                   <download available>
pypy-3.9.19-linux-x86_64-gnu                    <download available>
pypy-3.8.16-linux-x86_64-gnu                    <download available>
pypy-3.7.13-linux-x86_64-gnu                    <download available>

That's why you need to install development headers via the system package manager.

If you force uv to use a managed interpreter with --python-preference only-manage, the example works

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 --python-preference only-managed && uv pip install ihm"

zanieb · 2024-12-08T16:15:30Z

@wjakob please share a reproducible example

samypr100 · 2024-12-08T17:27:17Z

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 --python-preference only-managed && uv pip install ihm"

Thanks @zanieb, I've been incorrectly under assumption OP wanted to use gcc and not clang, hence my suggestions to use sysconfigpatcher.

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "uv venv -p 3.11.10 && uv tool install git+https://github.com/bluss/sysconfigpatcher && uv tool run sysconfigpatcher /root/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu && uv pip install ihm"

Do you think it's worth expanding in docs when is it reasonable to attempt to use sysconfigpatcher by adding a note to python-versions.md or maybe a troubleshooting faq.

zanieb · 2024-12-08T22:07:52Z

Do you think it's worth expanding in docs when is it reasonable to attempt to use sysconfigpatcher by adding a note to python-versions.md or maybe a troubleshooting faq.

It's sort of a priority for me to understand the problems people are encountering and fix it here rather than recommend sysconfigpatcher — it's nice that it exists and I'm sure we'll learn from it but I'd prefer a first-party solution.

Tristan971 · 2024-12-11T18:49:37Z

please share a reproducible example

Here's one:

FROM fedora:41

RUN dnf update -y && \
    dnf group install -y 'development-tools' && \
    dnf install -y \
      gcc \
      gcc-c++ \
      gtk3 \
      gtk3-devel \
      libxcrypt-compat \
      python3 \
      python3-devel \
      which

RUN curl -LsSf https://astral.sh/uv/install.sh | sh

Run the resulting image then:

# just ensuring we don't use the system-managed one which is 3.13
[root@0a8793bee0ed ~]# uv init --python ">=3.10,<3.11" repro
[root@0a8793bee0ed ~]# cd repro
[root@0a8793bee0ed repro]# uv add 'wxpython>=4.0.0'
# ... takes a minute then fails with...
...
      The configuration failed
      (complete log in /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/config.log)

Then we go and look at the detailed logs

[root@0a8793bee0ed repro]# cat /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/config.log
# ... bunch of noise until ...
['/usr/bin/g++', '-g', '-fwrapv', '-O3', '-fPIC', '-fno-strict-aliasing', '-I/install/include/python3.10', '-DPYTHONDIR="/root/.cache/uv/builds-v0/.tmpn5kGVI/lib/python3.10/site-packages"', '-DPYTHONARCHDIR="/root/.cache/uv/builds-v0/.tmpn5kGVI/lib/python3.10/site-packages"', '-DNDEBUG', '../test.cpp', '-c', '-o/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/.conf_check_ef50035bd89e2895b752adc35a4fe169/testbuild/test.cpp.1.o']
err: ../test.cpp:2:10: fatal error: Python.h: No such file or directory
    2 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.

from /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src: Test does not build: Traceback (most recent call last):
  File "/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/bin/.waf3-2.1.2-5b48554f897113218781486acfe4d78e/waflib/Configure.py", line 337, in run_build
    bld.compile()
  File "/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/bin/.waf3-2.1.2-5b48554f897113218781486acfe4d78e/waflib/Build.py", line 176, in compile
    raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
 -> task in 'testprog' failed with exit status 1 (run with -v to display more information)

Broken python installation? Get python-config now!
from /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src: The configuration failed

I can confirm that using sysconfigpatcher fixes this issue (the wxpython build takes a bit of an eternity, but it eventually does succeed)

[root@0a8793bee0ed repro]# uv tool install 'git+https://github.com/bluss/sysconfigpatcher'
 Updated https://github.com/bluss/sysconfigpatcher (c1ebf8a)
Resolved 1 package in 0.49ms
   Built sysconfigpatcher @ git+https://github.com/bluss/sysconfigpatcher@c1ebf8ab9274dcde255484d93ce0f1fd1f76a248
Prepared 1 package in 322ms
Installed 1 package in 0.99ms
 + sysconfigpatcher==0.3.1 (from git+https://github.com/bluss/sysconfigpatcher@c1ebf8ab9274dcde255484d93ce0f1fd1f76a248)
Installed 1 executable: sysconfigpatcher
[root@0a8793bee0ed repro]# realpath .venv/bin/python
/root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/bin/python3.10
[root@0a8793bee0ed repro]# sysconfigpatcher /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/bin/python3.10
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/_sysconfigdata__linux_x86_64-linux-gnu.py
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/pkgconfig/python-3.10-embed.pc
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/pkgconfig/python-3.10.pc
[root@0a8793bee0ed repro]# uv add 'wxpython>=4.0.0'
Resolved 4 packages in 36ms
   Built wxpython==4.2.2
Prepared 1 package in 2m 58s
Installed 3 packages in 29ms
 + numpy==2.2.0
 + six==1.17.0
 + wxpython==4.2.2

Also worth noting that UV's manager isn't the only one that leads to these kinds of issues; back then I experienced a similar (but not the same exact) issue with pyenv+poetry, linked here https://gitlab.com/ita1024/waf/-/issues/2420

zanieb · 2024-12-11T19:16:09Z

Thanks @Tristan971, I believe that one is a manifestation of astral-sh/python-build-standalone#374 which we're also tracking in #8429

The workaround for this is:

export CPPFLAGS="-I/root/.local/share/uv/python/cpython-3.10.16-linux-aarch64-gnu/include/python3.10"
export LDFLAGS="-L/root/.local/share/uv/python/cpython-3.10.16-linux-aarch64-gnu/lib"

(with the proper path to the Python you're using)

As you said it's taking forever to build so I can't confirm at this second, but it's definitely past that early failure.

Tristan971 · 2024-12-11T19:19:13Z

Great to see it’s already being looked at! :-)

charliermarsh · 2024-12-17T13:27:17Z

Gonna review this today to see if it's resolved with latest changes.

samypr100 · 2024-12-17T13:49:26Z

If it's any help, this would still fail

> docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:0.5.9-bookworm /bin/bash -c "uv venv -p 3.11.10 && uv pip install ihm"
Using CPython 3.11.10
Creating virtual environment at: .venv
Resolved 2 packages in 330ms
  x Failed to download and build `ihm==1.8`
  `-> Build backend failed to build wheel through `build_wheel` (exit status: 1)

      [stdout]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/__init__.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/dataset.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/representation.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/analysis.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/report.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/metadata.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/flr.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/cross_linkers.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/reader.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/restraint.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/source.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/startmodel.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/format.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/model.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/test.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/geometry.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/citations.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/format_bcif.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/protocol.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/multi_state_scheme.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/dictionary.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/location.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/dumper.py -> build/lib.linux-x86_64-cpython-311/ihm
      copying ihm/reference.py -> build/lib.linux-x86_64-cpython-311/ihm
      creating build/lib.linux-x86_64-cpython-311/ihm/util
      copying ihm/util/__init__.py -> build/lib.linux-x86_64-cpython-311/ihm/util
      copying ihm/util/make_mmcif.py -> build/lib.linux-x86_64-cpython-311/ihm/util
      running build_ext
      building 'ihm._format' extension
      creating build/temp.linux-x86_64-cpython-311/src
      clang -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3
      -Wall -fPIC -fPIC -Isrc -I/root/.cache/uv/builds-v0/.tmpOq7G05/include
      -I/root/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/include/python3.11 -c src/ihm_format.c -o
      build/temp.linux-x86_64-cpython-311/src/ihm_format.o

      [stderr]
      error: command 'clang' failed: No such file or directory

wjakob · 2024-12-17T13:52:01Z

@samypr100 You need to have a sane build setup (C++ dev environment) in your container. This is not related to the main issue, which was about the presence of Python headers.

samypr100 · 2024-12-17T16:30:24Z

@samypr100 You need to have a sane build setup (C++ dev environment) in your container. This is not related to the main issue, which was about the presence of Python headers.

Thanks, note this is a buildpack-deps based image so it has a sane build setup.

zanieb · 2024-12-17T16:50:39Z

I presume the point is that we'd need #9916 for that example to work?

charliermarsh · 2024-12-17T17:06:39Z

Yeah for me that's separate from this issue (though a good point from @samypr100).

charliermarsh · 2024-12-17T18:41:03Z

Confirmed that this works on latest (with CC=gcc).

samypr100 · 2024-12-18T01:38:39Z

My last example now also works as of 0.5.10 release without the need for CC=gcc 🥳

> docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:0.5.10-bookworm /bin/bash -c "uv venv -p 3.11.10 && uv pip install ihm"
Using CPython 3.11.10
Creating virtual environment at: .venv
Resolved 2 packages in 406ms
   Built ihm==1.8
Prepared 2 packages in 2.72s
Installed 2 packages in 2ms
 + ihm==1.8
 + msgpack==1.1.0

arogozhnikov mentioned this issue Nov 22, 2024

use python3.10 shipped with ubuntu22.04 chaidiscovery/chai-lab#169

Merged

samypr100 added the question Asking for clarification or support label Nov 23, 2024

zanieb mentioned this issue Dec 11, 2024

Improve sysconfig metadata when installing managed Python distributions #8429

Closed

charliermarsh self-assigned this Dec 17, 2024

charliermarsh closed this as completed Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python dev headers missing in venv #9347

Python dev headers missing in venv #9347

arogozhnikov commented Nov 22, 2024

samypr100 commented Nov 23, 2024

zanieb commented Dec 2, 2024

zanieb commented Dec 2, 2024 •

edited

Loading

arogozhnikov commented Dec 4, 2024

wjakob commented Dec 8, 2024

samypr100 commented Dec 8, 2024 •

edited

Loading

wjakob commented Dec 8, 2024

zanieb commented Dec 8, 2024

zanieb commented Dec 8, 2024

samypr100 commented Dec 8, 2024 •

edited

Loading

zanieb commented Dec 8, 2024

Tristan971 commented Dec 11, 2024 •

edited

Loading

zanieb commented Dec 11, 2024 •

edited

Loading

Tristan971 commented Dec 11, 2024

charliermarsh commented Dec 17, 2024

samypr100 commented Dec 17, 2024

wjakob commented Dec 17, 2024

samypr100 commented Dec 17, 2024

zanieb commented Dec 17, 2024

charliermarsh commented Dec 17, 2024

charliermarsh commented Dec 17, 2024

samypr100 commented Dec 18, 2024

Python dev headers missing in venv #9347

Python dev headers missing in venv #9347

Comments

arogozhnikov commented Nov 22, 2024

samypr100 commented Nov 23, 2024

zanieb commented Dec 2, 2024

zanieb commented Dec 2, 2024 • edited Loading

arogozhnikov commented Dec 4, 2024

wjakob commented Dec 8, 2024

samypr100 commented Dec 8, 2024 • edited Loading

wjakob commented Dec 8, 2024

zanieb commented Dec 8, 2024

zanieb commented Dec 8, 2024

samypr100 commented Dec 8, 2024 • edited Loading

zanieb commented Dec 8, 2024

Tristan971 commented Dec 11, 2024 • edited Loading

zanieb commented Dec 11, 2024 • edited Loading

Tristan971 commented Dec 11, 2024

charliermarsh commented Dec 17, 2024

samypr100 commented Dec 17, 2024

wjakob commented Dec 17, 2024

samypr100 commented Dec 17, 2024

zanieb commented Dec 17, 2024

charliermarsh commented Dec 17, 2024

charliermarsh commented Dec 17, 2024

samypr100 commented Dec 18, 2024

zanieb commented Dec 2, 2024 •

edited

Loading

samypr100 commented Dec 8, 2024 •

edited

Loading

samypr100 commented Dec 8, 2024 •

edited

Loading

Tristan971 commented Dec 11, 2024 •

edited

Loading

zanieb commented Dec 11, 2024 •

edited

Loading