installing 'datrie' fails during compile (but pip succeeds)

[samimia-swks]

uv 0.4.11 on Ubuntu 22.04

I can install datrie (a dependency of snakemake) by using pip in a pyenv managed python 3.9.6 or 3.12.3 with no issues. pypi is missing wheels for python >3.8 but the compile with gcc succeeds creating datrie-0.8.2-cp39-cp39-linux_x86_64.whl. See attached pip logs.

if I try the same with uv pip install datrie==0.8.2 --verbose I get error: command 'clang' failed: No such file or directory
So somehow uv seems to want to compile this with clang even though I have gcc and pip can use that. See attached uv logs.

If I do CC=$(which gcc) CXX=$(which g++) uv pip install it does use gcc but I get another error command '/tools/llvm/bin/llvm-ar' failed: No such file or directory further down the line.

The only thing that works is to install llvm and clang and then symlink llvm-ar from where it is installed to this seemingly hard-coded path '/tools/llvm/bin/llvm-ar'. This seems pretty hacky....

Now clearly this is mostly the fault of the datrie package, which is seemingly abandoned despite being a dependency of the very active and popular snakemake package, but my question is why does pip succeed where uv fails?

pip_datrie.log
uv_datrie.log
uv_datrie_gcc.log

[zanieb]

Interestingly, when I use pip, it builds with clang not gcc. Are you using the same Python interpreter with uv pip? It looks like it's using 3.12 but the pip logs are using 3.9.

[zanieb]

Can you reproduce this in CI or a container or something?

[samimia-swks]

pip was using a pyenv installed python 3.9.6, which I think compiles python from source.
uv was using uv installed python (which I think are pre-compiled). my original logs were with python 3.12 but here they are with 3.9.6. it's the same thing.
uv_datrie.log

I have also attached logs for both pip and uv after installing clang and llvm (apt install clang llvm). it seems like pip still chooses to use gcc, but so does uv ?
pip_datrie_with_clang_installed.log
uv_datrie_with_clang_installed.log

I don't have a CI or container setup unfortunately. But my colleague reproduced this issue on his own Ubuntu machine, if that counts for something. pip works for both of us.

[zanieb]

Are you still using the uv-managed Python? We build the Python distribution itself with clang so maybe that's the difference? You can pass --python <path> to make sure you're using the same pyenv interpreter to see if that makes a difference.

I don't have a CI or container setup unfortunately. But my colleague reproduced this issue on his own Ubuntu machine, if that counts for something.

We'll need some sort of minimal reproduction to investigate for real.

[samimia-swks]

Thanks for the quick response. See the logs below for an attempt at a minimal reproduction.
There are two python versions here:

• one is installed by 'pyenv install 3.9.6' which compiles from source with gcc (~/.pyenv/versions/3.9.6/bin/python)

• and another with 'uv python install 3.9.6' which as you said is a pre-compiled clang install (~/.local/share/uv/python/cpython-3.9.6-linux-x86_64-gnu/bin/python3 )

As you see in the log:

• 'pip install' installs datrie successfully for the pyenv python with both CC=gcc and CC=clang.
• 'uv pip install' fails with both CC=gcc and CC=clang.
• 'uv pip install --python ~/.pyenv/versions/3.9.6/bin/python' succeeds with both CC=gcc and CC=clang.

In short, the problem appears related to the python installed by uv.

log.txt

[samimia-swks]

@zanieb, any luck with reproducing this? Arman Samimi

…

Sent from my phone On Sep 18, 2024, at 7:09 PM, Zanie Blue ***@***.***> wrote:  CAUTION: The email below is from an external source. Use caution before opening attachments or clicking links. Are you still using the uv-managed Python? We build the Python distribution itself with clang so maybe that's the difference? You can pass --python <path> to make sure you're using the same pyenv interpreter to see if that makes a difference. I don't have a CI or container setup unfortunately. But my colleague reproduced this issue on his own Ubuntu machine, if that counts for something. We'll need some sort of minimal reproduction to investigate for real. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/astral-sh/uv/issues/7525*issuecomment-2359835189__;Iw!!MyQQGECaxY11k7S_!cVOVpZT4oCVUzE8kylbixoMEO26qzEmt9JrWRmOFAPbapWSkb2InXCXCgsxtyQDbJKc1aAOouE_1X1gM7L6oG2bDI6VrQXs$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A66EOSLEC2OFYYIKDLF6IUTZXIW5FAVCNFSM6AAAAABOOX23ZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJZHAZTKMJYHE__;!!MyQQGECaxY11k7S_!cVOVpZT4oCVUzE8kylbixoMEO26qzEmt9JrWRmOFAPbapWSkb2InXCXCgsxtyQDbJKc1aAOouE_1X1gM7L6oG2bDEfy1uLo$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[zanieb]

Hm. I thought a wrote a reply but it's not here :/

In brief, I think you're running into a quirk of the Python distribution (which is built with clang): https://github.com/indygreg/python-build-standalone/blob/main/docs/quirks.rst#references-to-build-time-paths

I tried to dig into why this specific path (/tools/llvm) is in the distribution but couldn't figure that out. Maybe because I was looking at an aarch64 build though.

[bluss]

Sysconfig contains this for a typical indygreg build (typical that I've looked at / linux):

AR = "/tools/llvm/bin/llvm-ar"

AR is one of the variables that sysconfigpatcher patches. It might be possible to try that tool and see if it helps the build (but no guarantees, it's empirical only.)

If you want to inspect those variables, here's the way to do it:

uv run python -m sysconfig

[zanieb]

Thanks! I wonder if we can/should reset some of these at build time.

[harryeslick]

I've run into the same problem installing datrie using rye.
I managed to get it working by adding the following env variable:

export AR=/usr/bin/ar

Following the discussions here:
numpy/numpy#17777 (comment)

[samimia-swks]

@zanieb @bluss your suggestion to use sysconfigpatcher did the trick! Maybe uv install should have a --patch-sysconfig option? it is an imperfect solution clearly, but doing nothing means that most packages with C extentions will fail to install if uv continues to use pre-compiled python builds.

[bluss]

@samimia-swks sysconfigpatcher is experimental for sure, glad it helps, and I hope its experiments can be adopted if they are deemed safe, see my hopeful thoughts at bluss/sysconfigpatcher#1 and #7369. I wonder if the uv team could say what kind of solution they would prefer and in that way open for the community to maybe offer help (pull requests).

[zanieb]

I'm not sure what the ideal solution is for us. We have the opportunity to make improvements at build time (i.e. stripping these from the distributions) as well as at install time (i.e. adding current platform relevant information). I think we have a fair bit of prototyping to do (and I'm happy to engage with community exploration) before we can choose a solution.

[zanieb]

Let's track this in #8429