Issue with dynamic linkage, how to find what env is in use?

[the-moog]

I have an issue where I am writing C extensions to Python and calling Python from C.

If I run Python, it works fine within the env, but the C applications that embed the same Python code do not. I thought this was because the venv was compiled as static, so I enabled --enable-shared - see (gist). But even after doing that it still does not work.

I can now see this is because the apps lack a shim, and that is required to set/unset LD_LIBRARY_PATH.

But, managing a lot of apps, some user created using libraries we provide, across a lot of installs on thousands of systems will be problematic.
We have to use pyenv as (for complicated reasons) we have to support multiple python versions concurrently. We still support products that are 15+ years old at the same time as ones that don't oficially exist yet.

I started looking at setting LD_LIBRARY_PATH dynamically, from within the app, so that it detects if it's in a pyenv and choose the correct path before loading the matching version of the real application as an overlay.

My question is, what is a reliable, portable, low cost way for a compiled app to detect 1: if the current tree is within a venv, 2: if pyenv is installed and 3: if the required Python version is present in the library of versions, before 4: installing the required version itself.

• search for .python-version up the tree? - Issue: That would miss activated virtualenvs elsewhere (which is likely)
• look for the VIRTUAL_ENV variable - Issue: I am I correct in my observation that pyenv does not set this?
• Search the entire (user accessible) file system and guess the best one?
• have the app write a shim for itself if one does not exist before it loads properly?
• other better, more official existing or non-existing ideas?

Prerequisite

• [✓] Make sure your problem is not listed in the common build problems.
• [✓] Make sure no duplicated issue has already been reported in the pyenv issues. You should look for closed issues, too.
• [?] Make sure you are not asking us to help solving your specific issue.
  • GitHub issues is opened mainly for development purposes. If you want to ask someone to help solving your problem, go to some community site like Gitter, StackOverflow, etc.
• [✓] Make sure your problem is not derived from packaging (e.g. Homebrew).
  • Please refer to the package documentation for the installation issues, etc.
• [✓] Make sure your problem is not derived from plugins.
  • This repository is maintaining pyenv and the default python-build plugin only. Please refrain from reporting issues of other plugins here.

Description

• [Ubuntu 16..04 (there is a good reason for this)] Platform information (e.g. Ubuntu Linux 16.04):
• [Intel Atom] OS architecture (e.g. amd64):
• [git/HEAD] pyenv version:
• [Many] Python version:
• [5.4.0] C Compiler information (e.g. gcc 7.3):
• [n/a] Please attach the debug trace of the failing command as a gist:
  • Run env PYENV_DEBUG=1 <faulty command> 2>&1 | tee trace.log and attach trace.log. E.g. if you have a problem with installing Python, run env PYENV_DEBUG=1 pyenv install -v <version> 2>&1 | tee trace.log (note the -v option to pyenv install).
• [n/a] If you have a problem with installing Python, please also attach config.log from the build directory
  • The build directory is reported after the "BUILD FAILED" message and is usually under /tmp.
• [n/a] If the build succeeds but the problem is still with the build process (e.g. the resulting Python is missing a feature), please attach
  • the debug trace from reinstalling the faulty version with env PYENV_DEBUG=1 pyenv install -f -k -v <version> 2>&1 | tee trace.log
• [n/a] * config.log from the build directory. When using pyenv install with -k as per above, the build directory will be under $PYENV_ROOT/sources.

[native-api]

1: if the current tree is within a venv,

Venvs/Virtualenvs work by customizing the Python initialization step.
So the easiest way should be to run the Python there and inspect its internal state for specific tells. See https://stackoverflow.com/questions/1871549/determine-if-python-is-running-inside-virtualenv

For setting up embedded CPython, also take a look at the embedding API and specifically Python path configuration to let the embedded interpreter find stuff.

When a version is compiled with --enable-shared (now the default), Pyenv sets rpath in the executables to let them find libpython.

2: if pyenv is installed and

pyenv in PATH. pyenv root shows the configuration root, installed alt versions reside under $(pyenv root)/versions.

3: if the required Python version is present in the library of versions, before

pyenv versions --bare

4: installing the required version itself.

pyenv install

[the-moog]

Compiling the LD library path into the exe will mean that if that app is later run outside the pyenv, and assuming the required python version is installed at a system level then, it will look for the .so in the wrong place.

Perhaps it would make sense to put all pyenv compiled shared libraries together in the same folder, then either pyenv or the user can (temorarily) set LD_LIBRARY_PATH (not great) or the user can put e.g. ~/.pyenv/python_all/lib into ld.so.conf or the app get a system wide python assuming it exists.
Setting LD_LIBRARY_PATH in the shell is not a great idea as it might break Python apps that exist in the current pyenv but are not linked with the current pyenvs Python version. Some sort of way to prefix the app at runtime would be better.

1: if the current tree is within a venv,

Venvs/Virtualenvs work by customizing the Python initialization step. So the easiest way should be to run the Python there
and inspect its internal state for specific tells.
See https://stackoverflow.com/questions/1871549/determine-if-python-is-running-inside-virtualenv

LOL! - I had already provided a new answer to that very question on SE

Those answers are what I expected. But that means spawning pyenv xxx yyy and text parsing results. I was thinking perhaps there was a 'libpyenv' that we could link with?

Btw: Somehow, on our pyenv installs --enable-shared is certainly not the default.
The instructions I gave to devs say use the git checkout as per instructions on the github site.
It was one of our engineers not being able to run some code (that I wrote) that required the .so that made me start looking into this. Hence why I wrote the hook script to attempt to turn that option on. That means perhaps there is a bug of some sort there?

I've raised an Issue #2649 that can be used to track a future PR on converting the hook script into a plugin as I've not got time to get into that right now.

[native-api]

Compiling the LD library path into the exe will mean that if that app is later run outside the pyenv, and assuming the required python version is installed at a system level then, it will look for the .so in the wrong place.
Perhaps it would make sense to put all pyenv compiled shared libraries together in the same folder, then either pyenv or the user can (temorarily) set LD_LIBRARY_PATH (not great) or the user can put e.g. ~/.pyenv/python_all/lib into ld.so.conf or the app get a system wide python assuming it exists.

Pyenv installs alternate versions of software. Which means, any shared libraries it uses cannot be in the default module loading path lest they override system ones.
So you either tell the linker statically to look for the module at a specific path, via rpath, or do it dynamically, via an envvar or e.g. an explicit path to dlopen().

Setting LD_LIBRARY_PATH in the shell is not a great idea as it might break Python apps that exist in the current pyenv but are not linked with the current pyenvs Python version. Some sort of way to prefix the app at runtime would be better.

You can unset it right after loading. In this light, using an explicit path is probably a better idea.

Btw: Somehow, on our pyenv installs --enable-shared is certainly not the default.

Create a separate issue and provide the necessary diagnostic info for us to be able to say anything specific. Currently, I can only conjecture that you're passing --disable-shared or are using an old version of Pyenv.

[native-api]

Those answers are what I expected. But that means spawning pyenv xxx yyy and text parsing results. I was thinking perhaps there was a 'libpyenv' that we could link with?

Pyenv is designed to be able to provide machine-readable output for scripting.

[native-api]

Those answers are what I expected. But that means spawning pyenv xxx yyy and text parsing results. I was thinking perhaps there was a 'libpyenv' that we could link with?

Pyenv is designed to be able to provide machine-readable output for scripting.

Pyenv is Bash-based. Thus even if we provided a C library, it would be doing exactly the same under the hood. So what's the point?

[the-moog]

Indeed, and I would agree bash powerful and a great shell, but sometimes the standard Linux command set + those embedded in bash are not enough when it comes to specific tasks. Yes it can do it, but it's not easy. There are many 'standard' tools that do essentially the same thing as a few extra lines in bash but that does not mean they should be deprecated. I would argue that not everybody in the world is a bash wizzard and that can it appear pyenv is lacking. For every one bash wizzard the are proably 100 Python programmers. Or to put it another way, by the same argument, if you have Python why do you need bash?

[native-api]

I think we have strayed off topic.

IIRC the task at hand was to link to the libpython.so that belongs to the currently-selected Pyenv-based version.

AFAICS, the most realistic way is to link with an explicit path:

pp="$(pyenv prefix)"
p="${pp%%:*}"    #if multiple versions are selected, `pyenv prefix' returns a colon-separated list
libpath="$(echo $p/lib/libpython*.*.so)"
<load $libpath>

I gave the logic in Bash, but it can be replicated in any language. It seems pretty trivial to me.
Does it really warrant a library? Remember, you'll have to locate that library in a similar way first, too, to be able to load it.

[the-moog]

Give me a while to take a look at your suggestion as I am not working on this right now.

[the-moog]

I also am working on something else right now. The issues are with python-config, this tells the toolchain where to find the components of the current python. If a venv is active it seems to get it wrong. I'm still trying to work out how to make a CtoPython linked C program work all the time regardless of what env is active when a build is done and when it's used subsequently. Just reading about Ermine but that is commercial so I'd have to go through our legal team to consider it. It looks like a more lightweight Flatpack? I wonder if there is a more barebones method? Ermine is a tricky name to Google alternates for..... Given only one .so is needed, really I just need to divert dlopen for that file to a mmap file. Of course I am also playing with pyenv building for just static, but it seems not to always work.

[native-api]

Some reproduction steps would certainly help (a sample program and what you're doing).
Currently, we don't have information to say anything specific beyond what is already said.

[the-moog]

Yep, I hear you. I am not able to work on the project being affected by pyenv right now as other work has pushed it to one side. I just found STATIFIER so mentioning it here so when I do get back onto this code I will not have forgotten about it. That may sidestep the issue completely - or will it. It says GPL which means if I statically link to any part of it my companies code has to be GPL, I don't think that will go down well..... For now you are right to close this one as I am unable to provide the detail needed.

[the-moog]

also
https://github.com/marook/statifier
https://github.com/tobimensch/preloadify
Now this is going somewhere......

[dmgolembiowski]

For anyone else in 2024 on linux or something similar, who wants to use an older Python to do something like swift repl, but you see:

$ swift repl

error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

Pyenv is a way forward. Without disrupting the Python system packages, one possible solution is
to install pyenv, update the dotfile(s), run pyenv install 3.9.
After that, assuming this is the only python version you have installed, navigate to a particular directory where the 3.9 version's shared lib is needed. Run pyenv local 3.9.19 (or whatever your edition is). Close your shell. Re-open it and navigate to that directory. Then add/use a script in that directory called source-me.sh which has the contents:

#!/usr/bin/env bash

echo "Attempting to update the LD_LIBRARY_PATH to $(pyenv prefix)/lib"
pp=$(pyenv prefix)
echo "Note that \`pyenv prefix\` returns multiple colon-separated values if multiple versions are installed"
p="${pp%%:*}"

export LD_LIBRARY_PATH="$p/lib:$LD_LIBRARY_PATH"

Courtesy of @native-api

and execute your shell's specific way of sourcing something new into it. In my case, I'm using zshell which operates similarly to bash. You next need to invoke:

source source-me.sh

Then you ought to be able to run swift repl to see:

Welcome to Swift version 5.10.1 (swift-5.10.1-RELEASE).
Type :help for assistance.
  1> 

Cheers!