How to find the correct value for home in pyvenv.cfg using documented APIs?

[konstin]

I'd like to create virtual environments and to do so, i need to fill in the value of home in pyvenv.cfg. All implementations I found use the undocumented, private sys._base_executable. Is there a platform-independent, ideally cross-interpreter way to determine the value of the python home for a new virtualenv using documented APIs?

cpython's venv modules (pypy contains the same module):

cpython/Lib/venv/__init__.py

Line 134 in e14930f

executable = sys._base_executable

virtualenv: https://github.com/pypa/virtualenv/blob/d9fdf48d69f0d0ca56140cf0381edbb5d6fe09f5/src/virtualenv/discovery/py_info.py#L136-L156

On linux/mac, using the parent of the resolved sys.executable seems to be sufficient, but specifically on windows it isn't clear what to use.

[pfmoore]

Turning that question around, what are you needing to do that you can't do by simply using the venv module? You say you want to create virtual environments, so "use venv" seems like an obvious answer...

(To be clear, what I'm saying is that we could enhance venv if needed, and that might be a better solution than making you put together the bits for yourself).

[konstin]

We're using a native reimplementation of virtualenv creation that are faster than the venv module and virtualenv and don't require a subprocess call / intergrate natively with the rest of the codebase. By caching interpreter metadata, we can run the whole venv creation process (bare venv, measured by hyperfine) in 4ms - 15ms depending on the host. We also found that a native reimplementation allows us to provides better errror messages. Linux/Mac support is working, but we're currently struggling with the correct windows implementation.

[pfmoore]

Cool. I don't think there's a documented API that does what sys._base_executable does. We could obviously make it documented (and I'd be OK with that, although maybe it would lock us into some choices we'd prefer not to be locked into - @zooba might have a view here, as I think he worked on the redirector-based implementation for Windows). Alternatively, or as well, I'd support splitting venv.EnvBuilder.ensure_directories into two parts, create_context that created the context object but did no filesystem changes, and ensure_directories that took a context object as input. That would give a supported way of getting all the information needed for a venv, without needing any OS calls. It would be 3.13+ only, of course, so you'd still need to rely on the undocumented _base_executable for 3.12 and earlier.

[zooba]

I think to make _base_executable public we'd need to ensure that it actually behaves the same across platforms. It should be closer than it was when introduced, but we just disabled a test for _base_executable because it wasn't reliable, which means there is more needed in order to make it something we would advertise.

If you know you're running in the base interpreter, sys.executable is fine. _base_executable is only really there for people who use a venv to create another venv. You might have another workaround for that in your system.

You could also consider implementing your venv with a different approach, such as .pth files or a custom python.c that passes additional directories at initialization. Right now, everything venv does is practically an internal implementation detail, and if you want to replace it at the "supported" level then all you need is to put an "activate" script in the same location that makes "python" launch the environment and you're "correct". Everything else is hacks.

[konstin]

If you know you're running in the base interpreter, sys.executable is fine. _base_executable is only really there for people who use a venv to create another venv. You might have another workaround for that in your system.

We unfortunately often have the case that we are already in a venv and then e.g. want to create a temporary build venv or because the user gave as a venv python.

You could also consider implementing your venv with a different approach, such as .pth files or a custom python.c that passes additional directories at initialization. Right now, everything venv does is practically an internal implementation detail, and if you want to replace it at the "supported" level then all you need is to put an "activate" script in the same location that makes "python" launch the environment and you're "correct". Everything else is hacks.

I've been looking at the virtualenv package which seems to rely on sys._base_executable to work at all.

Would it be possible to expose enough information to make this not a hack? I'm talking about standardizing an API to get the value of home, everything else is already there.

[zooba]

If you just want the value of home, then why not find and read pyvenv.cfg directly?

Alternatively, if you can dive into getpath and sort out why _base_executable isn't consistent across all platforms, we can expose that. But be warned that touching any of that code at all will break users (not even those attempting things like you're attempting), so it has to be done very carefully.

[konstin]

If you just want the value of home, then why not find and read pyvenv.cfg directly?

Sorry if i didn't phrase that clearly, but the intent is to have a general purpose (bare) venv creator that you can pass any python interpreter as base, be it a global one or a venv one. We cache a query to some interpreter metadata the first time we see an interpreter and can then create venvs quickly. We could create a venv once when we see a new interpreter and read the value of home, but isn't that as much an implementation detail as sys._base_executable?

I'd be happy not to change anything existing - after all python -m venv and virtualenv -p python work fine. It would be really useful though if they didn't depend on an undocumented private variable. If we were to e.g. add sys.home that is the parent of the base executable and can be used by all venv creators, this would be great for us and i think for the virtualenv devs, too. I also like @pfmoore's idea a lot of a create_context function in the venv module.

[zooba]

If you're passing the Python executable path as base, you should be able to easily look for an existing pyvenv.cfg and then just copy it directly into any new venvs you want to create with the same base. At least home ought to be identical. If there's no pyvenv.cfg, then use the executable path you were given by the user.

This is assuming you want a "compatible" venv, which would only be so that pip knows where to install stuff. If you don't care about pip, then it'll be way better to set PYTHONPATH and pass -S¹ to get venv-like isolation. My experience with a lot of production workloads has shown this is much more reliable than trying to use venv, but interactive/dev stuff is a bit more mixed.

Footnotes

1. Apparently we don't have an environment variable for no site? We do for no user site... maybe it's just not in the help message. ↩