virtualenv, Failed to find python version

[werneta]

My apologies if this has been asked and answered before, I did look through open and closed issues but didn't see anything quite like it.

I'm trying to run py-spy v0.1.10 in a virtualenv on a Debian unstable system. I'm receiving an "Error: Failed to find python version from target process" message whenever I attempt to launch the profiler via py-spy -- python2.7 my_script.py. I enabled logging (RUST_LOG=info) and saw that the python2.7 is read by py-spy as /path/to/python2, without its version number. Looking in the virtualenv's bin directory, I saw that there is a python2.7 symlinked to python2. This is the opposite of my system's installed python, where python2 is symlinked to python2.7.

I see in

py-spy/src/python_spy.rs

Line 215 in 27d20df

let tokens: Vec<&str> = python[6..].split('.').collect();

that the version number can be collected by looking for a python executable name of the form /path/to/pythonN.M and picking off the N.M. Since the python executable presented to py-spy while the virtualenv is activated doesn't have this form, the version number detection fails.

I'm able to work around this by using virtualenv --always-copy venv_target_dir, which doesn't generate symlinks in the virtualenv.

[benfred]

Hmm - the check to get the version from the filename is really supposed to be a last ditch measure, we should almost certainly be able to get it from the BSS section of the python binary.

Can you enable logging and paste the output here (export RUST_LOG=info) ?

[marc1n]

I have similar problem with virtualenv on Ubuntu 18.10:

$ python --version
Python 2.7.15+
$ python -m virtualenv --version
16.0.0
$ python -m virtualenv /tmp/test
New python executable in /tmp/test/bin/python
Please make sure you remove any previous custom paths from your /home/marcinr/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
$ source /tmp/test/bin/activate
(test) $ python
Python 2.7.15+ (default, Oct  2 2018, 22:12:08) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os; os.getpid()
2638

On other terminal:

$ source /tmp/test/bin/activate
(test) $ pip install py-spy
...
Successfully installed py-spy-0.1.10
(test) $ py-spy --pid 2638
Error: Failed to find python version from target process

py-spy.log

[jackvreeken]

I'm having the same issue on Windows 10 with Python 3.7.2. It works fine with 3.6.8 and 3.7.1.

The big difference I can find in the logs is that for Python 3.6.8 I see Found libpython binary @ C:\test\venv368\Scripts\python36.dll, and for Python 3.7.2 I see no such line. So it seems that for 3.6.8 it finds the version string in the BSS section of libpython (which it located), but since it cannot locate libpython for 3.7.2, it cannot determine the version.

I haven't looked to much into why/when this changed, but looking at my 3.6 or 3.7.1 venv\Scripts folder I see a whole bunch of files (including python3.dll and python36.dll/python37.dll), but for 3.7.2 I only see the activate scripts, easy-install, pip, and python executables. No libpythons.

Creating a virtualenv for 3.7.2 with the virtualenv package instead of the built-in venv module does result in libpython dlls in the Scripts folder. And with that virtual environment py-spy works as it should.
I had a similar issue with running tox a few weeks ago ("virtualenv is not compatible with this system or executable"), which was also resolved by using virtualenv instead of venv.

For Windows this all is probably caused by python/cpython@1c3de54#diff-8721d1362294720bad1dbbbaa4cc06e3

tl;dr:
Python's venv module's behavior changed from 3.7.1 to 3.7.2. Things break.

EDIT:
Added likely commit causing breakage on Windows, and updated version numbers accordingly (apparently 3.7.1 also worked fine).

[Pike]

I have a (related?) problem on mac, with a self-built python and a version 2.7.15+. I guess the + is throwing it off? This is not even in a virtualenv, though.

[benfred]

So - I think there are at least 2 different issues here:

• We don't detect version strings like '2.7.15+' because of the '+'. This is the problem seen by @Pike and @marc1n. This is trivial to fix (done here 794f6f6) - but I'm really wondering where the + is coming from. @Pike how did you build this? I can build v2.7.16 or v2.7.15 from source on my mac (by checking out the appropiate tag from git), but didn't see that version string.

• Windows virtualenv problems with 3.7.2. Looks like the python.exe in the virtualenv creates a subprocess that is the actual python process - and we don't support profiling subprocesses (yet). Best way to workaround is probably to pass in the PID of the actual python process, rather than have py-spy launch the redirector script itself.

[Pike]

If you build the branch while it's not released, it's showing the +, https://github.com/python/cpython/commits/2.7/Include/patchlevel.h has the history of that.

Thanks for the fix.

[benfred]

Added a new issue to track profiling subprocess (#124). Closing this one.