Cannot attach to long running process

[schooft]

I get the following error

$ sudo py-spy --pid 9134
Error: No such file or directory (os error 2)

strace shows that this happens at

open("/usr/bin/python3.5 (deleted)", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

The problem seems to be

$ sudo readlink /proc/9134/exe
/usr/bin/python3.5 (deleted)

/usr/bin/python3.5 does exist but was upgraded during the runtime of the process.

[aspin]

Any progress on this or ideas on what's going on? So far in the meantime I've been pip install py-spy==0.1.11, which I picked randomly since I haven't gotten this issue in the past.

[benfred]

@aspin : This bug is caused by the python executable being deleted while the program is run, making it so that we can't read in information from it. This bug will affect all versions of py-spy including 0.1.11.

Do you have a different problem? I'd be interested in hearing more about it if so

[aspin]

Hmm, I'm not sure the source of the problem then. It looks something like this for me:

$ /usr/local/bin/py-spy -V
py-spy 0.3.0
$ /usr/local/bin/py-spy top --pid 22159
Error: No such file or directory (os error 2)
$ /usr/local/bin/py-spy -V # after installing new version
py-spy 0.1.11
$ /usr/local/bin/py-spy top --pid 22159 # works

$ sudo readlink /proc/22159/exe
/usr/local/bin/python3.7
$ /usr/local/bin/python3.7
bash: /usr/local/bin/python3.7: No such file or directory

For a bit of a context, the process is running inside of a Docker container, but I still seem to be able to get its PID from ps aux. Any helpful information I can provide?

strace output (I'm not familiar with this):

$ strace /usr/local/bin/py-spy top --pid 22159
execve("/usr/local/bin/py-spy", ["/usr/local/bin/py-spy", "top", "--pid", "22159"], [/* 26 vars */]) = 0
mmap(NULL, 5384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fab8408e000
arch_prctl(ARCH_SET_FS, 0x7fab8408f3e0) = 0
set_tid_address(0x7fab8408f418)         = 832
rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER|SA_RESTART, 0x766273}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGSEGV, {0x750550, [], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x766273}, NULL, 8) = 0
rt_sigaction(SIGBUS, {0x750550, [], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x766273}, NULL, 8) = 0
sigaltstack(NULL, {ss_sp=0, ss_flags=SS_DISABLE, ss_size=0}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fab8408c000
sigaltstack({ss_sp=0x7fab8408c000, ss_flags=0, ss_size=8192}, NULL) = 0
brk(0)                                  = 0x1ac0000
brk(0x1ac1000)                          = 0x1ac1000
ioctl(2, TIOCGWINSZ, {ws_row=66, ws_col=272, ws_xpixel=0, ws_ypixel=0}) = 0
brk(0x1ac2000)                          = 0x1ac2000
brk(0x1ac3000)                          = 0x1ac3000
brk(0x1ac4000)                          = 0x1ac4000
brk(0x1ac5000)                          = 0x1ac5000
syscall_318(0x7fff809c8fc0, 0x10, 0x1, 0x8, 0x3, 0) = 0x10
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
mmap(NULL, 2109440, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fab83e89000
mprotect(0x7fab83e8a000, 2105344, PROT_READ|PROT_WRITE) = 0
clone(child_stack=0x7fab8408a6e8, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x7fab8408bb18, tls=0x7fab8408bae0, child_tidptr=0x7fab8408bb18) = 833
futex(0x7fab8408bbb8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fff809caef4, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "No such file or directory", 25No such file or directory) = 25
write(2, " (os error ", 11 (os error )             = 11
write(2, "2", 12)                        = 1
write(2, ")", 1))                        = 1
write(2, "\n", 1
)                       = 1
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=0, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7fab8408c000, 8192)            = 0
exit_group(1)                           = ?
+++ exited with 1 +++

[benfred]

@aspin Thanks for the info - I've managed to reproduce this. This is now being tracked in #199

[adamholm]

I'm encountering this same bug when using py-spy 0.3.2, but when attempting to record the pid of a python process started by Pylons web framework's paster command.

$ py-spy record -o profile.svg --pid 20627
Error: No such file or directory (os error 2)