Spawning child shell process while debugging via gdb causes hang

[d-bahr]

When debugging via gdb, spawning a child shell process via popen causes gdb to hang. The only way to fix it is to kill the gdb process.

This post exhibits the same problem I am having with a minimum repro (I am not the author of the post):
https://askubuntu.com/questions/927339/gdb-hangs-trying-to-open-proc-file-on-bash-on-windows-on-ubuntu-16-04-2-lts

The program hangs when popen is called.

Running the program outside of gdb works as expected. Also, running on a non-WSL linux instance (with or without gdb) works as expected.

Apologies if this issue is already documented. I didn't see it when I searched...

[fpqc]

I could have sworn @therealkenc mentioned this in another issue (where you have to send a SIGKILL to any child process of gdb from a separate terminal in order to kill it

[therealkenc]

That was #648 (message). I didn't chime in because I couldn't see the problem over there being popen() related, at least strictly speaking. Either way the repro on offer in the OP stands on its own merits. Good catch.

[aseering]

bump -- I finally found both time and tools to track down why I've been having trouble using GDB within WSL, for applications that I care about. Buried under a few layers of proprietary code, it turns out that it's this issue.

Here's a reproducer that uses system() rather than popen(): (Requires gcc; I always just install the build-essential package. Tested against WSL build 16273.)

parent.c:

#include <stdlib.h>

int main()
{
        system("./child");
        return 0;
}

child.c:

int main() { return 0; }

Build commands:

gcc parent.c -o parent
gcc child.c -o child

strace of parent:

$ strace -ff ./parent
execve("./parent", ["./parent"], [/* 22 vars */]) = 0
brk(NULL)                               = 0x1477000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c4fa20000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=81600, ...}) = 0
mmap(NULL, 81600, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3c4fa0c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c4f230000
mprotect(0x7f3c4f3f0000, 2097152, PROT_NONE) = 0
mmap(0x7f3c4f5f0000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7f3c4f5f0000
mmap(0x7f3c4f5f6000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c4f5f6000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c4fa00000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c4f9f0000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c4f9e0000
arch_prctl(ARCH_SET_FS, 0x7f3c4f9f0700) = 0
mprotect(0x7f3c4f5f0000, 16384, PROT_READ) = 0
mprotect(0x600000, 4096, PROT_READ)     = 0
mprotect(0x7f3c4f825000, 4096, PROT_READ) = 0
munmap(0x7f3c4fa0c000, 81600)           = 0
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f3c4f2654b0}, {SIG_DFL, [], SA_RESTORER, 0x7f53e5c254b0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f3c4f2654b0}, {SIG_DFL, [], SA_RESTORER, 0x7f53e5c254b0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
clone(strace: Process 2557 attached
child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffd81f977c) = 2557
[pid  2557] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f3c4f2654b0},  <unfinished ...>
[pid  2556] wait4(2557,  <unfinished ...>
[pid  2557] <... rt_sigaction resumed> NULL, 8) = 0
[pid  2557] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f3c4f2654b0}, NULL, 8) = 0
[pid  2557] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  2557] execve("/bin/sh", ["sh", "-c", "./child"], [/* 22 vars */]) = 0
[pid  2557] brk(NULL)                   = 0x7fffe99ad000
[pid  2557] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[pid  2557] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2f2930000
[pid  2557] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid  2557] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid  2557] fstat(3, {st_mode=S_IFREG|0644, st_size=81600, ...}) = 0
[pid  2557] mmap(NULL, 81600, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff2f291c000
[pid  2557] close(3)                    = 0
[pid  2557] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[pid  2557] open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid  2557] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
[pid  2557] fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
[pid  2557] mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff2f2230000
[pid  2557] mprotect(0x7ff2f23f0000, 2097152, PROT_NONE) = 0
[pid  2557] mmap(0x7ff2f25f0000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7ff2f25f0000
[pid  2557] mmap(0x7ff2f25f6000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff2f25f6000
[pid  2557] close(3)                    = 0
[pid  2557] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2f2910000
[pid  2557] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2f2900000
[pid  2557] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2f28f0000
[pid  2557] arch_prctl(ARCH_SET_FS, 0x7ff2f2900700) = 0
[pid  2557] mprotect(0x7ff2f25f0000, 16384, PROT_READ) = 0
[pid  2557] mprotect(0x7ff2f2c23000, 8192, PROT_READ) = 0
[pid  2557] mprotect(0x7ff2f2825000, 4096, PROT_READ) = 0
[pid  2557] munmap(0x7ff2f291c000, 81600) = 0
[pid  2557] getuid()                    = 1000
[pid  2557] getgid()                    = 1000
[pid  2557] getpid()                    = 2557
[pid  2557] rt_sigaction(SIGCHLD, {0x7ff2f2a17540, ~[RTMIN RT_1], SA_RESTORER, 0x7ff2f22654b0}, NULL, 8) = 0
[pid  2557] geteuid()                   = 1000
[pid  2557] brk(NULL)                   = 0x7fffe99ad000
[pid  2557] brk(0x7fffe99ce000)         = 0x7fffe99ce000
[pid  2557] getppid()                   = 2556
[pid  2557] stat("/home/aseering/test", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
[pid  2557] stat(".", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
[pid  2557] geteuid()                   = 1000
[pid  2557] getegid()                   = 1000
[pid  2557] rt_sigaction(SIGINT, NULL, {SIG_DFL, [], SA_RESTORER, 0x7f3c4f2654b0}, 8) = 0
[pid  2557] rt_sigaction(SIGINT, {0x7ff2f2a17540, ~[RTMIN RT_1], SA_RESTORER, 0x7ff2f22654b0}, NULL, 8) = 0
[pid  2557] rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], SA_RESTORER, 0x7f3c4f2654b0}, 8) = 0
[pid  2557] rt_sigaction(SIGQUIT, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7ff2f22654b0}, NULL, 8) = 0
[pid  2557] rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], SA_RESTORER, 0x7f53e5c254b0}, 8) = 0
[pid  2557] rt_sigaction(SIGTERM, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7ff2f22654b0}, NULL, 8) = 0
[pid  2557] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ff2f29009d0) = 2558
strace: Process 2558 attached
[pid  2557] wait4(-1,  <unfinished ...>
[pid  2558] execve("./child", ["./child"], [/* 22 vars */]) = 0
[pid  2558] brk(NULL)                   = 0xd14000
[pid  2558] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[pid  2558] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8b6b660000
[pid  2558] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid  2558] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid  2558] fstat(3, {st_mode=S_IFREG|0644, st_size=81600, ...}) = 0
[pid  2558] mmap(NULL, 81600, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8b6b64c000
[pid  2558] close(3)                    = 0
[pid  2558] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[pid  2558] open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid  2558] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
[pid  2558] fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
[pid  2558] mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8b6b030000
[pid  2558] mprotect(0x7f8b6b1f0000, 2097152, PROT_NONE) = 0
[pid  2558] mmap(0x7f8b6b3f0000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7f8b6b3f0000
[pid  2558] mmap(0x7f8b6b3f6000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8b6b3f6000
[pid  2558] close(3)                    = 0
[pid  2558] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8b6b640000
[pid  2558] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8b6b630000
[pid  2558] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8b6b620000
[pid  2558] arch_prctl(ARCH_SET_FS, 0x7f8b6b630700) = 0
[pid  2558] mprotect(0x7f8b6b3f0000, 16384, PROT_READ) = 0
[pid  2558] mprotect(0x600000, 4096, PROT_READ) = 0
[pid  2558] mprotect(0x7f8b6b625000, 4096, PROT_READ) = 0
[pid  2558] munmap(0x7f8b6b64c000, 81600) = 0
[pid  2558] exit_group(0)               = ?
[pid  2558] +++ exited with 0 +++
[pid  2557] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2558
[pid  2557] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2558, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid  2557] rt_sigreturn({mask=[]})     = 2558
[pid  2557] exit_group(0)               = ?
[pid  2557] +++ exited with 0 +++
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2557
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f3c4f2654b0}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f3c4f2654b0}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2557, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
exit_group(0)                           = ?
+++ exited with 0 +++

gdb of parent:

$ gdb ./parent -ex run
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./parent...(no debugging symbols found)...done.
Starting program: /home/aseering/test/parent
## GDB hangs at this point.  Can't interrupt with ctrl-C; can't kill; can only kill -9

As it turns out, this issue no longer affects me as directly. I'm working on different software these days and not hitting this issue most of the time. (I care more about I/O performance, but y'all are well aware of that issue :-) A more-elegant Docker story would also be nice.) I'm also not sure that this is the only issue with gdb; it's just the first one, I can't get past it to see what's underneath.

@benhillis FYI -- I'm pretty sure I promised to report this several months ago; apologies for being so lame in getting it out the door...

[therealkenc]

That's a dead ringer for #648 May7 (message) mentioned just above. Thanks for doing a tight repro.

[sunilmut]

Thanks @aseering for the simple repro steps. I am marking this as a bug. We will try to prioritize this.

[sunilmut]

This should be fixed in the Insider build 17017.