runtime: TestGdbBacktrace failures due to GDB "internal-error: wait returned unexpected status 0x0"

[bcmills]

2020-12-07T21:01:46-7ad6596/linux-mips64le-mengzhuo
2020-10-23T15:11:15-646531c/linux-mips64le-mengzhuo

--- FAIL: TestGdbBacktrace (1.51s)
    runtime-gdb_test.go:77: gdb version 8.1
    runtime-gdb_test.go:437: gdb output:
        Loading Go Runtime support.
        Breakpoint 1 at 0x7ece0: file /tmp/farm/tmp/go-build405377235/main.go, line 17.
        [New LWP 18663]
        [New LWP 18664]
        [New LWP 18665]
        [New LWP 18666]
        
        Thread 1 "a.exe" hit Breakpoint 1, 0x000000000007ece0 in main.eee (~r0=<optimized out>) at /tmp/farm/tmp/go-build405377235/main.go:17
        17	func eee() bool { return true }
        #0  0x000000000007ece0 in main.eee (~r0=<optimized out>) at /tmp/farm/tmp/go-build405377235/main.go:17
        #1  0x000000000007ecbc in main.ddd (~r0=<optimized out>) at /tmp/farm/tmp/go-build405377235/main.go:14
        #2  0x000000000007ec5c in main.ccc (~r0=<optimized out>) at /tmp/farm/tmp/go-build405377235/main.go:11
        #3  0x000000000007ec0c in main.bbb (~r0=<optimized out>) at /tmp/farm/tmp/go-build405377235/main.go:8
        #4  0x000000000007ebbc in main.aaa (~r0=<optimized out>) at /tmp/farm/tmp/go-build405377235/main.go:5
        #5  0x000000000007ed1c in main.main () at /tmp/farm/tmp/go-build405377235/main.go:22
        ../../gdb/linux-nat.c:2081: internal-error: wait returned unexpected status 0x0
        A problem internal to GDB has been detected,
        further debugging may prove unreliable.
        Quit this debugging session? (y or n) [answered Y; input not from terminal]
        
        This is a bug, please report it.  For instructions, see:
        <http://www.gnu.org/software/gdb/bugs/>.
        
        ../../gdb/linux-nat.c:2081: internal-error: wait returned unexpected status 0x0
        A problem internal to GDB has been detected,
        further debugging may prove unreliable.
        Create a core file of GDB? (y or n) [answered Y; input not from terminal]
    runtime-gdb_test.go:439: gdb exited with error: signal: aborted (core dumped)
FAIL
FAIL	runtime	44.687s

See previously #37405 (TestGdbBacktrace hanging on the same builder), #39228 (TestGdbBacktrace failures on Linux), #39204 (meta-bug about runtime GDB test flakiness).

CC @mengzhuo @ianlancetaylor

[mengzhuo]

https://sourceware.org/bugzilla/show_bug.cgi?id=20301

Can finally reproduce after a few tries.

What happens is that another thread calls exit() while I "next" this thread.
Most of the time gdb handles it nicely, but it also happens that I get the internal-error immediately:

[bcmills]

Looks like the same underlying GDB bug on linux-riscv64-jsing:

2021-05-10T19:19:34-dc50683/linux-riscv64-jsing

--- FAIL: TestGdbBacktrace (6.72s)
    runtime-gdb_test.go:76: gdb version 9.2
    runtime-gdb_test.go:428: gdb output:
        Loading Go Runtime support.
        Breakpoint 1 at 0x719a8: file /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go, line 17.
        [New LWP 2737319]
        [New LWP 2737320]
        [New LWP 2737321]
        
        Thread 1 "a.exe" hit Breakpoint 1, main.eee (~r0=<optimized out>) at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:17
        17	func eee() bool { return true }
        #0  main.eee (~r0=<optimized out>) at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:17
        #1  0x0000000000071994 in main.ddd (~r0=<optimized out>) at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:14
        #2  0x0000000000071954 in main.ccc (~r0=<optimized out>) at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:11
        #3  0x000000000007191c in main.bbb (~r0=<optimized out>) at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:8
        #4  0x00000000000718e4 in main.aaa (~r0=<optimized out>) at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:5
        #5  0x00000000000719dc in main.main () at /home/gopher/build/tmp/TestGdbBacktrace2306548485/001/main.go:21
        /build/gdb-D4eJJR/gdb-9.2/gdb/linux-nat.c:1963: internal-error: wait returned unexpected status 0x0
        A problem internal to GDB has been detected,
        further debugging may prove unreliable.
        Quit this debugging session? (y or n) [answered Y; input not from terminal]
        
        This is a bug, please report it.  For instructions, see:
        <http://www.gnu.org/software/gdb/bugs/>.
        
    runtime-gdb_test.go:430: gdb exited with error: signal: aborted
FAIL
FAIL	runtime	294.989s

CC @4a6f656c

[bcmills]

greplogs --dashboard -md -l -e 'FAIL: TestGdbBacktrace .*(?:\n .*)*: wait returned unexpected status' --since=2021-05-12

2022-01-12T00:01:48-8070e70/linux-riscv64-jsing
2021-10-26T22:05:53-80be4a4/linux-riscv64-unmatched

[bcmills]

greplogs --dashboard -md -l -e 'FAIL: TestGdbBacktrace .*(?:\n .*)*: wait returned unexpected status' --since=2022-01-27

2022-02-07T21:57:29-911c78f/linux-386-longtest

[bcmills]

Note that the most recent failure is on linux/386, which is a first-class port.

Given how little time is left in the Go 1.18 release cycle, marking as release-blocker for Go 1.19.

[bcmills]

Given that the failure is within GDB itself, we could resolve this issue by doing one or more of the following:

• Getting an upstream fix in GDB and updating the builders to pull it in.
• Identifying affected GDB versions and skipping the test on those versions.
• Modifying the test to skip itself if it detects this failure mode.
• Determining that the scenario under test cannot work reliably and removing the test for it.

[bcmills]

There are at least two upstream bugs reported against GDB for this symptom:
https://sourceware.org/bugzilla/show_bug.cgi?id=24628
https://sourceware.org/bugzilla/show_bug.cgi?id=28551

[gopherbot]

Change https://go.dev/cl/384234 mentions this issue: runtime: skip TestGdbBacktrace flakes matching a known GDB internal error

[bcmills]

Modifying the test to skip itself if it detects this failure mode.

Actually, we can do that one now, and we should: otherwise, this test may flake for Go users when they run go test all in their own module.

[bcmills]

@gopherbot, please backport to Go 1.17. This test still fails intermittently on the release branch, and the patch to skip for that failure mode is small and test-only.

[gopherbot]

Backport issue(s) opened: #53049 (for 1.17).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/408054 mentions this issue: [release-branch.go1.17] runtime: skip TestGdbBacktrace flakes matching a known GDB internal error