runtime: Go 1.14/Windows asynchronous preemption mechanism likely incompatible with debugging

[aarzilli]

If I understand correctly asynchronous preemption is implemented in Windows by:

1. Calling SuspendThread on the target thread
2. Waiting for SuspendThread to take effect by calling GetThreadContext
3. Injecting a function call on the target thread by manipulating the context and calling SetThreadContext
4. Calling ResumeThread on the target thread

This procedure being implemented by preemptM in src/runtime/os_windows.go.

However, from what I can observe, it seems that GetThreadContext will also return when the target thread get suspended after hitting a software breakpoint (placed by a debugger). This has two effects: first the breakpoint will be missed (because preemptM will manipulate the thread context to insert a function call, masking the breakpoint), secondly the return address for the injected call will be in the middle of an instruction (because the software breakpoint was partially overwriting an instruction).

A debugger that knows about this can work around it by checking if a thread is stopped on the entry point of asyncPreempt, and fix its return address. For this to work however preemptM needs to fully finish its context manipulation, if it ends up being stopped between the point where GetThreadContext returns and SetThreadContext is called the same problem will happen.

I'm not sure that there is a way to use software breakpoints with Go 1.14 with async preemption, as it currently is. Did I misunderstand anything?

[ALTree]

cc @aclements

[aclements]

Can you explain a bit more about how software breakpoints work under Windows (or point me to the relevant APIs)? I thought this would all be fine because SuspendThread acts like a semaphore, but I keep finding more and more little surprises with SuspendThread. :P

One possible, slightly awful work around may be for the debugging to poke a 1 into runtime.debug.asyncpreemptoff when it attaches. (Or if it's starting the process itself, add asyncpreemptoff=1 to GODEBUG.) Obviously not ideal because it may change program behavior.

[aarzilli]

Debuggers call WaitForDebugEvent, when a breakpoint happens (of any kind) the thread that executed the breakpoint is stopped and WaitForDebugEvent returns. Apparently this is also enough for any GetThreadContext call to return. A software breakpoint is just an INT 3 instruction overwriting the first byte of some other instruction, since executing it also updates the value of RIP the context returned by GetThreadContext will have RIP one byte inside whatever instruction was overwritten with the breakpoint.

One only-works-with-delve solution would be to add this between the SuspendThread call and the GetThreadContext call in preemptM:

if debuggerAttached {
    INT 3
}

with debuggerAttached being some global variable that the debugger is supposed to set after attaching. This should prevent (if I'm not wrong) preemptM from seeing the target thread's context before the debugger has a chance to and prevent preemptM from seeing the thread in a weird state.

The only other change needed in the debugger would be to ignore this breakpoint inside preemptM.

Of course this won't solve the problem for any debugger other than delve (also: I haven't tested this, so it's possible that it doesn't work at all).

[aclements]

Just checking my understanding: is the problem ultimately a race between two threads trying to change a third thread's context? Say, the debugger injects a breakpoint into the process. Thread D in the debugger blocks on WaitForDebugEvent. Thread A in the application hits the breakpoint, which suspends thread A (and all other threads) and unblocks thread D. Simultaneously, thread P in the runtime attempts to preempt thread A by also suspending it. But both thread D and P need to manipulate thread A's context: thread D to back up RIP by 1, and thread P to inject an asyncPreempt call.

It's tricky to come up with a specific sequence of steps that leads to this problem because WaitForDebugEvent suspends the whole process when any thread hits a debug event. The only way I can think of requires whole process suspend to be non-atomic:

1. Thread P calls SuspendThread to queue a suspend for thread A and GetThreadContext, which blocks.
2. Thread A hits a breakpoint, which enters the kernel. This suspends thread A and runs APCs for A, which wakes thread P's GetThreadContext. Suppose other threads haven't been suspended yet.
3. Thread P's GetThreadContext returns.
4. The kernel suspends all the other threads (including thread P).
5. Thread D wakes up from WaitForDebugEvent, uses GetThreadContext and SetThreadContext to back up RIP by 1, and resumes the process. Thread A still has a non-zero suspend count, so it remains suspended.
6. Thread P calls SetThreadContext to inject the asyncPreempt call, which that smashes thread D's context update, and resumes thread A.
7. Eventually asyncPreempt returns to the RIP after the INT3, which isn't a valid instruction (or is the wrong instruction). Badness.

This has a pretty narrow race window (I think), but could still happen, which is bad. Are there also "easier" ways to get into a bad state?

[aarzilli]

I think what happens is:

1. Thread A hits a breakpoint and enters the kernel
2. while thread A is in the kernel, thread P calls SuspendThread (which just increments a counter) and GetThreadContext which returns immediately because thread A is already suspended
3. after a while thread A successfully suspends the rest of the process
4. thread D wakes up from WaitForDebugEvent

the race window for this would be the scheduling quantum which is plenty big enough for this to happen regularly in a stress test. The way to reproduce this is to revert the changes I made to Delve to disable async preemption and then run TestBreakpointCounts (or better enable TestBreakpointCountsWithDetection and run that one) inside pkg/proc/proc_test.go.

I don't have a minimal reproducer at the moment but the code of Delve that actually runs with that test isn't a lot.

[aarzilli]

PS. I don't know for sure that that's what happens, I don't have access to the source code of windows, and it would imply an unfortunate implementation of GetThreadContext but I don't think that's unlikely since this isn't the "intended" use of SuspendThread/GetThreadContext.

[artli]

I seem to be having a related problem: when debugging in GoLand, the process of stepping over the source code lines often gets interrupted and instead of the next line the execution continues at runtime/preempt_amd64.s for no apparent reason.

I am not familiar with the details of the debugging process nor asynchronous preemption, but assuming my problem is connected to what is discussed in this thread, do you possibly know any kind of a temporary fix that can prevent this from happening and let me debug without interruption?

[networkimprov]

@artli did you try running with environment variable GODEBUG=asyncpreemptoff=1?

[artli]

@networkimprov, no, I haven't. Next time I have this issue I'try this method, thanks for the suggestion!

[aarzilli]

@artli the latest verison of Delve (1.4.0) should disable asynchronous preemption automatically on windows. And the symptoms are not what I would expect. Have you reported this to GoLand?

[artli]

@aarzilli, oh, turns out my installation of GoLand is way out of date and has Delve 1.1.0 vendored with it (for reference, the version can be checked by running <GoLand installation path>\plugins\go\lib\dlv\dlv.exe version). I'll update it and hope the problem goes away. Should have done that first, I guess :)

[ianlancetaylor]

Rolling forward to 1.16.

[odeke-em]

Rolling forward to 1.17.

[ianlancetaylor]

Moving to Backlog.