Skip to content
This repository has been archived by the owner on Jul 15, 2023. It is now read-only.

Pause command doesn't work #978

Closed
renannprado opened this issue May 8, 2017 · 19 comments
Closed

Pause command doesn't work #978

renannprado opened this issue May 8, 2017 · 19 comments
Assignees

Comments

@renannprado
Copy link

renannprado commented May 8, 2017

After starting a debug session, if you hit pause button, nothing happens.
This is important if you want to add breakpoints after running the application.
I was debugging the plugin and saw that pauseRequest was never called. So I wonder whether this is a bug in this plugin or in the whole vscode.

@renannprado renannprado changed the title Pause command is not called Pause command doesn't work May 8, 2017
@renannprado
Copy link
Author

@ramya-rao-a is this an issue with the plugin or with vscode?
if it is a problem with the plugin, maybe I can try to help fix it

@ramya-rao-a
Copy link
Contributor

I dont have a proper repro for this. Can you share a sample code with repro steps?

cc @roblourens

@renannprado
Copy link
Author

renannprado commented May 9, 2017

So here are the steps to reproduce:

  1. start the debugging session with no breakpoints
  2. try to put a breakpoint (it won't work, the requests to create the breakpoint get stuck in delve)
    2.1. the only way to make the breakpoint to be created in that case would be if we send Halt to delve by clicking in pause button
  3. click in the pause button and nothing happens...

Sample program

package main

import (
	"fmt"
	"sync"
	"time"
)

func main() {
	wg := &sync.WaitGroup{}

	wg.Add(1)

	go func() {
		for {
			fmt.Println("I'll just take a nap for a second... ZZZzz....")
			time.Sleep(time.Second)
		}
	}()

	wg.Wait()
}

@ramya-rao-a
Copy link
Contributor

Thanks for the steps @renannprado

cc @roblourens

@roblourens
Copy link
Member

When debugging starts, vscode sends a ThreadsRequest. goDebug.ts sends a ListGoroutines request to Delve, but Delve never responds. So then when you click Pause, vscode doesn't know which thread to pause, and never sends the pause request over. I suppose we need to figure out whether we're caling ListGoroutines at a bad time, whether it's a Delve bug, or if we should just add a timeout for it.

@roblourens
Copy link
Member

And as for setting the bp after launch, I don't know whether it should be possible to set a BP when Go is in a loop like that. I can't do it with Delve at least.

@renannprado
Copy link
Author

renannprado commented May 11, 2017

@roblourens yes, it is possible (will later update with an asciinema session to show)
the only way to set a breakpoint in this case would be to use the command Halt (using the pause button in vscode), then everything is paused on the process and you are free to set the breakpoints

@muravjov
Copy link

muravjov commented Jan 27, 2018

Hi, I searched why I cannot set breakpoints as verified in VSCode while the program is running (after continue) and found out that you cannot set breakpoints in delve until you stop/halt debuggee program via delve.

That is, if you try to set a breakpoint in vscode while the debuggee program runs, the request to delve CreateBreakpoint hangs forever (till debugging session' end) and even F6 (Pause) will not work.
Workaround: before setting breakpoint you hit F6 (Pause), then set breakpoint, then Continue (F5) and everything is ok.

So I suggest to make start/stop requests to delve for every breakpoint being set if debuggee is running. That is, to call pauseRequest()/continueRequest() in setBreakPointsRequest() , see at

https://github.com/Microsoft/vscode-go/blob/master/src/debugAdapter/goDebug.ts#L530


I've checked out Goland, https://www.jetbrains.com/go/ , and found out that they do just stop/start in such situations. Here is the log from delve (patched by me) under Goland control, showing that delve was started and stopped only for setting breakpoint:

2018/01/27 23:44:22 server.go:265: rpc command: RPCServer.Command, version 2
2018/01/27 23:44:22 server.go:292: rpc command, is sync: false
2018/01/27 23:44:22 debugger.go:490: Debugger.Command: start
2018/01/27 23:44:22 debugger.go:498: halting
2018/01/27 23:44:23 debugger.go:562: Debugger.Command: stop
2018/01/27 23:44:23 debugger.go:562: Debugger.Command: stop
2018/01/27 23:44:23 server.go:265: rpc command: RPCServer.CreateBreakpoint, version 2
2018/01/27 23:44:23 server.go:292: rpc command, is sync: true
2018/01/27 23:44:23 debugger.go:284: CreateBreakpoint: 1
2018/01/27 23:44:23 debugger.go:303: CreateBreakpoint: 2
2018/01/27 23:44:23 debugger.go:328: CreateBreakpoint: 3
2018/01/27 23:44:23 debugger.go:339: CreateBreakpoint: 4
2018/01/27 23:44:23 debugger.go:348: created breakpoint: &api.Breakpoint{ID:1, Name:"", Addr:0x8b8188, File:"/home/ilya/opt/programming/g-core/balancer/src/balancer/master.go", Line:95, FunctionName:"balancer.SendCommandTo", Cond:"", Tracepoint:false, Goroutine:false, Stacktrace:0, Variables:[]string(nil), LoadArgs:(*api.LoadConfig)(nil), LoadLocals:(*api.LoadConfig)(nil), HitCount:map[string]uint64{}, TotalHitCount:0x0}
2018/01/27 23:44:23 server.go:265: rpc command: RPCServer.Command, version 2
2018/01/27 23:44:23 server.go:292: rpc command, is sync: false
2018/01/27 23:44:23 debugger.go:490: Debugger.Command: start
2018/01/27 23:44:23 debugger.go:509: continuing

Also, I've checked https://github.com/derekparker/delve sources and found out why one should not request CreateBreakpoint command while debuggee is working: every command except halt (= Pause in vscode) try to lock processMutex before performing its action, but if it is already locked (while debuggee runs) so the request hangs forever:

// CreateBreakpoint creates a breakpoint.
func (d *Debugger) CreateBreakpoint(requestedBp *api.Breakpoint) (*api.Breakpoint, error) {
	d.processMutex.Lock()
	defer d.processMutex.Unlock()
...

See at https://github.com/derekparker/delve/blob/master/service/debugger/debugger.go#L284

@muravjov
Copy link

muravjov commented Jan 27, 2018

I am not so good with TypeScript and those promises so I cannot make PR for my suggestion myself, the right way. Please help me with that.

@ramya-rao-a
Copy link
Contributor

@muravjov Thanks for the investigation!

So I suggest to make start/stop requests to delve for every breakpoint being set if debuggee is running. That is, to call pauseRequest()/continueRequest() in setBreakPointsRequest()

This would work assuming that pauseRequest() itself works as expected and halts delve.
As @renannprado and @roblourens has discussed above, that doesnt seem to be happening.
Are you able to pause the program and then set breakpoints?

@muravjov
Copy link

Yes, I am if I do it in right order: at first pause the program, then set breakpoints.
Other way it hangs forever (till debug session end).
And I understand why - if you set breakpoint while debuggee runs, then you hit deadlock - see d.processMutex.Lock()

@ramya-rao-a
Copy link
Contributor

I am not able to pause to begin with. Can you share the program you are using? I was using the outyet program from https://github.com/golang/example/tree/master/outyet

@muravjov
Copy link

muravjov commented Feb 15, 2018

Yes, I checked both examples, https://github.com/golang/example/tree/master/outyet , and #978 (comment) , and the situation is the same, vscode-go' delve controller stops to send any command to delve.

At the start of debug session vscode-go calls two GoDebugSession' methods, continueRequest() and threadsRequest(), you may see it in Debug Console, ContinueRequest and ThreadsRequest correspondingly (with "trace": "verbose" in launch.json).

In the majority of cases ContinueRequest occurs earlier, and delve's command "continue" lockes processMutex' mutex, see above. The next ThreadsRequest never returns , so no next commands, including Pause, ever has chance to be invoked. Fail.

You may see that ThreadsRequest is locked by killing dlv with SIGABRT signal so it prints its tracktraces:

$ kill -s SIGABRT <pid of the dlv debugging the program>

Sample stacktraces of killed dlv:

SIGABRT: abort
PC=0x4608f1 m=0 sigcode=0

...

goroutine 34 [semacquire]:
sync.runtime_SemacquireMutex(0xc42011106c, 0xc42049e600)
        /home/ilya/opt/programming/golang/go1.9.1/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420111068)
        /home/ilya/opt/programming/golang/go1.9.1/src/sync/mutex.go:134 +0x25e
github.com/derekparker/delve/service/debugger.(*Debugger).Goroutines(0xc420111060, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/debugger/debugger.go:825 +0x82
github.com/derekparker/delve/service/rpc1.(*RPCServer).ListGoroutines(0xc420490060, 0x0, 0x0, 0xc420487b00, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpc1/server.go:286 +0x8b
reflect.Value.call(0xc42049e600, 0xc420172208, 0x13, 0xbb38de, 0x4, 0xc4200afc50, 0x3, 0x3, 0x0, 0x0, ...)
        /home/ilya/opt/programming/golang/go1.9.1/src/reflect/value.go:434 +0xccc
reflect.Value.Call(0xc42049e600, 0xc420172208, 0x13, 0xc4200afc50, 0x3, 0x3, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/go1.9.1/src/reflect/value.go:302 +0xd7
github.com/derekparker/delve/service/rpccommon.(*ServerImpl).serveJSONCodec.func1(0xc4200afdc0, 0xc4200afea0, 0xc42049e600, 0xc420172208, 0x13, 0xc420388a20, 0xb09160, 0xc4202ed0d0, 0x194, 0xac7b00, ...)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpccommon/server.go:305 +0x1b8
github.com/derekparker/delve/service/rpccommon.(*ServerImpl).serveJSONCodec(0xc420082be0, 0x7fca06d8b0c8, 0xc420162028)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpccommon/server.go:307 +0xa49
created by github.com/derekparker/delve/service/rpccommon.(*ServerImpl).Run.func1
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpccommon/server.go:150 +0x1dc


goroutine 25 [syscall]:
syscall.Syscall6(0x3d, 0xffffffffffffffff, 0xc42057ed34, 0x40000000, 0x0, 0x0, 0x0, 0xc4200244d8, 0xc420024420, 0x10000c42057ece8)
        /home/ilya/opt/programming/golang/go1.9.1/src/syscall/asm_linux_amd64.s:44 +0x5
github.com/derekparker/delve/vendor/golang.org/x/sys/unix.wait4(0xffffffffffffffff, 0xc42057ed34, 0x40000000, 0x0, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:179 +0xba
github.com/derekparker/delve/vendor/golang.org/x/sys/unix.Wait4(0xffffffffffffffff, 0xc420179450, 0x40000000, 0x0, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/vendor/golang.org/x/sys/unix/syscall_linux.go:295 +0x8e
github.com/derekparker/delve/pkg/proc/native.(*Process).wait(0xc42007fd40, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/pkg/proc/native/proc_linux.go:341 +0xf3
github.com/derekparker/delve/pkg/proc/native.(*Process).trapWait(0xc42007fd40, 0xffffffffffffffff, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/pkg/proc/native/proc_linux.go:198 +0xc2
github.com/derekparker/delve/pkg/proc/native.(*Process).ContinueOnce(0xc42007fd40, 0x0, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/pkg/proc/native/proc.go:282 +0x32c
github.com/derekparker/delve/pkg/proc.Continue(0xe7e880, 0xc42007fd40, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/pkg/proc/proc.go:103 +0x1dd
github.com/derekparker/delve/service/debugger.(*Debugger).Command(0xc420111060, 0xc420493020, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/debugger/debugger.go:510 +0x148d
github.com/derekparker/delve/service/rpc1.(*RPCServer).Command(0xc420490060, 0xc420493020, 0x7fca06dd7b90, 0xc42016a180)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpc1/server.go:54 +0x5a
reflect.Value.call(0xc42049e180, 0xc4201720b8, 0x13, 0xbb38de, 0x4, 0xc42057ff48, 0x3, 0x3, 0x0, 0x0, ...)
        /home/ilya/opt/programming/golang/go1.9.1/src/reflect/value.go:434 +0xccc
reflect.Value.Call(0xc42049e180, 0xc4201720b8, 0x13, 0xc4202e1f48, 0x3, 0x3, 0x0, 0x0, 0x0)
        /home/ilya/opt/programming/golang/go1.9.1/src/reflect/value.go:302 +0xd7
github.com/derekparker/delve/service/rpccommon.(*ServerImpl).serveJSONCodec.func2(0xc42016a180, 0xc42049e180, 0xc4201720b8, 0x13, 0xc420388360, 0xace000, 0xc420493020, 0x16)
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpccommon/server.go:324 +0x207
created by github.com/derekparker/delve/service/rpccommon.(*ServerImpl).serveJSONCodec
        /home/ilya/opt/programming/golang/delve/src/github.com/derekparker/delve/service/rpccommon/server.go:318 +0xdc1

...

Here we see that goroutine 34 is locked by processMutex - debugger.go:825 in (d *Debugger) Goroutines() - it is our ThreadsRequest . And goroutine goroutine 25 that locked processMutex - it is our ContinueRequest .

But there is another situation there, when ThreadsRequest is invoked earlier than ContinueRequest . In that case delve returns but again, for some reason, I still don't see ThreadsResponse. Thereby Pause/pauseRequest() is not working again (whereas button Stop/disconnectRequest() works, sending 'halt' command successfully). If I have time later I'll debug vscode-go and can say why it is.


So, workaround is simple now: always set a breakpoint and when control hits it deadlock is over.

@ramya-rao-a
Copy link
Contributor

ramya-rao-a commented Feb 18, 2018

@muravjov

The next ThreadsRequest never returns , so no next commands, including Pause, ever has chance to be invoked.

I can see this happening when I try to pause a running program which never had any breakpoints

If I have a breakpoint, it gets hit, I continue, and then after a while try to pause, then I see the PauseRequest being made, delve halting, a bunch of thread requests being fired.

So, workaround is simple now: always set a breakpoint and when control hits it deadlock is over.

How do we determine when the deadlock is hit? Also, we cannot set breakpoints when the program is running, we have to pause/halt it.

@muravjov
Copy link

muravjov commented Feb 18, 2018

@ramya-rao-a

How do we determine when the deadlock is hit?

From the delve's side you are to find out whate all goroutines are doing now = their stacktraces.
If you rebuild delve with a couple of changes mentioned here https://stackoverflow.com/a/19145992 , then you may get all stacktraces with this command:

curl http://localhost:9100/debug/pprof/goroutine?debug=2

On Linux and MacOs you may kill go program with signal SIGABRT with the same effect, rebuild is not needed.

@dariopb
Copy link

dariopb commented Apr 16, 2018

Is there any workaround for the Pause -> set breakpoint -> Continue issue? I'm running vsc 1.22.2 with go extension 0.6.78 and still hitting the issue: everything goes fine until I "Pause", then I can set a breakpoint (breakpoint is "verified" it seems) but then "Continue" does nothing and the program doesn't keep running.

@primalmotion
Copy link
Contributor

same issue here.

I have identified the following

  1. if I start the debugger without any breakpoint. then hit pause. nothing happens (not even a trace in vscode go). game over

  2. if I start the debugger with an initial breakpoint that it hits, then click on continue, then hit pause, I see the halt message sent to delve, and the debugger seems to have halted, I can set breakpoints, but vscode never noticed it is halted, and the pause button never turns to a continue button. game over

  3. if I start the debugger with an initial breakpoint that it doesn't hit, same as 1

@ariel-bentu
Copy link

ariel-bentu commented Jan 8, 2019

Hi,
I managed to get this working on my macos vscode, by adding pause before setting a breakpoint and continue after it is set:
in line 704, add this line
var shouldContinue = false;
if (this.continueRequestRunning) {
shouldContinue = true;
this.pauseRequest(null);
}
in line 728, insert this before:
if (shouldContinue) {
this.continueRequest(null)
}

For me, finally it allows setting new breakpoints while debugee is running.
I hope you find this value-able and add this

@ramya-rao-a
Copy link
Contributor

The Pause feature now works in the latest update to the Go extension (0.9.0)
Huge thanks to the amazing @xiphon who made this happen!

@vscodebot vscodebot bot locked and limited conversation to collaborators Mar 14, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants