Possible to create a simple interactive debugger?

[maxmcd]

I created this builtin:

debugger := starlark.NewBuiltin("debugger", func(thread *starlark.Thread, fn *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
	repl.REPL(thread, b.predeclared)
	return nil, nil
})

But, unsurprisingly, when it is run, the various local variables or declared global variables are not available in the interactive session.

Is implementing this kind of interactive debugger feasible with the current lib implementation?

Thanks!

[alandonovan]

Not yet. At one point I started writing an asynchronous concurrent debugger (like gdb) with breakpoints, but got distracted. Let me see if I can find it, then perhaps we can define a reasonable API for accessing the environment to support your use case, even if we don't add complete support for breakpoints etc.

[maxmcd]

Wonderful, thank you

[maxmcd]

Hey @alandonovan any update on this?

[adonovan]

Sorry, no progress.

This feature requires that the bytecode interpreter has a single-byte BREAKPOINT instruction, plus new public APIs to

• overwrite existing instructions with BREAKPOINT, saving the overwritten byte so it can be restored later;
• map program counter offsets to file/line/column and back again;
• read and update global and local variables;
• compile and evaluate expressions provided by the user within the environment that exists at a given breakpoint;
• notify the debugger of thread events such as thread creation, function entry/exit, and so on;

And it requires a great deal of care with respect to concurrency.

I think this is probably several weeks' work for someone who understands the implementation well.

[AlmogBaku]

In #408 I suggested a somehow naive implementation for basic breakpoints:
I suggest adding a callback in this few lines where the vmdebug is placed.

This way we can have a programmatically way for users to implement their own breakpoint mechanism naively.

WDYT?

[adonovan]

A debugger is much more than a call to a user-defined function before and after every function call; it requires control over execution at a statement or instruction level, a means of inspecting and perhaps altering program state, and helpers for mapping between source and executable representations of the program.

How would the proposed hook allow someone to step through the iterations of a loop looking at the local variables?

Also, a debugger should not slow down execution when it is not in use, as adding a callback here would.

[AlmogBaku]

I'm not proposing a fully-featured debugger, but a "half-cooked" way to build a naive debugging tool.
Inspecting the variables is already feasible using this: https://github.com/google/starlark-go/blob/master/starlark/debug.go
Adding a callback when it's set should not impact the execution time (because when a debugging callback is not attached, it's just skipped).

[adonovan]

I'm not proposing a fully-featured debugger, but a "half-cooked" way to build a naive debugging tool.

This issue is about an interactive debugger, but I think what you're asking for is merely a call-tracing hook. That's a separate and much simpler feature, and I think it would be reasonable to add if we can come up with a good specification.

// A CallTracer receives notifications of all Starlark function calls and returns made by the interpreter.
// Call/Return trace events immediately bracket each call to Callable.CallInternal.
type CallTracer interface {
  TraceCall(thread *Thread, args []Tuple, kwargs []Tuple)
  TraceReturn(thread *Thread, result Value, err error) (Value, error)
}

// SetCallTracer installs a call tracer that will receive notifications of function calls and returns
// in any thread. Not concurrency safe; may be called only when the interpreter is idle.
// SetCallTracer replaces any previously installed tracer.
func SetCallTracer(CallTracer)

Is this interface useful? It sees the raw tuples of arguments (to all Callables, not just Starlark functions), not the cooked result of setArgs.

[maxmcd]

That would definitely be useful for me!

Still very much interested in the full debugger. Very likely too complicated for me to implement, but greatly appreciate the list of functionality that would be required. I'll take a look to see if I can figure out how it would all hook into the current implementation.

[adonovan]

A slight variant would be to make CallTracer be a per-Thread field, which would permit finer-grained control over which threads are traced and would avoid the concurrency issues of a global variable. But to support blanket tracing of all threads
this might demand an additional SetThreadCreateHook (a global variable) to ensure that tracing can be added to threads without modifying every program location that creates a thread.

And another (orthogonal) variant would be a single-method "call decorator":

type Thread struct {
    ...
    CallDecorator func (thread Thread, fn Callable, args Tuple, kwargs []Tuple) (Value, error)
}

where nil is equivalent to an implementation of { return fn.CallInternal(thread, args, kwargs) }, but a custom implementation can do extra logic before and after the CallInternal call, including using local variables to communicate between the call and return logic.

[itayd]

hey @adonovan , was there any attempt at implementing this? i could definitely use this. if not, I might be able to tackle it in the next few weeks.

[laurentlb]

fyi - There's a Starlark debugger in the Java implementation:
https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/lib/starlarkdebug

If there's a similar interface here, the debugger clients could be shared.

[adonovan]

hey @adonovan , was there any attempt at implementing this? i could definitely use this. if not, I might be able to tackle it in the next few weeks.

There was, but I can't at this moment find my old code. The approach I took was a breakpoint-oriented debugger that would swap out a byte of bytecode for a dedicated breakpoint instruction and restore it later. The concurrency challenges of supporting single stepping are quite complex because (a) breakpoints can be added and removed from the shared code segments by any thread, and (b) every breakpoint needs an associated list of threads and conditions; synchronizing these data structures is fiddly. Overall it is not a small project. I would want to discuss a design before reviewing any code changes.

There's a Starlark debugger in the Java implementation:
https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/lib/starlarkdebug
If there's a similar interface here, the debugger clients could be shared.

The Java implementation is completely different: it's a tree-walking interpreter. I don't think there's much to learn from it.

[itayd]

Sorry for late response - i did not get notified that you replied, and I was travelling.

I am not looking for real full fledged debugger, just for tracing, like @maxmcd . a CallDecorator like you suggested in #304 (comment) can be very useful for me.