How to debug the debugger

This page describes how to debug the jupyter extension and debugpy when the user is running a 'debug' command. You should probably read all of the debug topics to familiarize yourself with how debugging works before following the steps below.

Debugging Extension side code

When the user clicks Debug Cell in a notebook, there are a number of entrypoints (as shown in the sequence diagram on the other wiki page)

You can debug these entry points just like debugging anything in the extension:

• Set a breakpoint
• Launch 'Extension' debugger launch.json entry.
• Perform the user task to start debugging

Entry Point	Why you might start here	What the cause might be
Debugging Manager::startDebuggingConfig	Debug cell isn't starting the debugger. This is the main entrypoint and it's what tells VS code to start the debugger	Debug config may not match, so check the config when starting.
Kernel Debug Adapter::handleMessage	Debug cell is starting the debugger (or entering debug mode) but breakpoints don't hit or the cell never looks like it's running.	The correct DAP messages are note being sent to the kernel.
Kernel Debug Adapter::onIOPubMessage	Debug cell is starting the debugger (or entering debug mode) but breakpoints don't hit or the cell never looks like it's running.	The resulting DAP messages from the debugpy are referencing invalid paths or showing an error.
Kernel Debug Adapter::dumpCell	Debug cell is starting the debugger and cell seems to run, but breakpoints don't bind or seem to jump around	dumpCell may be creating output files not referencable by VS code (like remote paths) or line number translation could be off
Kernel Debug Adapter::translate functions	Breakpoints move or don't bind. Wrong files open when stopping	File path translation is generating invalid paths for debugpy or there's a line number translation problem

Debugging Debugpy

Okay so what if all of those entry points look good but debugpy just doesn't seem to respond correctly?

There are three things you can do to debug debugpy:

Turn on logging

The easiest thing to do is turn on logging for debugpy with some environment variables:

Environment Variable	Value
PYDEVD_DEBUG	true
DEBUGPY_LOG_DIR	directory that already exists
PYDEVD_DEBUG_FILE	directory that already exists - use same as previous variable

Launching VS code with those environment variables will generate files like so in the directory specified:

These files are:

• debugpy.adapter - messages logged by the adapter process (process that is speaking DAP with VS code)
• debugpy.pydevd - messages logged by pydevd (generally has more information)
• debugpy.server - messages logged when debugpy is started in listen mode (more DAP messages)

Opening the adapter log should list out the the DAP messages and their contents. Here's an example:

D+00000.234: Client[1] --> {
                 "seq": 1,
                 "type": "request",
                 "command": "initialize",
                 "arguments": {
                     "clientID": "vscode",
                     "clientName": "Visual Studio Code - Insiders",
                     "adapterID": "Python Interactive Window Debug Adapter",
                     "pathFormat": "path",
                     "linesStartAt1": true,
                     "columnsStartAt1": true,
                     "supportsVariableType": true,
                     "supportsVariablePaging": true,
                     "supportsRunInTerminalRequest": true,
                     "locale": "en",
                     "supportsProgressReporting": true,
                     "supportsInvalidatedEvent": true,
                     "supportsMemoryReferences": true
                 }
             }

This is:

• a message sent from the client (VS Code) indicated by Client[1] -->
• an initialize message (first DAP message)

A response from the server would look like:

D+00000.281: Server[1] --> {
                 "pydevd_cmd_id": 502,
                 "seq": 10,
                 "type": "response",
                 "request_seq": 5,
                 "success": true,
                 "command": "initialize",
                 "body": {
                     "supportsConfigurationDoneRequest": true,
                     "supportsFunctionBreakpoints": true,
                     "supportsConditionalBreakpoints": true,
                     "supportsHitConditionalBreakpoints": true,
                     "supportsEvaluateForHovers": true,
                     "exceptionBreakpointFilters": [
                         {
                             "filter": "raised",
                             "label": "Raised Exceptions",
                             "default": false
                         },
                         {
                             "filter": "uncaught",
                             "label": "Uncaught Exceptions",
                             "default": true
                         },
                         {
                             "filter": "userUnhandled",
                             "label": "User Uncaught Exceptions",
                             "default": false
                         }
                     ],
                     "supportsStepBack": false,
                     "supportsSetVariable": true,
                     "supportsRestartFrame": false,
                     "supportsGotoTargetsRequest": true,
                     "supportsStepInTargetsRequest": true,
                     "supportsCompletionsRequest": true,
                     "completionTriggerCharacters": [],
                     "supportsModulesRequest": true,
                     "additionalModuleColumns": [],
                     "supportedChecksumAlgorithms": [],
                     "supportsRestartRequest": false,
                     "supportsExceptionOptions": true,
                     "supportsValueFormattingOptions": true,
                     "supportsExceptionInfoRequest": true,
                     "supportTerminateDebuggee": true,
                     "supportsDelayedStackTraceLoading": true,
                     "supportsLoadedSourcesRequest": false,
                     "supportsLogPoints": true,
                     "supportsTerminateThreadsRequest": false,
                     "supportsSetExpression": true,
                     "supportsTerminateRequest": true,
                     "supportsDataBreakpoints": false,
                     "supportsReadMemoryRequest": false,
                     "supportsDisassembleRequest": false,
                     "supportsClipboardContext": true,
                     "supportsDebuggerProperties": true,
                     "pydevd": {
                         "processId": 21600
                     }
                 }
             }

You can then verify that the server is responding with the expected messages and that they have expected results. For example you might verify that breakpoints on for the correct files, or that the stack frame response has the correct file in it.

Debug debugpy's DAP server

Okay so what if the server isn't sending the correct messages?

You can actually debug debugpy's adapter server. It's the process that is sitting and listening to messages from VS code. For example on my machine, debugging a cell in a notebook creates an adapter process like so:

Special variable

You need to set a special environment variable though. This variable lets debugpy debug itself (as of this commit, which will likely be in 1.63 of debugpy).

DEBUGPY_TRACE_DEBUGPY=1

Normally this should be off as it exposes all the hidden details of debugpy to VS code (messes up justmycode).

You need to set this before starting the kernel. There's a number of options to get the kernel to have this environment variable:

• Globally
• Terminal that you start VS code from
• .env file that the kernel uses on launche

Attach and debug

Once the kernel has that environment variable, you'd start a new instance of VS code and setup a launch.json like so in the new instance:

        {
            "name": "Python: Attach using Process Id",
            "type": "python",
            "request": "attach",
            "processId": "${command:pickProcess}",
            "justMyCode": false // This has to be false
        }

the you'd

• Debug a cell in the notebook (in the original VS code instance with the notebook open)
• Wait that one cell to finish debugging (this gets the adapter process started)
• Attach to the adapter process (12428 from before) (in the new VS code)
• Set breakpoints (in the new VS code)
• Debug a cell again (in the original VS code)

Where to set breakpoints?

Debugpy's adapter process is the middleman between VS code and pydevd in the kernel. It handles DAP messages and then forwards them onto pydevd.

In order to set breakpoints, you open the site-packages folder for the kernel you're running. For example, this is where the debugpy installation is for my debugPyLatest conda kernel:

C:\Users\aku91\miniconda3\envs\debugPyLatest\Lib\site-packages\debugpy

Some useful spots to set breakpionts:

Source location	Site-packages location	What it handles
Log write	\Lib\site-packages\debugpy\common\log.py	Every write to the log goes through here. If you break at this location you can use it to figure out how the adapter is structured. Note, you need to have the other environment variables set before this works.

Add extra logging to pydevd