Terminology used in the debugpy documentation

[nhadziosma1]

I have been struggling to understand the terminology which is being used in documentation for the debugpy package, and I would really appreciate if you could help me understand it a bit better.

To my understanding, there are three component which are involved in the debugging-process when you run the debugger in an IDE which uses the debugging-adapter-protocol (i.e. DAP):

1. The actual debugger for a specific programming-language
2. The debugger-adapter
3. The IDE itself, which has some generic debugger created inside of it, so that it can use the debugger-adapter to connect to the actual debugger

I think that the following screenshot describes well the relation between the three terms I described above:

Note: I am new to DAP, and this is my understanding of the DAP protocol, and how modern IDE-s (like VS Code, or PyCharm) implement the debugging functionality. The knowledge I have about DAP gained from the following article: the article about DAP

The terms which appear in the documentation of the package debugpy (mostly in the descriptions of functions defined in debugpy) that I have a problem with understanding are:

1. debug-server
2. DAP-client

Can you please help me map these terms to the ones which are given in the explanation which I gave above, or an additional explanation if they refer to something completely different?

I am guessing correctly that the "DAP-client" refers to the IDE itself, and that "debug-server" refers to the actual debugger for a certain programming-language. However, even I am guessing correctly, I lack the basic understanding of how to what do the following descriptions of functions refer to:

• debugpy.connet(...) - Connect the debug-server to a DAP-client that is waiting for incoming connections.
• debugpy.listen(...) - Set up the debug-server to listen for incoming DAP-client connections.

What does it meant that "a DAP-client is waiting for an incoming connection"? If "DAP-client" refers to the IDE, how can the IDE be waiting for a connection?
What does it mean that "a debugging-server is to listen to an incoming DAP-client connection"? If "debug-server" refers to an actual debugger of a programming-language, how can the actual debugger be waiting for a connection?

Thank you in advance for your help.

[rchiodo]

If "DAP-client" refers to the IDE, how can the IDE be waiting for a connection?

The IDE can either open a port itself (and then debugpy.connect would be used) or the IDE can connect to a port created by debugpy (debupy.listen). This proposal on web assembly actually has a diagram:

https://github.com/microsoft/debugpy/wiki/Debugpy-with-Webassembly-(proposal)#debugpy-and-how-it-connects

What does it mean that "a debugging-server is to listen to an incoming DAP-client connection"? If "debug-server" refers to an actual debugger of a programming-language, how can the actual debugger be waiting for a connection?

There are actually multiple processes involved when using debugpy. There's the VS code process (DAP client), the Debug adapter (debugpy.adapter subfolder), and the app itself. The debug adapter is listening for DAP requests and then it forwards the necessary messages to the debugpy code injected into the app. The adapter process is there to manage the lifetime of the app (like being able to stop debugging by killing the process).

The app isn't really just running the user code. It gets started by debugpy so that there are other threads running in the process that handle DAP messages. Then it uses something like runpy to run the user code.

[nhadziosma1]

Hello Ricardo.

Firstly, thank you for your prompt reply.

However, I am afraid that your answer did not completely clarify the terminology for me. It would probably be essential for me to understand more basic terminology in order to underrated your response, so I would be grateful to you if you could help me by trying to answer the questions below.

Before diving into the questions, lets agree on the following definitions of 'client', 'server', and 'client-server relation':

A client-server relationship describes how a server can provide resources or services to one or more clients.
The client is the service requester.
The server is the service provider.

The questions:

• I found the following photo on the internet:
  
  and if we take the same logic to this package (i.e. library), is debugpy a debugger-adapter, or something else?

• Is the role of a DAP-client to request a connection to a debugger, and the role of a DAP-server to make the connection possible (and he does it by opening a port)?

  • According to the link which you sent me (i.e. the graph whose screenshot if given below), the term "DAP-server" can refer to both the IDE (i.e. VS Code) and to the debugpy, depending on which one of the two opens the port. Same holds for "DAP-client". Is that true?
    

  • What happens when a port opens (and do different thigs happen depending if the IDE opens the port, and if debugpy does it)?

  • I tried opening a port using the code debugpy.connect("localhost", portNum), and it succeeded one time (I don't know if it is important, but for context, I finish debugging by a key interrupt). However, if I run the code for the second time, I get an error saying that the port is already in use. Then I have to change the portNum, and run the code again in order to debug it again (and this change does not only have to be done inside the debugpy.connect(...) command, but also inside the debugging-configurations in launch.json). It is very likely that I am missing some feature which allows you to reuse the port which has already been open, but I ma not sure which one is it.

I am probably lacking the fundamental understandings on how the package (i.e. library) should work, so I would politely ask you to try explain it me on a basic level. Thank you in advance.

[rchiodo]

and if we take the same logic to this package (i.e. library), is debugpy a debugger-adapter, or something else?

That diagram is misleading. Each box in that diagram is not a process.

Debugpy starts 3 processes when you debug from VS code:

• debugpy launcher (launches the adapter and the process your debugging) - not in that diagram
• debugpy adapter (listens to DAP messages and forwards them to the process. It sticks around so that the process can be killed but messages can still be processed) - the box marked Debug adapter in the diagram
• debuggee (where debugpy is inside of the debuggee) - the two boxes marked debugger and debuggee

Maybe it would be better to understand why you want to know how debugpy works? What are you using it for? You seem interested in the server/client aspect of debugpy. Are you trying to do something remote?

[nhadziosma1]

I am trying to debug an app in VS Code which is was created based on a template repo, and it is imagined to be run the app through a defined CLI command. The template repo is constantly changing, and with every change, the CLI commands for running the code are being adjusted by another team. So, my team uses the template, but seamlessly can use the same CLI command to run their apps which they build on top of the template repo.

Since I am not able to simply press 'Run' or 'Debug', but run the app through a CLI command, I thought of creating a wrapper around the CLI command using debugpy which would allow me to debug the code, and not to constantly print-out messages.

In order to make it possible to debug the app I tried using debugpy, by doing the following:

1. adjusting the launch.json file in which I added the debugging-configurations (where the debugging-configurations called (debugger) Locally run chat is created for the sake of using debugpy.):

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [{"name": "(debugger) Current File with Arguments",
                        "type": "debugpy",
                        "request": "launch",
                        "program": "${file}",
                        "cwd": " ${workspaceFolder}",
                        "console": "integratedTerminal",
                        "args": ["${command:pickArgs}"]
                       },

                       {"name": "(debugger) Locally run chat",
                        "type": "debugpy",
                        "request": "attach",
                        "connect": {"host": "localhost",
                                    "port": 5681
                                   },
                        "pathMappings": [{"localRoot": "${workspaceFolder}",
                                          "remoteRoot": "."
                                         }
                                        ]
                       }  
    ]
}

2. create a wrapper function which starts debugpy, that allows me to debug the app using a CLI command:

def debug_chat():
    try:
        debugpy.connect(("localhost", 5681))
        debugpy.wait_for_client()
        chat()

    finally:
        #A function which would make it possible to free a port (like 'debugpy.disconnect(...)') does not exist in 'debugpy'
        #debugpy.disconnect()
        print("Debugging session ended.")

The adjustments which I made to the code seemed to work fine when I run it for the first time, but whenever I tried to run the code for the second time, I got an error message:
ConnectionRefusedError: [Errno 111] Connection refused
and whenever I change the port number in both launch.json and debugpy.connect(...) to a different number (but the number of the ports in both places had to be the same), I was again able to debug the code once without the error.

At that point it seamed to me that I am occupying a port each time I run the debugpy.connect(...) function, but I am never making it free after the end of execution.

Then I thought that there must exist some kind of a debugpy.diconnect(...) function which would allow me not to free the port after the end of execution in the finally block. To my surprise, after digging through the documentation of debugpy, I got to know that such a function does not exist in debugpy, but I also got to know that I didn't understand the difference between debugpy.connect(...) and debugpy.listen(...).

Then I decided to dig a bit deeper into the logic behind the functions, and that is when I got to the definitions which are using the terminology from the client-server model, and terms such as DAP-client and DAP-server.

I am not interested necessarly in what happens under the hood in the debugpy package (i.e. library) in the sense you explain it, through which processes are being run. Instead, I want to understand the basic concepts, and I am interested in:

1. Getting high-level understanding of how does debugpy make the debugger work (and I assume that such a high-level explanation could be used to describe any package/library which allows you to debug a project/app in an IDE, and not only debugpy).
2. What is the meaning of the terms like DAP-server, DAP-client, and what is the role of the port?
3. What does it mean that I am debugging a remote (which I think is not happening in my case, but it is essential to understand it, as that is the main use of the package/library debugpy. This would hopefully help me understand the documentation better)?
4. What is the difference between debugpy.connect(...) and debugpy.listen(...)?
5. Is there a possibility of using the same port throughout multiple runs of the debugger, and is there a way to free it after each run when using debugpy?

Note: I would be super grateful if you could answer each of my questions one by one, and do it in a high-level manner (imagining that I am the biggest dummy for the topic). I think that it would be the only way for me to fully understand the package/library, its capabilities, and intended purposes.

[rchiodo]

Okay so I'll just repeat what you described in order to make sure I understand it.

You want to debug an app which is launched by another app - a template app as you called it. The command line args to the launched app keep changing so you can't just launch it yourself?

My first thought is why not debug the template app itself? Maybe it's not python code. Debugpy can debug child processes but it will only work if the parent process is also python.

My second thought is to do to what you're describing. Have your app wait for the debugger to attach, maybe with an environment variable or something that dictates waiting.

I'll try answering your other questions:

1. High level understanding of the python debugger: This wiki page is where @int19h tried to describe what you're asking for. I don't think I could do a better job. You might also look at the API page to understand how to use the debugpy API.
2. DAP-Client generally refers to the IDE. DAP-Server generally refers to a process that debugpy starts. That picture flips the nomenclature around, but generally the client is the one starting the whole thing. The server is listening for requests from the client. They communicate over a port.
3. Remote debugging means the process you're debugging is on a different machine than where the IDE is. For debugpy this means opening up a port that can be seen externally (which is a security risk) and it mitigates this security risk by enforcing a key that needs to be passed back and forth.
4. Difference between connect and listen. Connect is for when debugpy is connecting to a port already owned by the IDE. Listen is for when debugpy owns the port and the IDE attaches to it. Listen is easier in my opinion, otherwise you have to start the IDE before the process.
5. I believe what's happening is with connect you're not starting the IDE before the python process (or soon enough). Try this instead:

import debugpy
debugpy.listen(5679)  # listen for incoming DAP client connections (but open the port here)
debugpy.wait_for_client()  # wait for a client to connect

print("After attach")

Then in VS code use this launch.json

    {
      "name": "Python Debugger: Attach to listening debugpy",
      "type": "debugpy",
      "request": "attach",
      "connect": {
        "port": 5679
      },
    },

That works for me to repeatedly attach and disconnect from a process. Every time the process shuts down, the port is freed so it can be used again.