debugging django app in kubernetes with debugpy attach times out with requests

[jankovicgd]

Hey all, not sure if this is the best place for this issue, please let me know.

Environment data

• VS Code version: 1.56.2
• Extension version (available under the Extensions sidebar): v2021.5.842923320
• OS and version: Ubuntu 20.04
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.5
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): kubernetes pod
• Relevant/affected Python packages and their versions:
• Relevant/affected Python-related VS Code extensions and their versions: debugpy==1.3.0
• Value of the python.languageServer setting: Pylance

Expected behaviour

Debugging as usual without interruption.

Actual behaviour

The debugger "times out" and can no longer debug and step through the code when sending a web request to a pod running in minikube.

Steps to reproduce:

1. Run a minikube cluster with a django application
2. Forward debug port 5678:5678 to pod
3. Attach debugger to service with the following:

launch.json

{
   "name": "Django Attach",
   "type": "python",
   "request": "attach",
   "connect": {
     "host": "localhost",
     "port": 5678
   },
   "pathMappings": [
    ...
   ] 
}

manage.py

import os
import sys
import debugpy

def initialize_debugger():
    if not os.getenv('RUN_MAIN'):

        import debugpy
        debugpy.listen(('0.0.0.0', 5678))
        sys.stdout.write('Start the VS Code debugger now, waiting...\n')
        debugpy.wait_for_client()

...
 def main()
     # django boilerplate above...
      initialize_debugger() 
      execute_from_command_line(sys.argv)

4. Send request to service hitting a breakpoint
5. Debug for ~60 secs
6. Debugger times out

Tried sending the request via the ingress, and also forwarded the port of the app to localhost and sent a request circumventing the ingress but same results appeared. This happens only when trying to debug requests. Using (django) cli and running something like

kubectl exec -it <pod_id> -- python3 -m debugpy --wait-for-client --listen 0.0.0.0:5678 -m <module_to_debug> <args>

works without timeouts and I can debug without issues.

Logs

Wasn't really getting anything, the logs below seem to be unrelated i.e they didn't appear consistently. Also I am not sure how I can provide more info.

Developer console

[Violation] Forced reflow while executing JavaScript took 30ms

Extension host logs

[2021-05-17 12:05:30.396] [exthost] [error] [ms-python.python] provider FAILED
[2021-05-17 12:05:30.396] [exthost] [error] Canceled: Canceled
	at E.makeCancelError (/home/nikola/.vscode/extensions/ms-python.python-2021.5.842923320/out/client/extension.js:9:183478)
	at E.handleFailedRequest (/home/nikola/.vscode/extensions/ms-python.python-2021.5.842923320/out/client/extension.js:9:183329)
	at /home/nikola/.vscode/extensions/ms-python.python-2021.5.842923320/out/client/extension.js:9:143671

Nginx (ingress) sends a 502 Bad Gateway. Tried here also setting various timeouts but nothing helped.

Let me know if you need any more info, might be of a more difficult setup to test, but it is a bit annoying when being limited in debugging.

[jankovicgd]

I tried also debugpy.log_to(path) and this is the result.

I+00000.017: Linux-5.8.0-50-generic-x86_64-with-glibc2.29 x86_64
             CPython 3.8.5 (64-bit)
             debugpy 1.3.0

I+00000.032: Initial environment:
             
             System paths:
                 sys.prefix: /usr
                 sys.base_prefix: /usr
                 sys.real_prefix: <missing>
                 site.getsitepackages(): /usr/lib/python3.8/dist-packages
                                         /usr/lib/python3/dist-packages
                                         /usr/local/lib/python3.8/dist-packages
                 site.getusersitepackages(): /root/.local/lib/python3.8/site-packages
                 sysconfig.get_path('stdlib'): /usr/lib/python3.8
                 sysconfig.get_path('platstdlib'): /usr/lib/python3.8
                 sysconfig.get_path('purelib'): /usr/lib/python3.8/site-packages
                 sysconfig.get_path('platlib'): /usr/lib/python3.8/site-packages
                 sysconfig.get_path('include'): /usr/include/python3.8
                 sysconfig.get_path('scripts'): /usr/bin
                 sysconfig.get_path('data'): /usr
                 os.__file__: /usr/lib/python3.8/os.py
                 threading.__file__: /usr/lib/python3.8/threading.py

D+00000.032: wait_for_client()

[fabioz]

I'm not exactly sure how to help here... the error seems to be in the infrastructure used and not really in the debugger.

I think that the issue is that the connection is flaky. Can you do some ssh tunneling instead to see if it'd be more consistent?

[jankovicgd]

This is all done locally, i.e the cluster is local, not sure if an ssh tunnel is necessary (and possible?) here. I might try to setup a repository that can easily replicate this and run some sample and see where it goes.

[fabioz]

@int19h @karthiknadig

In the pydevd side, there are some tweaks to the socket to keep the connection alive:
https://github.com/microsoft/debugpy/blob/v1.3.0/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py#L476

So, this should work from pydevd <-> adapter, but I can't see anything to keep the connection alive from adapter <-> vscode... do you know of any such setting (either on the debugpy or vscode side)?

If it there's nothing similar, this could be the root reason for this issue...

[int19h]

I'm pretty sure we had SO_KEEPALIVE on for our sockets, but we must have regressed it at some point.

[fabioz]

@jankovicgd can you can try to update your local copy with the patch from: https://github.com/microsoft/debugpy/pull/639/files to see if it works for you with those changes?

[jankovicgd]

Sorry for the long radio silence, basically stopped working on this until I needed to again (which is now), but @fabioz was right. It is an infrastructure problem, and now that I'm older and wiser I was able to quickly figure it out. The kubernetes pod was hitting a liveness probe check that ultimately made it restart while a debug session is ongoing, so it wasn't even an issue in debugpy. Apologies for the noise 😓. But I was able to find the thread by googling so maybe this can serve as a reminder to someone else attaching debugpy to kubernetes pods that only serve a single threaded synchronous debug process.