debugpy 1.6.4 PermissionError while debugging Jupyter Notebook in VS Code on MacOS

[TheMatt2]

I am suspicious this bug was introduced by changes made for #1031

Environment data

• debugpy version: 1.6.4
• OS and version: MacOS 12.6.1 (21G217)
• Python version: Python 3.8/3.10/3.11 via Homebrew
• VS Code version: 1.74.0 Commit: 5235c6bb189b60b01b1f49062f4ffa42384f8c91
  • Python Extension: v2022.20.1
  • Pylance Extension: v2022.12.20
  • Jupyter Extension: v2022.11.1003412109

Actual behavior

Debugging a Jupyter Notebook in VS Code with debugpy 1.6.4 fails with a PermissionError on MacOS when the debugger tries to debug into a Python source file. This seems to be particularly an issue when trying working with a file on a Google Drive Volume.

Debugging works as intended if I install debugpy 1.6.3 by pip install debugpy==1.6.3
This has been tested on Python 3.8/3.10/3.11 with the same results.

Error Message:
(Personal identifiers removed)

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_file_utils.py", line 613, in _original_file_to_client
    return cache[filename]
           ~~~~~^^^^^^^^^^
KeyError: '/Users/<username>/Library/CloudStorage/[email protected]/My Drive/test.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 252, in _on_run
    self.process_net_command_json(self.py_db, json_contents)
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py", line 193, in process_net_command_json
    cmd = on_request(py_db, request)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py", line 971, in on_stacktrace_request
    self.api.request_stack(py_db, request.seq, thread_id, fmt=fmt, start_frame=start_frame, levels=levels)
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_api.py", line 207, in request_stack
    if internal_get_thread_stack.can_be_executed_by(get_current_thread_id(threading.current_thread())):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 655, in can_be_executed_by
    self._cmd = py_db.cmd_factory.make_get_thread_stack_message(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_net_command_factory_json.py", line 233, in make_get_thread_stack_message
    for frame_id, frame, method_name, original_filename, filename_in_utf8, lineno, applied_mapping, show_as_current_frame, line_col_info in self._iter_visible_frames_info(
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_net_command_factory_xml.py", line 196, in _iter_visible_frames_info
    new_filename_in_utf8, applied_mapping = pydevd_file_utils.map_file_to_client(filename_in_utf8)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_file_utils.py", line 615, in _original_file_to_client
    translated = _path_to_expected_str(get_path_with_real_case(absolute_path(filename)))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_file_utils.py", line 275, in get_path_with_real_case
    return _resolve_listing_parts(found, parts, filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_file_utils.py", line 168, in _resolve_listing_parts
    return _resolve_listing(resolved, iter(parts_in_lowercase))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_file_utils.py", line 150, in _resolve_listing
    dir_contents = cache[resolved_lower] = os_listdir(resolved)
                                           ^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 1] Operation not permitted: '/Users/<username>/Library/CloudStorage/[email protected]'

The error occurs specifically when trying to debug code that should trigger opening another window to display code in a Python file.

It seems the use of Google Drive is relevant here because trying to list the contents of the directory does cause a PermissionError:

import os
os.listdir('/Users/<username>/Library/CloudStorage/[email protected]')
# PermissionError: [Errno 1] Operation not permitted: ...

Expected behavior

I would expect debugpy to not be listing the contents of all directories above the Jupyter notebook being worked on,
or at list be able to recover from a PermissionError, as previous versions of debugpy have done.

Steps to reproduce:

1. Install Google Drive Desktop https://www.google.com/drive/download/
2. Create a file test.py in the Google Drive Volume. (The file can be blank)
3. Open VS Code
4. In VS Code Install Python and Jupyter Notebook Extensions
5. Make sure debugpy 1.6.4 is installed on the Python installation.
6. In VS Code open the folder that contains test.py
7. Create a Jupyter Notebook named test_notebook.ipynb and save it to the same folder.
8. Create a cell in the notebook that includes the line import test
9. Use the dropdown menu on the left of the cell to debug the cell.
10. Using the debug window step one line.
11. Observe the error message.

(Brownie points for finding a bug that intersects with software from Microsoft, Apple, and Google?)

[fabioz]

The change is that on 1.6.4 Mac OS is treated the same as Windows for the filesystem (because Mac OS is not case sensitive by default), thus debugpy now tries to get the actual case of the file from the filesystem (which in this case means doing a bunch of listdir calls on folders to get the actual case), but it didn't account for PermissionError.

I guess that the fix will be just ignoring the PermissionError (actually, handling OSError is probably better) and returning the initial path (hoping the user made the launch with the proper case -- I think the only bad thing here if the case doesn't match is that VSCode may end up opening an additional tab for the file even if it was already open... as a note, this is what always happened in 1.6.3 anyways -- breakpoints are normalized internally, so, hitting breakpoints shouldn't be an issue).

I'll provide a PR shortly...

Reference: this was added in: ae189da to fix #1031