debugpy: breakpoints not working in core integrations like tuya

[auphof]

The problem

Any breakpoints created in vscode debugger in a component such as Tuya don't trigger.
For instance I have a breakpoint set at

core/homeassistant/components/tuya/__init__.py

Line 150 in cfd1f78

LOGGER.debug(

. The Tuya debug log is enabled and I can see these in the log records from my Tuya devices, but the breakpoint does not occur.

I have followed instructions from the integration https://www.home-assistant.io/integrations/debugpy. I can remote attach successfully , all the threads are displayed in VSCODE debugger. if i attempt to pause a thread such as the thread Tuya is operating on (Thread ID obtained from Log Record) it does not happen immediately (Possibly async related) but periodically it does pause at some random spot, I can used Debug Console to interrogate the HA objects and step from there, however it never pauses at the intended breakpoint

I note in the Log of the HA container that when i start the DebugPy integration Service it talks about requiring python to be started with -Xfrozen_modules=off

When I apply that in the service run python3 -Xfrozen_modules=off -m homeassistant --config /config then breakpoints function as intended

Expected Outcome

Debug breakpoints in production HA container just work as per integration provided

I realize that might not be achievable so maybe an appropriate Developer Doc is created to support this. see my suggested method below, I am willing to put this as a PR against the develop docs, I am also willing to try other approaches.

A method of Debugging Production HA Containers

To Debug the latest HA Container I built a new container from the following dockerfile.

This Dockerfile is designed to set up a debugpy tuned containerized environment for debugging Home Assistant, The file outlines several steps to customize the container for fixing the debugpy breakpoints issue performance and potential integration improvements. Here's a breakdown of what each part does:

1. Base Image: The Dockerfile starts with FROM homeassistant/home-assistant:latest, which specifies that the container will be built from the latest version of the Home Assistant Docker image. This image contains all the necessary dependencies to run Home Assistant.

2. Customizing Service Script:

   • The first RUN command replaces the default service script for Home Assistant with a custom one. The script is written to the file /usr/src/homeassistant/rootfs/etc/services.d/home-assistant/run.
   • A similar operation is performed again, but this time the script is written to /etc/services.d/home-assistant/run. It's written to both places but i think it would work if only written to /etc/services.d/home-assistant/run. Both scripts:
     • Start with navigating to the /config directory, where Home Assistant's configuration files are expected to be found. If this directory does not exist, the script exits with an error.
     • Check if the DISABLE_JEMALLOC environment variable is not set. If it's not set, the script configures the container to use jemalloc as the memory allocator for Home Assistant, which can offer performance benefits. It does this by setting the LD_PRELOAD environment variable to preload libjemalloc.so.2 and configuring jemalloc settings via MALLOC_CONF.
     • Execute Home Assistant using Python 3, with a specific flag to disable frozen modules for improved performance or compatibility and specifying the configuration directory.

3. Commented Out Code for Tuya SDK:

   • There are commented-out sections that indicate optional steps for testing or using a patched version of the Tuya device sharing SDK. These sections are not active in the current state of the Dockerfile but suggest a way to uninstall the official Tuya SDK and replace it with a version from a specific Git repository. This could be useful for integrating or testing features not available in the official SDK release.
   • Another commented-out command shows how to patch the Tuya component within Home Assistant to use a different version of the Tuya SDK. This is done by modifying the manifest.json file of the Tuya component to require a new version of the SDK.

The commented-out sections offer flexibility for developers/testers working with customized but not released 3rd party python modules. Enabling the live testing of changes developed in the Devcontainer approach

FROM homeassistant/home-assistant:latest

RUN \
  cat > /usr/src/homeassistant/rootfs/etc/services.d/home-assistant/run <<EOF
#!/usr/bin/with-contenv bashio
# ==============================================================================
# Start Home Assistant service
# ==============================================================================

cd /config || bashio::exit.nok "Can't find config folder!"

# Enable mimalloc for Home Assistant Core, unless disabled
if [[ -z "${DISABLE_JEMALLOC+x}" ]]; then
  export LD_PRELOAD="/usr/local/lib/libjemalloc.so.2"
  export MALLOC_CONF="background_thread:true,metadata_thp:auto,dirty_decay_ms:20000,muzzy_decay_ms:20000"
fi
exec python3 -Xfrozen_modules=off -m homeassistant --config /config
EOF


RUN \
  cat > /etc/services.d/home-assistant/run <<EOF
#!/usr/bin/with-contenv bashio
# ==============================================================================
# Start Home Assistant service
# ==============================================================================

cd /config || bashio::exit.nok "Can't find config folder!"

# Enable mimalloc for Home Assistant Core, unless disabled
if [[ -z "${DISABLE_JEMALLOC+x}" ]]; then
  export LD_PRELOAD="/usr/local/lib/libjemalloc.so.2"
  export MALLOC_CONF="background_thread:true,metadata_thp:auto,dirty_decay_ms:20000,muzzy_decay_ms:20000"
fi
exec python3 -Xfrozen_modules=off -m homeassistant --config /config
EOF


# Uncomment for testing against a  patched tuya-device-sharing-sdk
#RUN \
#  pip uninstall -y tuya-device-sharing-sdk && \
#  pip install git+https://github.com/auphof/tuya-device-sharing-sdk.git@dev-au-fix-pir-24w06b#egg=tuya-device-sharing-sdk


# Patch the Tuya component manifest to use the updated version
# file is /usr/src/homeassistant/homeassistant/components/tuya/manifest.json
# find `"requirements": ["tuya-device-sharing-sdk==0.1.9"]` and
# replace with `"requirements": ["tuya-device-sharing-sdk==0.1.10"]`


## Assuming /usr/src/homeassistant is already populated with the Home Assistant code
#RUN sed -i 's/"requirements": \["tuya-device-sharing-sdk==0.1.9"\]/"requirements": ["tuya-device-sharing-sdk==0.1.10"]/g' #/usr/src/homeassistant/homeassistant/components/tuya/manifest.json

Build the modified image on a target platform

# In my case I am targeting a PI4 , so the following is done in a folder on a  pi4

#1. Edit and place a copy of the above dockerfile in a file `dockerfile.update` in the appropriate dir
mkdir -p ha-container-patch && cd ha-container-patch
cat > dockerfile.update <<EOF
#<<<<paste and edit the relevant dockerfile parts from above in place of this line>>>>
EOF

#2. Define the name for the image, ie  repository/tag for the customized HA container Image
# example TARGET_REPO_TAG="10.1.1.155:5101/homeassist:patched1a"  #where this is my local network private docker image registry
TARGET_REPO_TAG="localhost/homeassist:patched-1a" #accessible on this machine

#3. Build new image and push if required
sudo docker build -f dockerfile.update -t $TARGET_REPO_TAG.
#Push If you are using a local private registry (Useful for HA container images being so large > 1.4Gb)
# NB:         Configuration of a local private registry is not covered here
#sudo docker push 10.1.1.155:5101/homeassist:patched1a

#4 Use $TARGET_REPO_TAG in docker run or docker-compose to run with the adjusted and debugable container

Disclosure

Interpretation of the dockerfile.update was suggested and edited with aid of ChatGPT4, my human eyes, brain and fingers then modified appropriately 😁

What version of Home Assistant Core has the issue?

2024.2.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

seen in TUYA

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

System Information

version	core-2024.2.1
installation_type	Unknown
dev	false
hassio	false
docker	false
user	root
virtualenv	false
python_version	3.12.1
os_name	Linux
os_version	6.6.14-200.fc39.aarch64
arch	aarch64
timezone	Pacific/Auckland
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.34.0
Stage	running
Available Repositories	1397
Downloaded Repositories	2
HACS Data	ok

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Dashboards

dashboards	2
resources	1
views	1
mode	storage

Recorder

oldest_recorder_run	February 10, 2024 at 1:45 AM
current_recorder_run	February 15, 2024 at 1:56 PM
estimated_db_size	3.82 MiB
database_engine	sqlite
database_version	3.44.2

[home-assistant]

Hey there @frenck, mind taking a look at this issue as it has been labeled with an integration (debugpy) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of debugpy can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign debugpy Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

debugpy documentation
debugpy source
_{^{(message by IssueLinks)}}

[sHedC]

Hi,

Was looking at this issue for ages in my custom component, not sure if this helps but when I switch to Python 3.11.8 debuggng and breakpoints work fine. On 3.12.x debuging starts but breakpoints are ignored, I could not find out why.

Only thing I got was this but pretty sure this is not the issue, can always be wrong.

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Not sure if I can offer any other help but happy to do so if I can.

[rohankapoorcom]

I can confirm the same problem with 3.12.x. Switching back to Python 3.11.x allows breakpoints to work for both custom integrations as well as core.

[KThaulow]

When I apply that in the service run python3 -Xfrozen_modules=off -m homeassistant --config /config then breakpoints function as intended

@auphof can you elaborate on how you got this working? Can you do that from the DebugPy integration Service?

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[sHedC]

I am able to run with debug breakpoints now, slow startup but working.

[Devirex]

I had only partially luck with the configuration above. The internal debugpy module is not catching any breakpoints in my case, even if i could see the debug log messages in the container log.

Solution: I did it like this and it worked:

Two new files in the same directory of the dockerfile

• run1.sh:

#!/usr/bin/with-contenv bashio
# ==============================================================================
# Start Home Assistant service
# ==============================================================================

cd /config || bashio::exit.nok "Can't find config folder!"

# Enable mimalloc for Home Assistant Core, unless disabled
if [[ -z "${DISABLE_JEMALLOC+x}" ]]; then
  export LD_PRELOAD="/usr/local/lib/libjemalloc.so.2"
  export MALLOC_CONF="background_thread:true,metadata_thp:auto,dirty_decay_ms:20000,muzzy_decay_ms:20000"
fi
exec python3 -Xfrozen_modules=off -m debugpy --wait-for-client --listen 0.0.0.0:5678 -m homeassistant --config /config

• run2.sh:

#!/usr/bin/with-contenv bashio
# ==============================================================================
# Start Home Assistant service
# ==============================================================================

cd /config || bashio::exit.nok "Can't find config folder!"

# Enable mimalloc for Home Assistant Core, unless disabled
if [[ -z "${DISABLE_JEMALLOC+x}" ]]; then
  export LD_PRELOAD="/usr/local/lib/libjemalloc.so.2"
  export MALLOC_CONF="background_thread:true,metadata_thp:auto,dirty_decay_ms:20000,muzzy_decay_ms:20000"
fi
exec python3 -Xfrozen_modules=off -m debugpy --wait-for-client --listen 0.0.0.0:5678 -m homeassistant --config /config

• dockerfile:

FROM homeassistant/home-assistant:latest

COPY run1.sh /usr/src/homeassistant/rootfs/etc/services.d/home-assistant/run
COPY run2.sh /etc/services.d/home-assistant/run
RUN chown root:root /usr/src/homeassistant/rootfs/etc/services.d/home-assistant/run && chmod +x /usr/src/homeassistant/rootfs/etc/services.d/home-assistant/run
RUN chown root:root /etc/services.d/home-assistant/run && chmod +x /etc/services.d/home-assistant/run

Build it:
docker build -f dockerfile -t localhost/homeassist:patched-1a .

docker run or docker-compose up and then Remote Attach with VSCode:

• launch.json:

{
      // Debug by attaching to remote Home Assistant server using Remote Python Debugger.
      // See https://www.home-assistant.io/integrations/debugpy/
      "name": "Home Assistant: Attach Remote",
      "type": "debugpy",
      "request": "attach",
      "connect": {
        "port": 5678,
        "host": "your-ha-instance"
      },
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "/usr/src/homeassistant"
        }
      ]
}

Everything is catched right from the start.

[kushagharahi]

On 2024.9: @Devirex's instructions worked for me. It seems like the crux of the issue is that -Xfrozen_modules=off needs to be passed to debugpy but the only way to pass it in is through commandline args

[ab623]

Apparently from Python 3.13 you can pass this via an environment flag: https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_FROZEN_MODULES

[stalniy]

Seems that frozen modules has nothing to do with the fix. And actually what fixes debugging is that debugpy module is included earlier than homeassistant module. I was able to debug with this command:

python3 -m debugpy --listen 0.0.0.0:5678 -m homeassistant --config "${PWD}/config" --debug