Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HAOS Core: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to threading #110538

Closed
diamant-x opened this issue Feb 14, 2024 · 25 comments
Closed

HAOS Core: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to threading #110538

diamant-x opened this issue Feb 14, 2024 · 25 comments
Labels
stale

Comments

@diamant-x
Copy link

diamant-x commented Feb 14, 2024 •
edited
Loading

The problem

After 2024.02 upgrade, my HA randomly restarts on its own at least 2-3 times per day.
Haven't found a pattern to it. System resources usage seems normal. HA created a .fault log file which I'm attaching from the last occurence today. I'm also adding the log file from before and after the restart; can't see anything odd.

Based on the error seems to be some issue when using the multiprocessing in some core function.

I'm happy to provide more details.
I installed HA on a new SD card back on early December and was running just fine until the previous version, so shouldn't be tear-down related AFAIK.

What version of Home Assistant Core has the issue?

core-2024.2.1

What was the last working version of Home Assistant Core?

core-2024.1.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Unknown (iperf3?)

Link to integration documentation on our website

Unknown

Diagnostics information

home-assistant.log

I'm changing the file extensions so that it can be uploaded.
home-assistant.1.log

home-assistant.fault.log

Example YAML snippet

N/A

Anything in the logs that might be useful for us?

Attached above the fault log file.

Additional information

System Information

version core-2024.2.1
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.1
os_name Linux
os_version 6.1.63-haos-raspi
arch aarch64
timezone Europe/Madrid
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
GitHub API Calls Remaining 5000
Installed Version 1.34.0
Stage running
Available Repositories 1396
Downloaded Repositories 27
Home Assistant Cloud
logged_in false
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 11.5
update_channel stable
supervisor_version supervisor-2024.01.1
agent_version 1.6.0
docker_version 24.0.7
disk_total 116.5 GB
disk_used 77.6 GB
healthy true
supported true
board rpi4-64
supervisor_api ok
version_api ok
installed_addons AdGuard Home (5.0.3), ESPHome (2023.12.9), FTP (5.0.1), Firefly iii (6.1.9), Firefly iii Data Importer (1.4.3), Home Assistant Google Drive Backup (0.112.1), InfluxDB2 (0.1.0.0), WireGuard (0.10.2), Nginx Proxy Manager (1.0.1), Samba share (12.3.0), Prowlarr NAS (nightly-1.13.3.4259-ls111), Vaultwarden (Bitwarden) (0.21.1), chrony (3.0.1), motionEye (0.20.0), qBittorrent (4.6.3-4), Sonarr (4.0.1.1096), Radarr (5.2.6.8376), Lidarr NAS (2.0.7.3849), Plex Media Server (3.4.1), MariaDB (2.6.1), Bazarr NAS (1.4.1), DOODS2 (1), Castsponsorskip (0.7.2), phpMyAdmin (0.9.1), Changedetection.io (0.45.14), Browserless Chrome (2.0.0), File editor (5.8.0)
Dashboards
dashboards 7
resources 17
views 17
mode storage
Recorder
oldest_recorder_run 17 de diciembre de 2023, 12:29
current_recorder_run 14 de febrero de 2024, 04:29
estimated_db_size 4104.64 MiB
database_engine mysql
database_version 10.6.12
@diamant-x
Copy link
Author

This is too unstable, here is a chart showing how right after the update is installed the restarts start occurring.

image

If I look closer, latest days all reboots happen exactly at around 4hours 10 minutes sin last restart:
image

With same error stack:
home-assistant.fault..log

@slovdahl slovdahl mentioned this issue Feb 16, 2024
Closed
@slovdahl
Copy link
Contributor

I'm seeing something similar since upgrading from 2024.1 to 2024.2.1 (home-assistant.log.fault.txt). Raspberry Pi 4, Ubuntu 20.04 64-bit, using Home Assistant Container.

$ uname -r -v -i
5.4.0-1101-raspi #113-Ubuntu SMP PREEMPT Fri Jan 12 18:10:28 UTC 2024 aarch64

@slovdahl
Copy link
Contributor

Might be fixed in 2024.2.2: #110464 (comment)

@diamant-x
Copy link
Author

Might be fixed in 2024.2.2: #110464 (comment)

That one mentions python 3.11, but my fault stack strace only shows python 3.12. let's see if we're lucky and it gets fixed too but could potentially not be the same.

@diamant-x
Copy link
Author

I confirm is still an issue in 2024.2.2. I can't believe is just a couple of us?

Versión core-2024.2.2
Tipo de instalación Home Assistant OS
Desarrollo false
Supervisor true
Docker true
Usuario root
Entorno virtual false
Versión de Python 3.12.1
Familia de Sistema Operativo Linux
Versión del Sistema Operativo 6.1.63-haos-raspi
Arquitectura de CPU aarch64

@DutchCruZer4
Copy link

DutchCruZer4 commented Feb 19, 2024 •
edited
Loading

I confirm is still an issue in 2024.2.2. I can't believe is just a couple of us?
Versión core-2024.2.2
Tipo de instalación Home Assistant OS
Desarrollo false
Supervisor true
Docker true
Usuario root
Entorno virtual false
Versión de Python 3.12.1
Familia de Sistema Operativo Linux
Versión del Sistema Operativo 6.1.63-haos-raspi
Arquitectura de CPU aarch64

Nope not a couple of us. I have the same issue.

I'm running on Proxmox VE 8.1

**Fatal Python error: Segmentation fault

Thread 0x00007f1b3d0ffb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3d71fb30 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 359 in wait
File "/usr/local/lib/python3.12/threading.py", line 655 in wait
File "/usr/src/homeassistant/homeassistant/components/stream/init.py", line 410 in _run_worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3de3fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3ee46b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3f133b30 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 355 in wait
File "/usr/local/lib/python3.12/queue.py", line 171 in get
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/api.py", line 367 in dispatch_events
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/api.py", line 202 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b3f246b30 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 355 in wait
File "/usr/local/lib/python3.12/site-packages/watchdog/utils/delayed_queue.py", line 51 in get
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify_buffer.py", line 41 in read_event
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify.py", line 128 in queue_events
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/api.py", line 145 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4186cb30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify_c.py", line 304 in read_events
File "/usr/local/lib/python3.12/site-packages/watchdog/observers/inotify_buffer.py", line 89 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b43c7fb30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/scapy/supersocket.py", line 264 in select
File "/usr/local/lib/python3.12/site-packages/scapy/sendrecv.py", line 1219 in _run
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4537fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b40c59b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b40d6cb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4287fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b46496b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b465a9b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b466bcb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b467cfb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b480efb30 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 359 in wait
File "/usr/local/lib/python3.12/queue.py", line 180 in get
File "/usr/local/lib/python3.12/site-packages/aiosqlite/core.py", line 99 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5bb5eb30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b48f0fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4a519b30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4a919b30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4f1cfb30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 579 in run_once
File "/usr/local/lib/python3.12/site-packages/pychromecast/socket_client.py", line 540 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5588fb30 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 359 in wait
File "/usr/local/lib/python3.12/threading.py", line 655 in wait
File "/usr/local/lib/python3.12/site-packages/pychromecast/discovery.py", line 304 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5608fb30 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4bd2cb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4c43fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b4e75cb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5196fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/src/homeassistant/homeassistant/components/recorder/executor.py", line 17 in _worker_with_shutdown_hook
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b51f8fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/src/homeassistant/homeassistant/components/recorder/executor.py", line 17 in _worker_with_shutdown_hook
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b595afb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b2ccb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b3e3b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b4fab30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b611b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b728b30 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/paho/mqtt/client.py", line 1150 in _loop
File "/usr/local/lib/python3.12/site-packages/paho/mqtt/client.py", line 1756 in loop_forever
File "/usr/local/lib/python3.12/site-packages/paho/mqtt/client.py", line 3591 in _thread_main
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5b83fb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5bd75b30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b5be8cb30 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00007f1b5c69fb30 (most recent call first):
File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 152 in _orjson_default_encoder
File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 180 in save_json
File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 364 in _write_data
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b67d5db38 (most recent call first):
File "/usr/local/lib/python3.12/selectors.py", line 462 in select
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1936 in _run_once
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 638 in run_forever
File "/config/custom_components/ble_monitor/init.py", line 702 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b67f74b38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b78c5ab38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b7a76fb38 (most recent call first):
File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 872 in _run_event_loop
File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 778 in _run
File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 698 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b845ffb38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b84ffbb38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b850ffb38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b859ffb38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b87b97b38 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b87cbfb38 (most recent call first):
File "/usr/local/lib/python3.12/logging/handlers.py", line 1527 in dequeue
File "/usr/local/lib/python3.12/logging/handlers.py", line 1578 in _monitor
File "/usr/local/lib/python3.12/threading.py", line 1010 in run
File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007f1b8cc01b08 (most recent call first):
File "/usr/local/lib/python3.12/selectors.py", line 462 in select
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1936 in _run_once
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 638 in run_forever
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 671 in run_until_complete
File "/usr/src/homeassistant/homeassistant/runner.py", line 188 in run
File "/usr/src/homeassistant/homeassistant/main.py", line 209 in main
File "/usr/src/homeassistant/homeassistant/main.py", line 221 in
File "", line 88 in _run_code
File "", line 198 in _run_module_as_main**

@diamant-x
Copy link
Author

Not sure if could be related, but the AEMET weather integration started failing at a similar moment:

image

Not sure if there's any specific way to point to AEMET developer to check if this might be related?

@DutchCruZer4
Copy link

Not sure if could be related, but the AEMET weather integration started failing at a similar moment:

image

Not sure if there's any specific way to point to AEMET developer to check if this might be related?

Found my problem. It was the hacs watchman plugin what got screwed. Disabled it and everything is fine now.

@rds76
Copy link

rds76 commented Feb 21, 2024

I closed #110464 but after 4days it crashed again, same symptoms - Fatal Python error: Segmentation fault, most recent version 2024.2.2 with updated orjson. Looks like Shelly is always involved

Current thread 0xb7d606c0 (most recent call first):
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 72 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1528 in json_fragment
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 70 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/json.py", line 48 in json_encoder_default
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1136 in json_fragment
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 70 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/websocket_api/messages.py", line 138 in _partial_cached_event_message
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/websocket_api/messages.py", line 122 in cached_event_message
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/websocket_api/commands.py", line 132 in _forward_events_unconditional
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1253 in async_fire
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1924 in async_set
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 1172 in _async_write_ha_state
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 989 in async_write_ha_state
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/shelly/entity.py", line 341 in _update_callback
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 183 in async_update_listeners
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 439 in async_set_updated_data
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/components/shelly/coordinator.py", line 336 in _async_handle_update
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/aioshelly/block_device/device.py", line 248 in _update_s
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/aioshelly/block_device/device.py", line 196 in _coap_message_received
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/aioshelly/block_device/coap.py", line 185 in datagram_received
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1169 in _read_ready
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80 in _run
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922 in _run_once
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607 in run_forever
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640 in run_until_complete
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/runner.py", line 188 in run
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/__main__.py", line 209 in main
  File "/opt/venv/homeassistant/bin/hass", line 8 in <module>

@diamant-x
Copy link
Author

Found my problem. It was the hacs watchman plugin what got screwed. Disabled it and everything is fine now.

How did you figured out which one it was?
I tried disabling a couple of HACS which haven't been updated in some time (so potentially not supporting Python 3.12) but without success.

@DutchCruZer4
Copy link

Found my problem. It was the hacs watchman plugin what got screwed. Disabled it and everything is fine now.

How did you figured out which one it was?

I tried disabling a couple of HACS which haven't been updated in some time (so potentially not supporting Python 3.12) but without success.

I had an ! on my device_tracker.yaml file. I checked it and it gave me an error on the file with a line couldn't be empty or only had special characters. I first disabled my ble_tracker and rebooted. Same fault. Then I disabled the watchman plugin and rebooted and fault was gone. It also had a lot of devices in the log and errors that devices where not found.

I think it brought my Home Assistant install in a constant error loop and that it got to much so the core got rebooted every now and then.

@diamant-x diamant-x mentioned this issue Feb 21, 2024
Closed
@diamant-x
Copy link
Author

Ok, now is clear something is wrong.
I rebooted into 'safe mode' HA to ensure if its was a HACS, or something else that was messing only to find that it keeps crashing at around ~4h:10m just as shown in OP.

Now I think I know what it is: iperf integration, which I've set it such scan interval to be that one. Not a coincidence the last log message before crash is precisely a server unavailable. Nothing new that shouldn't crash HA.

  • How can I update/redirect this ticket to iperf's integration maintainer?

home-assistant.1.log
home-assistant.fault.log

@diamant-x diamant-x changed the title HAOS: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to multiprocessing iperf3: HAOS: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to multiprocessing Feb 21, 2024
@diamant-x
Copy link
Author

Hey there @rohankapoorcom , mind taking a look at this issue as it has been labeled with an integration (iperf3) you are listed as a code owner for? Thanks!

@rds76
Copy link

rds76 commented Feb 21, 2024

Forget it is iPerf. Mine is still crashing and I'm not using iperf at all. It is definitelly core issue!

@diamant-x
Copy link
Author

Forget it is iPerf. Mine is still crashing and I'm not using iperf at all. It is definitelly core issue!

Crap, thanks for the update, will revert the ticket title and integration. Something that bothers me: in my last fault that shows the segmentation fault, it mentions some custom integrations that shouldn't have been enabled as I was in safe boot mode.
I'll try to remove them from the custom_components folder and try again into safeboot to see if it has any effects.

Another work action I took, I tried to understand if Python 3.12 had any main changes regarding threading and could find there has been important changes https://peps.python.org/pep-0684/ Not sure if HA devs already announced an impact due to this?

@diamant-x diamant-x changed the title iperf3: HAOS: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to multiprocessing HAOS Core: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to multiprocessing Feb 21, 2024
@diamant-x diamant-x changed the title HAOS Core: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to multiprocessing HAOS Core: "Fatal Python error: Segmentation fault" in RaspberryPi4 after 2024.02 upgrade related to threading Feb 21, 2024
@rds76
Copy link

rds76 commented Feb 21, 2024

Well, I'm still on Python 3.11, 1st thoughts was that, but I see segfaults on 3.12 here.. Must be related to 2024.02 as with previous release I had weeks of uptime with no issues. Such an error is mostly related to stack memory, buffer overflow, etc., not threading.

@FreddyX078
Copy link

FreddyX078 commented Feb 22, 2024 •
edited
Loading

Add me to the list, same problem here. Using 2024.2.2 with Python 3.12.
HA (HAOS on Beelink mini) restarts aprox 10 - 20 times a day :-(

MemTest ran for a couple of hours, no problem detected.
Booted into safe mode but same crash happend.

Error from supervisor:
24-02-22 11:36:27 WARNING (MainThread) [supervisor.homeassistant.websocket] Connection is closed
24-02-22 11:36:30 WARNING (MainThread) [supervisor.homeassistant.core] Watchdog found Home Assistant failed, restarting...

@rds76
Copy link

rds76 commented Feb 22, 2024

This is my commont part where it ends with segfault (3 crashed in recent 24h)

  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 72 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/core.py", line 1528 in json_fragment
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/backports/functools.py", line 70 in __get__
  File "/opt/venv/homeassistant-py3.11/lib/python3.11/site-packages/homeassistant/helpers/json.py", line 48 in json_encoder_default

I suspect orjson ijl/orjson#458

@DavidBuchanan314
Copy link

afaik ijl/orjson#458 has been an issue since day 1, if you're seeing "new" crashes I think it's more likely to be related to ijl/orjson#457

@Zaczero
Copy link

Zaczero commented Feb 22, 2024

This is most likely due to ijl/orjson#452. The crashes affect all orjson versions starting from 3.9.11. The crash rate was reduced in 3.9.14, but it's apparently still present. The HOC seems to use 3.9.14.

Warning! Personal Opinion:
I migrated from orjson to msgspec which has comparable performance and a similar set of features. I really don't like how the whole situation was/is handled by the orjson maintainers. They seem to prioritize performance over stability too much, which is just not acceptable for me. Also, they don't seem to listen to feedback, and response times are meh. orjson is also currently missing a security policy which is yet another red flag.

@FreddyX078
Copy link

An short update: after deleting all refresh-tokes, system is up for 8 hours.
Hopefuly it will survive the night.

@diamant-x
Copy link
Author

custom integrations that shouldn't have been enabled as I was in safe boot mode. I'll try to remove them from the custom_components folder and try again into safeboot to see if it has any effects.

An update from my side. I did this and the system didn't reboot in a full day. I re-added the custom_components folder and performed a normal restart again and the system now is not crashing. I hope it remains like that.
I would recommend all of you to try:

  1. Rename the 'custom_components' folder to 'temp_custom_components'
  2. Reboot into Safe Boot mode.
  3. See if it crashes or not.
  4. Rename back from step 1
  5. Reboot normal to reenable customization.

I think this might merit for a new bug ticket:

  • Safe boot reboot mode in HA loads anything inside the custom_components folders.

@diamant-x
Copy link
Author

diamant-x commented Feb 26, 2024 •
edited
Loading

the system now is not crashing.

It crashed again. core 2024.2.3

home-assistant.fault.log

System Information

version core-2024.2.3
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.1
os_name Linux
os_version 6.1.63-haos-raspi
arch aarch64
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
GitHub API Calls Remaining 5000
Installed Version 1.34.0
Stage running
Available Repositories 1405
Downloaded Repositories 27
Home Assistant Cloud
logged_in false
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 11.5
update_channel stable
supervisor_version supervisor-2024.02.0
agent_version 1.6.0
docker_version 24.0.7
disk_total 116.5 GB
disk_used 75.8 GB
healthy true
supported true
board rpi4-64
supervisor_api ok
version_api ok
installed_addons AdGuard Home (5.0.3), MariaDB (2.6.1)
Dashboards
dashboards 7
resources 17
views 17
mode storage
Recorder
oldest_recorder_run 30 de diciembre de 2023, 10:35
current_recorder_run 26 de febrero de 2024, 15:02
estimated_db_size 4023.14 MiB
database_engine mysql
database_version 10.6.12

@diamant-x
Copy link
Author

diamant-x commented Mar 4, 2024 •
edited
Loading

Still an issue in core-2024.2.5

home-assistant.fault.log

I'm wondering, could it be a Database related issue? I'm using the addon MariaDB Current version: 2.6.1.

System Information

version core-2024.2.5
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.1
os_name Linux
os_version 6.1.73-haos-raspi
arch aarch64
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
GitHub API Calls Remaining 4851
Installed Version 1.34.0
Stage running
Available Repositories 1405
Downloaded Repositories 27
Home Assistant Cloud
logged_in false
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 12.0
update_channel stable
supervisor_version supervisor-2024.02.1
agent_version 1.6.0
docker_version 24.0.7
disk_total 116.5 GB
disk_used 76.5 GB
healthy true
supported true
board rpi4-64
supervisor_api ok
version_api ok
Dashboards
dashboards 7
resources 17
views 17
mode storage
Recorder
oldest_recorder_run 4 de enero de 2024, 16:33
current_recorder_run 4 de marzo de 2024, 14:21
estimated_db_size 4241.77 MiB
database_engine mysql
database_version 10.6.12

@issue-triage-workflows
Copy link

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

@issue-triage-workflows issue-triage-workflows bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 9, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Jul 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@slovdahl @diamant-x @Zaczero @DavidBuchanan314 @rds76 @FreddyX078 @DutchCruZer4