Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System unstable after switching from SD-Card to NVMe SSD #3674

Closed
RubenKelevra opened this issue Nov 15, 2024 · 5 comments · Fixed by #3687
Closed

System unstable after switching from SD-Card to NVMe SSD #3674

RubenKelevra opened this issue Nov 15, 2024 · 5 comments · Fixed by #3687
Labels
board/raspberrypi Raspberry Pi Boards bug usb-ssd USB SSD related issue

Comments

@RubenKelevra
Copy link
Contributor

Describe the issue you are experiencing

I'm using a Raspberry Pi 4B with 8 GB of memory, and have been using a MicroSD-Card for around 5 years.

To get more performance, I switched now to an NVMe SSD, which is in a case and connected via an USB3 Cable to one of the USB3 ports of the Raspberry Pi 4B.

I've updated the BIOS of the Raspberry Pi around 2 weeks ago and did a fresh install onto the NVMe and then restored a backup onto it.

I think I can also rule out an issue with the SSD, as I used it on my notebook without any issues until I switched it out 6 months ago with a larger size.

The issue I'm seeing is that HA suddenly looses access to the Zigbee network after hours or days working fine. When I access the UI, it works, but most functions in the settings page are blocked. And from some of the error messages it's pretty clear, that there's something wrong with the access of the disk:

Screenshot_20241116_001625

The disk LED is blinking, as if HAOS is constantly accessing the disk.

Rebooting the system via the UI does not work. I can't access the system page in the HA UI, to get access to the logs and add-ons are not accessible:

Screenshot_20241116_001458

After reboot the system works fine again, for hours or days.

I've tried so far: I've replaced the USB cable, to see if there's an issue with the cable.

What operating system image do you use?

rpi4-64 (Raspberry Pi 4/400 64-bit OS)

What version of Home Assistant Operating System is installed?

13.2

Did the problem occur after upgrading the Operating System?

No

Hardware details

Rasberry Pi 4B 8 GB
Samsung NVMe SSD
NVMe to USB3 case

Steps to reproduce the issue

  1. Run HAOS with HA on it on this hardware

Anything in the Supervisor logs that might be useful for us?

No, as I can't access it when this happens.

Anything in the Host logs that might be useful for us?

No, as I can't access it when this happens.

System information

System Information

version core-2024.11.1
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.31-haos-raspi
arch aarch64
timezone Europe/Berlin
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
HACS Data ok
GitHub API Calls Remaining 5000
Installed Version 2.0.1
Stage running
Available Repositories 1527
Downloaded Repositories 17
Home Assistant Cloud
logged_in true
subscription_expiration xxx
relayer_connected false
relayer_region null
remote_enabled false
remote_connected false
alexa_enabled false
google_enabled true
cloud_ice_servers_enabled true
remote_server null
certificate_status null
instance_id xxx
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 13.2
update_channel stable
supervisor_version supervisor-2024.11.2
agent_version 1.6.0
docker_version 27.2.0
disk_total 234.0 GB
disk_used 34.7 GB
healthy true
supported true
host_connectivity true
supervisor_connectivity true
ntp_synchronized true
virtualization
board rpi4-64
supervisor_api ok
version_api ok
installed_addons Mosquitto broker (6.4.1), Grocy (0.22.1), Home Assistant Google Drive Backup (0.112.1), Advanced SSH & Web Terminal (19.0.0), Log Viewer (0.17.1), Dnsmasq (1.8.1), phpMyAdmin (0.10.0), ESPHome (2024.10.3), Epic Games Free (debian-2024-10-23), Portainer (2.23.0), Tor (5.0.1), File editor (5.8.0), AppDaemon (0.16.7), Tailscale (0.23.2), openWakeWord (1.10.0), Piper (1.5.2), Whisper (2.2.0), MQTT IO (0.4.0), Postgres 15 (15.5-7), Castsponsorskip (0.8.0), Zigbee2MQTT (1.41.0-1)
Dashboards
dashboards 6
resources 2
views 26
mode storage
Recorder
oldest_recorder_run August 22, 2024 at 15:01
current_recorder_run November 16, 2024 at 00:18
estimated_db_size 9425.49 MiB
database_engine sqlite
database_version 3.45.3

Additional information

No response
@RubenKelevra RubenKelevra mentioned this issue Nov 15, 2024
Closed
@RubenKelevra
Copy link
Contributor Author

Another crash today, this time HA threw a 500:

Screenshot_20241116_101237

@RubenKelevra
Copy link
Contributor Author

I've let the host log open and there's finally something substantial in it. To be clear, the system works fine right now, but there are issues logged with the storage:

2024-11-16 19:49:28.360 homeassistant kernel: audit: type=1334 audit(1731786568.355:494): prog-id=129 op=UNLOAD
2024-11-16 19:49:28.360 homeassistant kernel: audit: type=1334 audit(1731786568.355:495): prog-id=128 op=UNLOAD
2024-11-16 19:49:28.360 homeassistant kernel: audit: type=1334 audit(1731786568.355:496): prog-id=127 op=UNLOAD
2024-11-16 19:49:28.693 homeassistant systemd[1]: systemd-timedated.service: Deactivated successfully.
2024-11-16 19:49:28.724 homeassistant kernel: audit: type=1334 audit(1731786568.719:497): prog-id=132 op=UNLOAD
2024-11-16 19:49:28.724 homeassistant kernel: audit: type=1334 audit(1731786568.719:498): prog-id=131 op=UNLOAD
2024-11-16 19:49:28.724 homeassistant kernel: audit: type=1334 audit(1731786568.719:499): prog-id=130 op=UNLOAD
2024-11-16 19:57:27.087 homeassistant kernel: xhci_hcd 0000:01:00.0: ERROR Transfer event for unknown stream ring slot 2 ep 7
2024-11-16 19:57:27.088 homeassistant kernel: xhci_hcd 0000:01:00.0: @000000040f004820 00000000 00000000 05000000 02088001
2024-11-16 19:58:19.572 homeassistant kernel: sd 0:0:0:0: [sda] tag#15 uas_eh_abort_handler 0 uas-tag 24 inflight: CMD OUT 
2024-11-16 19:58:19.574 homeassistant kernel: sd 0:0:0:0: [sda] tag#15 CDB: opcode=0x2a 2a 00 01 24 0c 28 00 00 40 00
2024-11-16 19:58:19.576 homeassistant kernel: sd 0:0:0:0: [sda] tag#14 uas_eh_abort_handler 0 uas-tag 23 inflight: CMD OUT 
2024-11-16 19:58:19.577 homeassistant kernel: sd 0:0:0:0: [sda] tag#14 CDB: opcode=0x2a 2a 00 01 24 0c 10 00 00 08 00
2024-11-16 19:58:19.578 homeassistant kernel: sd 0:0:0:0: [sda] tag#13 uas_eh_abort_handler 0 uas-tag 22 inflight: CMD OUT 
2024-11-16 19:58:19.579 homeassistant kernel: sd 0:0:0:0: [sda] tag#13 CDB: opcode=0x2a 2a 00 01 24 0b f0 00 00 18 00
2024-11-16 19:58:19.581 homeassistant kernel: sd 0:0:0:0: [sda] tag#12 uas_eh_abort_handler 0 uas-tag 21 inflight: CMD OUT 
2024-11-16 19:58:19.582 homeassistant kernel: sd 0:0:0:0: [sda] tag#12 CDB: opcode=0x2a 2a 00 01 24 0b d8 00 00 08 00
2024-11-16 19:58:19.583 homeassistant kernel: sd 0:0:0:0: [sda] tag#11 uas_eh_abort_handler 0 uas-tag 20 inflight: CMD 
2024-11-16 19:58:19.584 homeassistant kernel: sd 0:0:0:0: [sda] tag#11 CDB: opcode=0x2a 2a 00 01 24 0b a8 00 00 28 00
2024-11-16 19:58:19.585 homeassistant kernel: sd 0:0:0:0: [sda] tag#17 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
2024-11-16 19:58:19.587 homeassistant kernel: sd 0:0:0:0: [sda] tag#17 CDB: opcode=0x2a 2a 00 00 54 40 08 00 00 08 00
2024-11-16 19:58:19.588 homeassistant kernel: sd 0:0:0:0: [sda] tag#16 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT 
2024-11-16 19:58:19.589 homeassistant kernel: sd 0:0:0:0: [sda] tag#16 CDB: opcode=0x2a 2a 00 00 52 f6 10 00 00 08 00
2024-11-16 19:58:19.590 homeassistant kernel: sd 0:0:0:0: [sda] tag#18 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
2024-11-16 19:58:19.591 homeassistant kernel: sd 0:0:0:0: [sda] tag#18 CDB: opcode=0x2a 2a 00 00 12 ca 48 00 00 02 00
2024-11-16 19:58:19.591 homeassistant kernel: sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
2024-11-16 19:58:19.592 homeassistant kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 29 5b 90 00 02 f8 00
2024-11-16 19:58:19.593 homeassistant kernel: sd 0:0:0:0: [sda] tag#23 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT 
2024-11-16 19:58:19.594 homeassistant kernel: sd 0:0:0:0: [sda] tag#23 CDB: opcode=0x2a 2a 00 00 ab c8 00 00 04 00 00
2024-11-16 19:58:19.594 homeassistant kernel: sd 0:0:0:0: [sda] tag#21 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT 
2024-11-16 19:58:19.595 homeassistant kernel: sd 0:0:0:0: [sda] tag#21 CDB: opcode=0x2a 2a 00 01 53 93 08 00 00 08 00
2024-11-16 19:58:19.596 homeassistant kernel: sd 0:0:0:0: [sda] tag#20 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT 
2024-11-16 19:58:19.596 homeassistant kernel: sd 0:0:0:0: [sda] tag#20 CDB: opcode=0x2a 2a 00 00 ab cc 00 00 00 d0 00
2024-11-16 19:58:19.597 homeassistant kernel: sd 0:0:0:0: [sda] tag#19 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT 
2024-11-16 19:58:19.598 homeassistant kernel: sd 0:0:0:0: [sda] tag#19 CDB: opcode=0x2a 2a 00 04 76 d8 10 00 00 50 00
2024-11-16 19:58:19.599 homeassistant kernel: sd 0:0:0:0: [sda] tag#8 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT 
2024-11-16 19:58:19.599 homeassistant kernel: sd 0:0:0:0: [sda] tag#8 CDB: opcode=0x2a 2a 00 00 ab cc d0 00 02 f8 00
2024-11-16 19:58:19.600 homeassistant kernel: sd 0:0:0:0: [sda] tag#22 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT 
2024-11-16 19:58:19.601 homeassistant kernel: sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x2a 2a 00 00 26 04 c8 00 00 80 00
2024-11-16 19:58:19.602 homeassistant kernel: sd 0:0:0:0: [sda] tag#28 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT 
2024-11-16 19:58:19.603 homeassistant kernel: sd 0:0:0:0: [sda] tag#28 CDB: opcode=0x2a 2a 00 01 24 0e 00 00 00 08 00
2024-11-16 19:58:19.604 homeassistant kernel: sd 0:0:0:0: [sda] tag#27 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD OUT 
2024-11-16 19:58:19.605 homeassistant kernel: sd 0:0:0:0: [sda] tag#27 CDB: opcode=0x2a 2a 00 01 24 0d e8 00 00 08 00
2024-11-16 19:58:19.606 homeassistant kernel: sd 0:0:0:0: [sda] tag#26 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT 
2024-11-16 19:58:19.606 homeassistant kernel: sd 0:0:0:0: [sda] tag#26 CDB: opcode=0x2a 2a 00 01 24 0d b8 00 00 28 00
2024-11-16 19:58:19.607 homeassistant kernel: sd 0:0:0:0: [sda] tag#25 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD OUT 
2024-11-16 19:58:19.608 homeassistant kernel: sd 0:0:0:0: [sda] tag#25 CDB: opcode=0x2a 2a 00 01 24 0d 98 00 00 10 00
2024-11-16 19:58:19.609 homeassistant kernel: sd 0:0:0:0: [sda] tag#24 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD OUT 
2024-11-16 19:58:19.609 homeassistant kernel: sd 0:0:0:0: [sda] tag#24 CDB: opcode=0x2a 2a 00 01 24 0d 88 00 00 08 00
2024-11-16 19:58:19.610 homeassistant kernel: sd 0:0:0:0: [sda] tag#10 uas_eh_abort_handler 0 uas-tag 29 inflight: CMD OUT 
2024-11-16 19:58:19.611 homeassistant kernel: sd 0:0:0:0: [sda] tag#10 CDB: opcode=0x2a 2a 00 01 24 0f 38 00 00 18 00
2024-11-16 19:58:19.611 homeassistant kernel: sd 0:0:0:0: [sda] tag#9 uas_eh_abort_handler 0 uas-tag 28 inflight: CMD OUT 
2024-11-16 19:58:19.612 homeassistant kernel: sd 0:0:0:0: [sda] tag#9 CDB: opcode=0x2a 2a 00 01 24 0f 28 00 00 08 00
2024-11-16 19:58:19.613 homeassistant kernel: sd 0:0:0:0: [sda] tag#7 uas_eh_abort_handler 0 uas-tag 27 inflight: CMD OUT 
2024-11-16 19:58:19.614 homeassistant kernel: sd 0:0:0:0: [sda] tag#7 CDB: opcode=0x2a 2a 00 01 24 0e e0 00 00 20 00
2024-11-16 19:58:19.615 homeassistant kernel: sd 0:0:0:0: [sda] tag#6 uas_eh_abort_handler 0 uas-tag 26 inflight: CMD OUT 
2024-11-16 19:58:19.615 homeassistant kernel: sd 0:0:0:0: [sda] tag#6 CDB: opcode=0x2a 2a 00 01 24 0e d0 00 00 08 00
2024-11-16 19:58:19.616 homeassistant kernel: sd 0:0:0:0: [sda] tag#5 uas_eh_abort_handler 0 uas-tag 25 inflight: CMD OUT 
2024-11-16 19:58:19.617 homeassistant kernel: sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x2a 2a 00 01 24 0e 98 00 00 28 00
2024-11-16 19:58:19.618 homeassistant kernel: sd 0:0:0:0: [sda] tag#4 uas_eh_abort_handler 0 uas-tag 19 inflight: CMD OUT 
2024-11-16 19:58:19.619 homeassistant kernel: sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x2a 2a 00 01 24 0e 80 00 00 08 00
2024-11-16 19:58:19.620 homeassistant kernel: sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0 uas-tag 18 inflight: CMD OUT 
2024-11-16 19:58:19.620 homeassistant kernel: sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x2a 2a 00 01 24 0e 70 00 00 08 00
2024-11-16 19:58:19.621 homeassistant kernel: sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 17 inflight: CMD OUT 
2024-11-16 19:58:19.622 homeassistant kernel: sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x2a 2a 00 01 24 0e 20 00 00 40 00
2024-11-16 19:58:19.623 homeassistant kernel: sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD OUT 
2024-11-16 19:58:19.623 homeassistant kernel: sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x2a 2a 00 01 24 0e 10 00 00 08 00
2024-11-16 19:58:19.624 homeassistant kernel: scsi host0: uas_eh_device_reset_handler start
2024-11-16 19:58:19.625 homeassistant kernel: usb 2-1: reset SuperSpeed USB device number 2 using xhci_hcd
2024-11-16 19:58:19.626 homeassistant kernel: scsi host0: uas_eh_device_reset_handler success

@RubenKelevra
Copy link
Contributor Author

Switching to a USB2 port on the Raspberry Pi mitigates the problem. However I don't see how any of the hardware is an issue. I'm using a 10 Gbit/s cable while the Raspberry Pi is only capable of 5 Gbit/s and the Samsung NVMe ran my desktop fine for 3 years und ArchLinux.

So I guess there's a driver issue or something like that?

How can I switch from UAS to BOT in HAOS over USB3?

@sairon
Copy link
Member

sairon commented Nov 18, 2024

Most likely we will need to know a bit more about this thing:

NVMe to USB3 case

It's unfortunately fairly common issue that the UAS mode doesn't work reliably with Raspberry Pi but it can be mitigated quite easily - you just need to find the USB VID/PID of the controller in the enclosure and put it to the list of quirks in cmdline.txt in the boot partition (you can see last addition in #3281). If you confirm the issue is resolved then, we can add this to the default cmdline.txt to make sure new installations are fine in the future (also note that cmdline.txt is mostly untouched by OS upgrade, so any changes you make there are persisted).

@sairon sairon added usb-ssd USB SSD related issue board/raspberrypi Raspberry Pi Boards labels Nov 18, 2024
@RubenKelevra RubenKelevra mentioned this issue Nov 21, 2024
Closed
RubenKelevra added a commit to RubenKelevra/home-assistant_operating-system that referenced this issue Nov 21, 2024
@RubenKelevra
add device quirk for JMicron's "External" USB3-to-NVMe adapter
fad28b0 
Vendor ID: 152d
Product ID: a583

fixes home-assistant#3674
RubenKelevra added a commit to RubenKelevra/home-assistant_operating-system that referenced this issue Nov 21, 2024
@RubenKelevra
add device quirk for JMicron's "External" USB3-to-NVMe adapter
1a236f4 
Vendor ID: 152d
Product ID: a583

fixes home-assistant#3674
@RubenKelevra RubenKelevra mentioned this issue Nov 21, 2024
Merged
@sairon sairon closed this as completed in a042dc0 Nov 21, 2024
sairon pushed a commit that referenced this issue Nov 22, 2024
@RubenKelevra @sairon
Disable UAS via usb-storage.quirks on RPi for JMicron JMS583
1291848 
Vendor ID: 152d
Product ID: a583

fixes #3674

(cherry picked from commit a042dc0)
@RubenKelevra
Copy link
Contributor Author

I've updated to Home Assistant OS 14.0, powered the host off, switched to the USB3 port, but the issue persists.

@sairon how can I check in Home Assistant OS which protocol is used via USB3?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
board/raspberrypi Raspberry Pi Boards bug usb-ssd USB SSD related issue
Projects
None yet
Milestone
No milestone
2 participants
@sairon @RubenKelevra