U-Boot fails to boot from USB SSD sometimes on RPi 3b+

[mmalina]

Describe the issue you are experiencing

Today I updated my HAOS from 7.0 to 7.1. After reboot, I am stuck on "No ethernet found". This is displayed several times and eventually I am shown a HAOS> prompt. My RPi 3b+ is connected to ethernet and I never had connectivity issues until this update today. Reboot doesn't help.

What operating system image do you use?

rpi3 (Raspberry Pi 3 32-bit OS)

What version of Home Assistant Operating System is installed?

7.1

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

1. Update from HAOS 7.0 to 7.1
2. Wait for HAOS to reboot
3. The OS fails to connect to ethernet

Anything in the Supervisor logs that might be useful for us?

I have no connectivity to my RPi because of this, so I can't check. (I'm not sure if HA Core even started at all.)

Anything in the Host logs that might be useful for us?

I have no connectivity to my RPi because of this, so I can't check. (I'm not sure if HA Core even started at all.)

System Health information

No response

Additional information

No response

[mmalina]

Here's a photo of my screen after updating to 7.1:

[mmalina]

Net:
No ethernet found.
starting USB.
I Bus usbe7e980000: usb dr mode not found
1 USB DUCZ
; scanning bus usbe7e980000 for devices... 4 USB Devices) found
scanning usb for storage devices... O Storage Device(s) found
WARNING at drivers/mmc/bcm2835_sdhost.c:414/bcm2835_send_command()!
WARNING at drivers/nmc/bcm2835_sdhost.c:414/bcm2835_send_command()!
Card did not respond to voltage select! : -110
)Card did not respond to voltage select! :
-110
Device O: unknown device
I No ethernet found.
I missing environment variable: pxeuuid
I missing environment variable: bootfile
I Retrieving file: pxelinux.cfg/00000000
I No ethernet found.
I missing environment variable: bootfile
I Retrieving file: pxelinux.cfg/0000000
1 Mo ethernet found.
I missing environment variable: bootfile
Retrieving file: pxelinux.cfg/000000
I No ethernet found.
I missing environment variable: bootfile
I Retrieuing file: pxelinux.cfg/00000
1 Mo ethernet found.
I missing environment variable: bootfile
Retrieving file: pxelinux.c$g/0000
I No ethernet found.
I missing enuironment variable: bootfile
I Retrieving file: pxelinux.c£g/000
No ethernet found.
I missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00
1 No ethernet found.
missing environment variable: bootfile
Retrieving file: pxelinux.cfm/o
I No ethernet found.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm-bcn2B3x-rpi
I No ethernet found.
I missing environment variable: bootfile
I Retrieuing file:
pxelinux.cfg/default-arm-bcm283x
No ethernet round.
I missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arn
I No ethernet found.
missing environment variable: bootfile
I Retrieuing file: pxelinux.cfg/default
No ethernet found.
Config file not found
1No ethernet round.
HAOS>

[mmalina]

Hmm, this still happened after several more reboots. Then I left my RPi off for a few minutes and when I tried again, it booted normally. But unless it's some kind of hardware problem, it still seems like some sort of a bug in HAOS. I would appreciate if somebody could comment on this.

[agners]

This messages are from the boot loader. It seems the boot loader can't access your boot medium.

What medium are you booting HAOS from? Which version did you upgrade from?

[mmalina]

I'm booting from an external SSD disk connected via an adapter - this one: Sabrent SATA Cable Adapter, USB 3.0 to SSD / 2.5-inch SATA Hard Drive Adapter [Optimized for SSD, Supports UASP SATA III] (EC-SSHD)

I upgraded from HAOS 7.0 to 7.1.

But as I said, after several more attempts, it eventually booted alright.

So do you think that the Sabrent adapter could be unreliable? Or could it be Raspberry Pi itself?

[agners]

But as I said, after several more attempts, it eventually booted alright.

Sometimes its only reboot which is affected, sometimes its by chance (works say 2 out of 10 times, probably related to some timing/race condition in USB reset).

USB SSD booting on Raspberry Pi has been proven to be a bit brittle. There are various reasons, it can be related to power, the Raspberry Pi USB 3.0 controller behaving a bit "different", USB SSDs (adapters) which just violate spec etc. etc.

In your case, it seems that the HAOS boot loader U-Boot has troubles finding the USB SSD. We updated U-Boot from 2021.04 to 2021.10 in 7.0. So maybe it has been introduce with that, and you were lucky the first time you updated to 7.0...

[agners]

Can you share the full kernel log after a reboot (Log Provider: Host in Supervisor System tab)?

[mmalina]

OK, thanks for the details. So probably not much can be done, right? To be honest this is not the first time I had issues booting - it did happen in the past, but previously there would be no output in the console at all.

I rebooted just now and the log is attached here:
boot.log
.

[agners]

To be honest this is not the first time I had issues booting - it did happen in the past, but previously there would be no output in the console at all.

There was another change with OS 7.0: U-Boot now shows something on the screen (previously, U-Boot was not using the display at all). So this means that you had that problem always, but now you actually see that it is U-Boot which has issues detecting the USB SSD.

OK, thanks for the details. So probably not much can be done, right?

It would need digging into the U-Boot USB stack and see what fails exactly, and try to figure out if we can fix it/work around it. But I need to buy the same hardware and time to look into it. I might look into it at one point, so I'll keep that issue open. Maybe it gets fixed upstream, and a new U-Boot version will make it magically disappear 🤞 .

Thanks for the kernel log, the relevant information is here VID 152d/ PID 1561. This adapter is using a JMicron JMS561U two ports SATA 6Gb/s bridge.

[    4.068125] usb 1-1.1.2: new high-speed USB device number 4 using dwc_otg
[    4.279240] usb 1-1.1.2: New USB device found, idVendor=152d, idProduct=1561, bcdDevice= 2.04
[    4.286700] usb 1-1.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    4.294146] usb 1-1.1.2: Product: SABRENT
[    4.301526] usb 1-1.1.2: Manufacturer: SABRENT
[    4.308868] usb 1-1.1.2: SerialNumber: DB9876543214E
[    4.316998] usb 1-1.1.2: The driver for the USB controller dwc_otg_hcd does not support scatter-gather which is
[    4.321375] usb 1-1.1.2: required by the UAS driver. Please try an other USB controller if you wish to use UAS.
[    4.324800] usb-storage 1-1.1.2:1.0: USB Mass Storage device detected
[    4.328777] scsi host0: usb-storage 1-1.1.2:1.0

[mmalina]

There was another change with OS 7.0: U-Boot now shows something on the screen (previously, U-Boot was not using the display at all). So this means that you had that problem always, but now you actually see that it is U-Boot which has issues detecting the USB SSD.

I see, that would make sense. What's weird is that whenever this happens it tends to happen repeatedly until I leave the RPi powered off for a few minutes.

It would need digging into the U-Boot USB stack and see what fails exactly, and try to figure out if we can fix it/work around it. But I need to buy the same hardware and time to look into it. I might look into it at one point, so I'll keep that issue open. Maybe it gets fixed upstream, and a new U-Boot version will make it magically disappear

OK, let me know if there's any other way I can help with debugging. I could also try a couple more reboots to see if I can reproduce it like that at all. It would be a shame if you eventually got the hw and then failed to reproduce it :)

[kuroxtyan]

I got the same problem. My adapter name is Orico M2L2-N03C3

[iXPert12]

I have the same issue, also with an Orico enclosure. I have tested with the official raspbian distribution and it boots fine.

[kuroxtyan]

I have the same issue, also with an Orico enclosure. I have tested with the official raspbian distribution and it boots fine.

Yes, I,m trying raspbian-> works great! Problem with HAOS.

[piffio]

I believe this is the same issue reported in #1095.
In that issue I shared a detailed report of a debug session that points at some issues with U-Boot detecting the USB devices. See the details here.
I haven't tried yet with 7.2 but I'm assuming the same problem still stands

[agners]

@iXPert12 @kuroxtyan it is the bootloader U-Boot HAOS is using which causes problems. In Raspbian the Raspberry pi firmware boots Linux directly, and bypasses the U-Boot.

Now U-Boot is crucial to HAOS as it enables fallback in case of failed updates etc. a feature Raspbian doesn't has (if you apt-get fails, you will have to manually investigate).

But of course the automatic fallback relies on a working U-Boot 🙈

The problem is that the USB stack is complex. Linux and Windows have many work arounds for buggy hardware etc. U-Boot just isn't as widely tested in USB boot scenarious. That is why HAOS recommends to not boot from USB SSD directly, instead using a SD card to boot the OS and use the data disk feature to offload all data, the Core and all Add-ons to the USB SSD. This is much more reliable as only Linux will need to talk to the USB SSD (U-Boot will boot from the SD card directly).

[mmalina]

Thanks for the note, @agners . Either using external data disk was not supported yet when I installed HA in spring 2019, or I missed this information. Maybe one day I'll move to that setup (which would require a reinstall for me).

[iXPert12]

Ok, so after more testing,I found a way to boot up using usb boot with my Orico enclosure containing an Intel SSD. The first one, that worked also on a rpi zero 2 was powering the enclosure through a usb hub(which provides additional power). The second method that worked is connecting the enclosure with an Y cable(2 usb connectors, 1 for data and power and second for power only), and plugging in the power usb into an additional usb adapter before plugging in the rpi(with the data usb connected). It seems that rpi doesn't provide enough power for the enclosure to initialize the ssd correctly.

[github-actions]

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[mmalina]

I think this is still an issue.

[github-actions]

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[c05m1x]

The problem still persists. Raspberry Pi 3+ and HA flashed via RPI Imager onto a 240GB SSD.

[xdubx]

yes v 9.5 have the same problem.

[agners]

Make sure to use the official power supply and try with a powered hub in between to rule out power issues.

If that doesn't help, I'd suggest to use the data disk setup (using SD card to boot the system and move the data to the SSD).

[xdubx]

I tried many combinations it seems that the bootloader don't recognises the usb to sata adapter. I tried 3 adapter. With HAOS. Only my extern hhd as may backup drive gets detected under 'usb tree'. The other one lists as vendor specific and don't show any information about the type.

I will try on the setups rapsby lite to exclude the device combination does not work and post the kernel log here.

The funny thing i can boot the /boot section from the external ssd/hhd to get the CLI.

Make sure to use the official power supply and try with a powered hub in between to rule out power issues.

If that doesn't help, I'd suggest to use the data disk setup (using SD card to boot the system and move the data to the SSD).

yes the last step is the data disk. :(

[xdubx]

To be honest this is not the first time I had issues booting - it did happen in the past, but previously there would be no output in the console at all.

There was another change with OS 7.0: U-Boot now shows something on the screen (previously, U-Boot was not using the display at all). So this means that you had that problem always, but now you actually see that it is U-Boot which has issues detecting the USB SSD.

OK, thanks for the details. So probably not much can be done, right?

It would need digging into the U-Boot USB stack and see what fails exactly, and try to figure out if we can fix it/work around it. But I need to buy the same hardware and time to look into it. I might look into it at one point, so I'll keep that issue open. Maybe it gets fixed upstream, and a new U-Boot version will make it magically disappear 🤞 .

Thanks for the kernel log, the relevant information is here VID 152d/ PID 1561. This adapter is using a JMicron JMS561U two ports SATA 6Gb/s bridge.

[    4.068125] usb 1-1.1.2: new high-speed USB device number 4 using dwc_otg
[    4.279240] usb 1-1.1.2: New USB device found, idVendor=152d, idProduct=1561, bcdDevice= 2.04
[    4.286700] usb 1-1.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    4.294146] usb 1-1.1.2: Product: SABRENT
[    4.301526] usb 1-1.1.2: Manufacturer: SABRENT
[    4.308868] usb 1-1.1.2: SerialNumber: DB9876543214E
[    4.316998] usb 1-1.1.2: The driver for the USB controller dwc_otg_hcd does not support scatter-gather which is
[    4.321375] usb 1-1.1.2: required by the UAS driver. Please try an other USB controller if you wish to use UAS.
[    4.324800] usb-storage 1-1.1.2:1.0: USB Mass Storage device detected
[    4.328777] scsi host0: usb-storage 1-1.1.2:1.0

Its now some months later, but did you try to disable UAS in the cmdlist.txt as a kernel cmd?

[xdubx]

@agners raspberry os and lite boots without problem from the same setup. Its a problem with the HAOS bootloader / uboot whatever. Like mention before from kuroxtyan.

I add the hdd as data drive. I hope in the next releases someone fix the issue.

[github-actions]

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[erikstenow]

The issue stille exists on a brand new installation of HA

[erikstenow]

My startup screen

[dansmitt]

Still have this problem with my Raspberry PI 3b+

[thedalrok]

I also have this problem - started (I think) when updated HaOS to 11.2 (Raspberry Pi 3 model B).

[lvfmc85]

I have the same issue on a rpi3b. Any fix?

[ADX88]

Still same issue overhere, trying with rpi3b+ and a SSD giving same error.

[Sokar-PL]

Again, same issue here. 3,5 HDD + SATA Adapter + RPi3B+. I tried 2 different adapters and did not work...
Any workaround for this issue?

[gerhardkoepfle]

Sill same issue with rpi3b+ and SSD. All other OS and Software boots correctly from SSD, only HAOS does not.

[itbeyond]

I think the issue is a simple setting for uboot. If I boot my HAOS it shows 0 Storage Devices. I then execute

setenv usb_pgood_delay 4000
usb reset

it then shows 1 storage device found. It seems some drives take a little longer to come online and this setting this value will allow the device to boot the value can be between 1 - 40000ms, I found 4000 worked every time for my M.2 SSD drive.
After this execute

boot

Your system will boot as expected.

THE PROBLEM remains however that the setenv will not persist to the USB drives uboot enviroment (saveenv etc are not loaded). Does anyone know how to fix/persisit this detail as a power off reboot will require these cmds be entered again.

[fboulange]

I do have the same issue RPi3B+ with Ugreen USB to SATA adaptor using a Verbatim SSD.

It works but it is not consistent as usb_pgood_delay is not saved as @itbeyond mention above.

I've asked for some help in HA forum :

https://community.home-assistant.io/t/rpi3b-save-u-boot-env-to-boot-form-ssd/741301

[mmbakker]

I think the issue is a simple setting for uboot. If I boot my HAOS it shows 0 Storage Devices. I then execute

setenv usb_pgood_delay 4000
usb reset

it then shows 1 storage device found. It seems some drives take a little longer to come online and this setting this value will allow the device to boot the value can be between 1 - 40000ms, I found 4000 worked every time for my M.2 SSD drive. After this execute

boot

Your system will boot as expected.

THE PROBLEM remains however that the setenv will not persist to the USB drives uboot enviroment (saveenv etc are not loaded). Does anyone know how to fix/persisit this detail as a power off reboot will require these cmds be entered again.

Did this and, although it didn't solve my issue directly, it did give me the final clue to be able to solve my problem.

After running usb reset, I noticed it listed 0 storage devices. Tried both USB3 ports on my RPi4B 4GB, and both showed the same thing.

What I did prior to this is usb info 1 and usb info 2 (as mentioned by xdubx), which showed a USB hub (which in reality was the M2 drive) and a keyboard.

Since I have my M2 drive connected via a USB3 extension cable and the storage drive wasn't detected, I got to wonder if it would simply list my cable as the hub, but "couldn't see past it". That led me to replacing the USB3 extension cable to see if that would make a difference.

That fixed it for me.

[marksteward]

Same "0 storage device(s) found" issue with a standard SanDisk USB stick which doesn't pull significant power. The Raspberry Pi bootloader has an option to extend the timeout but I assume U-boot triggers its own re-enumeration.

[dansmitt]

To be honest guys, I don't think that anything happens on a closed case. We should either reopen it or create a new one.

[g0mml]

just as a FYI: Issue still persists (Raspberry PI 3B+)

[spockfish]

I'm having this issue with u-boot (outside of Home Assistant though) and had this resolved by putting

core_freq_fixed=1

in the config.txt

Maybe this can help out here as well.