-
-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use unmodified (blob-free) U-Boot for the Rock64 board #145506
Conversation
eae9654
to
fe85a24
Compare
I'd like to hear confirmation from Rock64 board owners, but it sure looks likely good given the patch you linked (and your experience). |
I've tested this on hardware and it looks good! I can boot a lightly-modified generic Aarch64 image using U-Boot from this branch on a 2 GB and a 4 GB Rock64. I get to login, automatic login works, and I get a working shell. Nothing seems off poking around for a minute. Steps I took:
|
(I don't know why I didn't think about it when reviewing initially) On an RK3399 system, I was able to produce bad results only under stress. Stressors included:
Either at once, or even (less reliably) independently. So maybe run a nixos system nix eval in a loop and run memtester in a loop for a while? I generally could crash that specific platform reliably in a few minutes. So probably a successful 20-30 minutes run could be a good indicator? |
I've cherry-picked this onto my RockPro64/Rock64 U-Boot PR: #146725 I'll see if I can run some of that testing, using U-Boot built from my branch. |
I had a bit of trouble with this. TL;DR:
Creating a bootable SD card as before but using
I had a random reset shortly after booting into Linux the first time, and a kernel oops the second time: Kernel output
Running
Running with the Versions: Rock64 v2.0 4 GB
|
This in itself doesn't look good. As far as evaluating, using |
I am getting some memtester failures:
I'll let this run for a bit longer, then try with the current master U-Boot to try and rule out my board being bad. |
Don't forget to test against the build with the proprietary ram training! |
Results from more testing:
I'll do a run with the 2 GB board and proprietary RAM training to complete the set, but this seems to mean one of these:
|
I wouldn't discount either outright. But considering the proprietary ram training fails it's odd. (To be fair, knowing which binary to use is an open question too, which is generally answered through a big dose of cargo-culting.) I guess it may help to know what issue @lopsided98's rock64 had, and if it was a 4GB variant. |
Yes, mine is 4GB. It just started locking up more and more often. I haven't had a chance to look into it, but I can probably do some testing in a few days when I come home from school. |
2 GB with proprietary RAM training: no issues. |
To add a data point, I ran the following tests on my Rock64 v3 (4GB RAM):
I ran it with both proprietary RAM training (uboot 2021.04 from NixOS 21.05) and upstream RAM training (uboot 2021.10 from this PR). No errors / crashes occurred. I ran it on NixOS 21.05 with kernel 5.10.79 though. I have two v3 boards (4GB and 1GB). @mirrexagon: are both of your boards v2? On the pine64 forum [1], [2] some people say that on the v2 version RAM instabilities are a known hardware issue. [1] https://forum.pine64.org/showthread.php?tid=11209 |
Quoting from one of the threads:
It would be interesting to figure out which RAM training is cargo-culted there.
Hmmm... What if I read more...
Oooh... - ./tools/mkimage -n rk3328 -T rksd -d ${rkbin}/rk33/rk3328_ddr_786MHz_v1.13.bin idbloader.img That could be it. Assuming this helps, anything we do won't make matters worse. But maybe there's a way forward for an alternative build. It makes no sense to harm perfs on the good machines. First "trivial" fix to try would be using the 333MHz training blob. Then, a "better" fix would be to patch the OSS training to do 333MHz... I looked quickly and it looks tricky... I wouldn't call it an OSS training, but more like a few blobby bits into OSS harness. Note that we don't need the miniloader. To recap:
We already have the first point done (here, rebased in the other PR). The other would need to be tested, if anyone's up to it. Hopefully that will help @lopsided98's board too, assuming v2 4GB. If all of this pans out, it might be warranted to add a note about it on the Pine64 wiki. (And the NixOS on ARM wiki pages.) |
Yep, both are v2.
On this, I tried quickly poking around Ayufan's U-Boot repos. In the vendor-fork-based U-Boot there is this: ifeq (rock64,$(BOARD_TARGET))
UBOOT_DEFCONFIG ?= rock64-rk3328_defconfig
UBOOT_TPL ?= tmp/rkbin/rk33/rk3328_ddr_786MHz_v1.13.bin
BL31 ?= tmp/rkbin/rk33/rk3328_bl31_v1.39.bin
BOARD_CHIP ?= rk3328 But in the latest upstream-based U-Boot [1] which seems to be used in the latest Linux image builds [2] doesn't specify this. It has a I was looking for a sign that Ayufan was using 333 MHz training, but I haven't seen it yet. I also noticed that (what I assume is) upstream rkbin only has 333 MHz and 400 MHz binaries for the RK3328 [3], and I can't find any 786 MHz ones in commit history. The RK3328 binaries are also up to v1.17. Ayufan's rkbin repo only has up to v1.13 [4], and the forum user was using v1.16.
I'll test with the 333 MHz training when I can. I'll pull the latest v1.17 binary from upstream rkbin, unless there's a reason to try a different version. [1] https://github.com/ayufan-rock64/linux-mainline-u-boot/releases/tag/2021.07-ayufan-2019-ga679a8ce |
My board is a 4GB v2.0. It was running U-Boot 2020.10 with the proprietary v1.13 768MHz RAM training. I know I have had problems (presumably memory errors) with this setup, but I just ran memtester overnight and it reported no errors. I also tested a few loops of memtester with the upstream memory training (this PR) and had no errors either. |
@lopsided98 it's entirely possible just memtester is not enough load. For the assumedly similar issue I had with the Helios64, it required other load in addition to cause issues. E.g. ZFS scrub was a surefire way to cause issues. CPU load seemed enough, e.g. evaluating the NixOS system. Load was assumed to be a factor as reducing the CPU speed and locking cpu frequency scaling helped (but never fixed) the issues. |
My Rock64 used to have a ZFS pool connected, which likely helped trigger the issues. Unfortunately, I moved that pool to my RockPro64 since I started experiencing issues with the Rock64, so it isn't easy to test anymore. I've been trying to reproduce the problem using stress-ng alongside memtester, but it has almost finished two successful memtester loops now. |
On my 4 GB board I ran the same test as before (
So, the way forward is to have two Rock64 U-Boot builds, one with upstream RAM training and one with proprietary 333 or 400 MHz RAM training? In my PR #146725, I'm exposing the combined upstream TPL+SPL+U-Boot Proper image So I'm wondering if it might be better to do the change in this PR so this can be merged first, and I can figure out combined images without holding this up. |
Considering the current build uses So we can merge this PR without thinking about the new variant. Unless I'm missing something? |
That's a good point, I think you are right. |
The main thing is now we know that for the non-marginal hardware this is still good, as it was. This is the scariest thing to get wrong. The failure mode can be extremely harmful, if it ends up corrupting in-memory data, instead of outright crashing. |
I added a quick note about the memory corruption to the NixOS wiki here: https://nixos.wiki/wiki/NixOS_on_ARM/PINE64_ROCK64#Status |
This PR was superseded by #191538 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/installing-nixos-on-rock64-2024-experience/40954/1 |
Motivation for this change
I think we can simplify the
ubootRock64
package and make it blob-free.Back then proprietary blobs were used because of "random memory corruption". I think that has been fixed in the meantime, maybe by these patches. I ran my two Rock64 boards with a blob-free U-boot for some months now and didn't encounter any issues.
The
extraConfig
can be removed too because it is in mainline u-boot now.Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)