Ubuntu 24.04 LTS/24.10 misses efifb in initramfs (upstream bug) #1641
Comments
|
@nestire thanks for reporting this. #1640 removed I will diff nitrokey 2.4.1 release linux config with heads master and #1640 further more. Meanwhile, can you post a screenshot of the kexec command line where the kexec leads to a black screen please? |
|
I will try to add kexec output to dmesg in that PR so that I can ask you to turn on DEBUG + TRACE configuration option and provide me logs to diagnose this better. |
|
@nestire fa70cba was added under #1640 After flashing, go under configuration settings, activate DEBUG+Tracing option, save in flash (will flash config back to flash) and then try to boot up to last prompt asking you if you really want to boot, on which please say no. You will land in recovery shell. put a supported FS based usb thumdrive, call And drop the log here please. |
#1642 (comment) ? There is no Lines 14 to 20 in 82179e4 |
I flash this image here are the logs : |
So FB from coreboot, discovered by Heads Kernel as efifb and passed down to next kernel in kexec -l not in cause.
All seems good. Will have to replicate. |
|
Hypothesises: |
|
As referred from other issue, that version of Ubuntu can be booted on x230. Not with a TPM Disk Unlock Key (DUK) from opened bug, but with Disk Recovery Key passphrase from Ubuntu initramfs to unlock LUKS container. This means that the "Heads black screen after kexec" stroked again. Will follow past hypothesis. Normally, Ubuntu, and all recent OSes provides efifb/simplefb instead of other legacy options, which you were able to get into at the installer. The the installer figures out what is available, loads other modules and keep trace of loaded modules that functioned to construct the initramfs to be deployed to /boot, which heads launches alongside of the kernel in its kexec callfor whatever reason here, if the installer worked, the troubleshooting path will be to see what gets loaded and what is included in the constructed initramfs that is then attempted to boot, and fails to drive the display post kexec call. |
|
lsmod_installer.txt modules.builtin_initrd_finished_install.txt So this might also be a ubuntu installer issue here? |
|
ubuntu install just worked for me. So It seems that they fixed it. Also just before booting on it in Heads I got something similar to this :
|
Did you mean same rom as @nestire ?
Or you mean latest non beta iso?
The vt kernel command line appended is the result of booting in unsafe mode and it to change console background color to red. |
|
@tlaurion I mean the same installation image of Ubuntu |
|
My bad, ubuntu 24 is still not working.. Yesterday I figured out that Installing without LUKS encryption will make ubuntu bootable but OEM factory reset will not work + errors with dongle |
|
Ok for further PR to fix this issue: From #1640 (comment)
|
@nestire Interesting. Will check your logs and leave proper debugging trace |
Analysis:
Result: on boot you see nothing after 12 seconds as per installer because nothing drives the framebuffer. @daringer @alexgithublab : Was there an option to install proprietary drivers that was not ticked in in the installer? |
|
Nothing much Heads can do here. Heads pass over the sysfb correctly (sysfb: VRAM smaller than advertised might be a coreboot bug) . But that doesn't prevent the installer, with firmware + drm + i915+xe to drive the display fb. Can you reinstall making sure firmware blobs are permitted and reply in this issue? Only thing I can think of is the "primary display" config setting under coreboot, but highly doubt this would fix this which seems to be initamfs not being packed with required drivers |
|
On nv41, Q4.2.1 dom0 And QubesOS stays in the dark because no efifb driver can do the sysfb->efifb takeover because Xen in the way and not figured out. On non-Xen, efifb from sysfb should take the console right away. But efifb doesn't seem to kick in here prior of i915 being completely fired up. |
|
Still not booting with the proprietary drivers installed |
@alexgithublab please provide same logs as @nestire here with blobs installed at installer phase. Statement: I'm trying to be as verbose possible so I don't have to fix, alone, everyone's issues from new platforms and all possible OSes, for all present and future sold platforms. Trying to leave traces for others to follow and be able to collaborate in code if not money. More collaboration is needed for this project to thrive, unless workis expected to be paid from profits on said sold platforms. "Free" as in free software should not be "free" as in free beer (from my bank account perspective). I'm trying to give a fishing line here, not to be confounded with consenting unlimited supply of free fishes forever. Thanks for your understanding. |
|
here are the logs of ubuntu install with drivers installed : dmesg-blob.txt |
…se 1.7.2, their fork's commit 3a9aa3a4692f3dd49732f5b4e3ec54be385f0969 nv41/ns50 coreboot configs bumped doing: make BOARD=nitropad-nv41 make BOARD=nitropad-ns50 coreboot.modify_and_save_oldconfig_in_place make BOARD=nitropad-nv41 coreboot.modify_and_save_oldconfig_in_place TODO Nitrokey: - stay as close as possible to dasharo+heads novacustom's paid dasharo coreboot fork to not duplicate work - test those configs, review those configs, push those configs upstream and collaborate upstream into issues under - https://github.com/Dasharo/dasharo-issues/ TODO Heads: - verify if cross-build toolchain could be shared to reduce CI overhead of building so many coreboot buildstacks - patches under patches/coreboot-nitrokey-clevo_release/ currently neglected because not applying clean - patches/coreboot-nitrokey-clevo_release/0001-dasharo-hardcode-configurations.patch - I see that coreboot KCONFIG option now permits to prefer S3. Sufficient? - patches/coreboot-nitrokey-clevo_release/0002-dasharo-hardcode-me.patch - EFI variable store patch still required? Might make bug vanish under linuxboot#1641, who knows. This is testing branch https://github.com/tlaurion/heads/tree/nitrokey_board_unification_clean-enable_htop_validated_autoboot-novacustom_coreboot_version_bump - https://github.com/tlaurion/heads/tree/nitrokey_board_unification_clean-enable_htop_validated_autoboot alternative, maybe better suited to resolve ubuntu issue, booting in debug with latest coreboot for novacustom heads+dasharo Signed-off-by: Thierry Laurion <[email protected]>
|
@alexgithublab : the gpu blobs are now present under installer's created initramfs?
@alexgithublab, if i'm not mistaken, @nestire logs at were extracted from /boot's initramfs, where your log seems to come from the /lib from within the installer's shell. The former helps but unfortunately not the latter. Nothing tells us that those were packed under initramfs and made available when the system boots. As asked under #1640 to @nestire : can we have a serial output here? the fact that we suffer from "black screen after kexec call" is problematic since we cannot get what the kernel is doing leading to not taking over the fb console. It happened in the past, sometimes plymouth/systemd takes for granted that for LUKS decryption key password, the console should be ready; you might have to type the LUKS password in the dark to get a working system to debug what is happening there. @alexgithublab you also said you installed ubuntu without encryption before. If that still the case? If so, this means that my previous comment can be discarded and we need a way to dump kernel logs somewhere if we cannot get serial output. |
|
@alexgithublab if this is testing latop, I would be interested to see if the roms currently built on CircleCI for alternative branch with coreboot+config+blobs versioned bump permits to boot your currently installed Ubuntu 24.04? Those roms are built under https://app.circleci.com/pipelines/github/tlaurion/heads/2490/workflows/3add7434-cf6c-4935-a1c6-f88cdc3005d0 for https://github.com/tlaurion/heads/tree/nitrokey_board_unification_clean-enable_htop_validated_autoboot-novacustom_coreboot_version_bump branch |
|
Cannot replicate failed build for ns50 locally with make Clearing CricleCI cache by setting |
|
Hey
This works also on our 2.4.1 Release What not Works:
I compared the initrd's of both installs and did not find any difference besides the added crypto modules What I will test next is if this also accours with if you upgrade from ubuntu 22.04. -> 24.04 and will come back with that information |
|
Ok did the Upgrade from 22.04 -> 23.04 -> 24.04 (directly to 24.04 is not working at the moment) the Kernel from 23.04 6.5.0-28 is working, the kernel 6.8.0-31 (24.04 hwe kernel) is not. |
|
@nestire now not sure how to tackle this but find what needs to be passed to board config through KERNEL_ADD additional statements to be passed at OS install so that equivalent of plymouth whatever loads kernel driver BEFORE the luks prompt.... Maybe you could share full Tldr: something fishy on Ubuntu side in the way they deal with FB readiness tests here, nothing Heads related unfortunately so prépare yourself to collaborate upstream (debian) or downstream (Ubuntu). They might have started their legacy bios deprecation. Expectation here would be efifb->simplefb->drm+fb->luks prompt, where at time of luks prompt, something is missing and why typing in the dark mitigated FB unreadyness. Best you can do is let the prompt sit for a couple of seconds, type in the dark, and provide full logs here for upstream discussions on source of problem. @nestire @alexgithublab you understand the problem? |
|
Here is the journal What I see is the: |
Yeah, that's a dead end. I see this in all boards being libgfxinit based as well as gop based FB init. So not thought to be source of the problem. As stated before, the fact that you can type the luks passphrase in the dark and then it works shows that it's not heads related issue but something missing in kernel_add board config or a bug upsream. What, I do not know. |
nestire
changed the title
@nestire as from 24.02.01 coreboot bump results you posted at #1723 (comment), and Nitrokey providing Ubuntu OEM images, this needs revisiting to support Nitrokey users. sysfb (coreboot) might be causing issues, not sure why on Ubuntu but not other OSes. This is coreboot <-> Ubuntu issues needing deeper troubleshooting where Heads is just using sysfb+efifb from GOP/libgfxinit so we need to bring both support communities together to resolve this issue. Willing to open bug reports upstream and cross-link them here for tracking so that Nitrokey user base has professional UX? |
|
@tlaurion yeah I'm up to put some more effort into this, but I'm also at a dead end after unsuccessfully testing some kernel options, and kernels (mainline also did not work) etc.. Also I don't find other people online who have the same problem so this seems limited to heads/coreboot. There is one issue that seems to be a similar problem in fedora 38 but I'm not sure if it is actually the same https://discussion.fedoraproject.org/t/luks-encryption-password-screen-not-showing/85683 . And I would need to test this which I did not yet find the time to do. I'm not convinced that we get canonical to act on this, but with coreboot this might be a different story. We will discuss this, and cross link the ticket for coreboot here |
|
naive question: did anybody crosscheck if other similarly "recent" kernels show the same behavior ? |
Heads or kexec'ed final OS kernel? Recap without rereading logs:
So something is not kosher on Ubuntu side of things in their legacy bios support with efifb->drm+3d FB takeover. At least, that's the current hypothesis. Maybe, I say maybe, kexec and/or coreboot fixed FB landover more nicely and/or Ubuntu Legacy BIOS deprecation plan went further ahead of other OSes, but intuition is something missing under initrd to play nicely since not affecting everything and typing LUKS in the dark, or setting DUK in Heads mitigate the whole issue. |
|
Ok based on this I had a look at the Kernel Config in Ubuntu and saw that solved it in part (plymouth did not start but that might have to do with how i build the kernel) thx @tlaurion for the summary that helped a lot |
Please cross link opened issue to this one and this one to upstream opened issue there and ping me where needed. |
|
ok the changes to this where discussed here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1965303 commented there instead of opening a new bug |
@daringer @nestire @jans23
|
|
@nestire ping |
|
@daringer @nestire @jans23 : ping Discussed at https://matrix.to/#/!RNcjJXCGHiyxXCHpKv:matrix.org/$YSYqj1K7qdJvbjT4NRvEYn4t3-9tl9vvyJdEoWHjOgs?via=matrix.org&via=nitro.chat&via=matrix.3mdeb.com (this is Dasharo Premier Support channel: accessible only to subscribers) End user asked what OSes were supported by Heads. What needs to be added under https://osresearch.net/InstallingOS/#generic-os-installation ? @nestire : bug to be opened upstream referring to #1641 (comment)
|
|
Pinged @nestire @daringer once more at Nitrokey/ubuntu-oem#14 (comment) |
|
I'm not in details but AFAIK this is a bug upstream in Ubuntu which is why I don't see how we can practically solve it on our end. |
|
TLDR: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1965303/comments/39 was good, but needed to be in separate ubuntu issue per their requirements; not just a comment on now upstream closed issue created. I reported stallness of upstream since on August 1st at #1641 (comment). Not sure what I can do more here without taking issue ownership/leadership until issue is fixed upstream, which I won't do anymore since work is unseen, unrecognized and unpaid. Which leads to shared leadership.... On shared leadership: Please take the lead and push upstream to fix with my thorough explanations at #1641 (comment). Let's work together pushing upstream to fix their non-compliances. As can be seen, this bug was raised downstream (Heads) on April 17th 2024 and as of October 29th 2024, is still not fixed nor moving forward. It could have been fixed in Ubuntu 24.10 if ball wasn't dropped. This is example of needed shared leadership to support ecosystem users, where Nitrokey ships unattended ISO which won't work out of the box without workaround (typing LUKS passphrase in the dark, setup TPM DUK) OR pushing upstream to fix their non-compliance, which will happen in their next release upon efifb addition in their initramfs. Issue opened upstream = easy fix . The sooner the better collaborating upstream as for everything. Let's make this issue an example of collaboration fixing upstream. Tag me where/if needed, refer to #1641 (comment) if needed and poke me here if my input is needed upstream. I do not use Ubuntu. I have no users depending on Ubuntu. Known issue, known fix. TLDR. TODO: @daringer @nestire please https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1965303/comments/40 output of prior attempt of reporting upstream and open new issue as upstream required taking #1641 (comment) as input for that new created issue. They need to pack efifb into initramfs. Easy fix but need poking upstream until done. Thank you, Maintainership of Heads: Please don't expect me to do this upstream/downstream canary in cold mine alone (anymore) for all existing OSes/toolstack ecosystem dead angles on compliance. I can explain things so that we can share leadership fixing upstream/downstream (not doing things correctly) to better it together, but those things (as can be seen with current at stake issue) don't fix themselves magically without someone taking lead until completion/ providing upstream bugfixes (meaning behind curtain work, involving unseen time/unpaid time needed to move things forward until compliant). TLDR: @nestire @daringer take #1641 (comment) as input and create issue upstream under unbutu, as said upstream under Time to shine so the ecosystem moves forward without too much dead angles impacting our users for years. Thanks. |
tlaurion
changed the title
....@daringer ping |
tlaurion
changed the title
After a successful install via USB Drive Ubuntu is not booting and hangs at the point of "Starting new Kernel".
It reacts to CRTL+ALT+ENTF but will not continue to boot in any case .
Tried also the "Unsafe Boot" Option. No error message is shown, also adding "-d" here did not bring up any message.
Test it with the Nitrokey Heads Release 2.4.1 and with the test build from #1640 on nitropad ns70/nv41/t430
The text was updated successfully, but these errors were encountered: