Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dual boot support for bootupd with static GRUB config installations #530

Open
AdamWill opened this issue Jan 24, 2024 · 37 comments
Open

Dual boot support for bootupd with static GRUB config installations #530

AdamWill opened this issue Jan 24, 2024 · 37 comments
Labels
enhancement New feature or request f41 Related to Fedora 41 fedora-change Needs a Fedora Change kinoite Also affect Fedora Kinoite rawhide

Comments

@AdamWill
Copy link

AdamWill commented Jan 24, 2024

bootupd & static GRUB configs don't support dual boot yet.


Previous title: Install of Silverblue installer image with bootupd enabled fails due to grub2-mkconfig failure

Describe the bug
After https://pagure.io/workstation-ostree-config/c/ad61c4d56e45eef8a7ac8af47f72fb4f463892e9?branch=main landed and the openQA ostree installer test started generating ostrees with bootupd, install of the Silverblue image built around the ostree started to fail. The ostree admin instutil set-kargs command that is run to configure the kernel args on the bootupd path fails, apparently because a grub2-mkconfig command it runs fails. It does not tell me why the grub2-mkconfig run fails.

To Reproduce
Please describe the steps needed to reproduce the bug:

  1. Get or build a Silverblue installer image containing an ostree with bootupd enabled, like this one
  2. Try and install it. Observe the failure during post-install bootloader configuration

Expected behavior
The install should complete successfully.

Additional context
I tried reproducing the issue locally, then switching to a VT and running the failing ostree admin instutil set-kargs command locally, after chroot /mnt/sysroot. That does indeed fail, with the error error: Bootloader write config: grub2-mkconfig: Child process exited with code 1. It does not pass along anything about the grub2-mkconfig failure, and /mnt/sysroot is read-only so I can't do any hacks to try and get the output from grub2-mkconfig dumped anywhere, really. If I run the ostree command with -v it gives me more verbose output before the grub2-mkconfig command fails, but nothing more useful about the actual failure:

OT: using fuse: 0
OT: Deployment (long id) unlocked=0
OT: Using bootloader configuration: auto
OT: Using bootloader: OstreeBootloaderGrub2
OT: Using grub2-mkconfig chroot: (null)

@travier has reverted the bootupd change once more for now, but we should figure out the failure so it can be turned on. Again.

@AdamWill AdamWill added the bug Something isn't working label Jan 24, 2024
@travier
Copy link
Member

travier commented Jan 25, 2024

Looks like it failed because it used a previous commit without bootupd installed thus anaconda could not detect it and use the new bootupd aware installation method.

We'll give it a try again now that the compose build failures should be fixed.

@travier
Copy link
Member

travier commented Jan 25, 2024

@travier travier added rawhide f40 Related to Fedora 40 enhancement New feature or request kinoite Also affect Fedora Kinoite fedora-change Needs a Fedora Change and removed bug Something isn't working labels Jan 25, 2024
@AdamWill
Copy link
Author

No, that diagnosis definitely does not sound right.

I checked the last pass and first fail in openQA when this started happening, and bootupd being present in the ostree is exactly the difference: bootupd was not in the ostree in the last pass, and was in the ostree in the first failure.

anaconda uses exactly the presence of bootupd in the ostree as its basis for deciding whether to go down this path at all: rhinstaller/anaconda@8e690d5

if bootupd isn't there, it won't go down this path, it will use its conventional bootloader install code.

And the openQA test builds an ostree before it builds the ostree installer image. It cannot really 'use an old ostree' like the compose process can, it doesn't have any. The only ostree it can use is the one it builds. If that step fails, the test will die and never reach the ostree installer build phase.

@AdamWill
Copy link
Author

The ISO I uploaded for testing is from this openQA test. If you check its ostree build log, you will see bootupd listed there.

@travier
Copy link
Member

travier commented Jan 26, 2024

Ah thanks, I had missed that this was from an image built during the openqa test run. Then this likely means that we'll have to do another Anaconda fix. CC @jkonecny12

@jkonecny12
Copy link

I see. I wonder how to resolve this. I guess we need to add a check if it is container source?

@jkonecny12
Copy link

I wonder @travier should we use the bootupd for the ostree standard installation or only for containers?

@jkonecny12
Copy link

Also would it be possible to remove bootupd from the ostree base layer? What is the purpose.

We are currently over capacity, if possible we need to minimize our work.

@travier
Copy link
Member

travier commented Jan 29, 2024

I wonder @travier should we use the bootupd for the ostree standard installation or only for containers?

We don't have that option as the same manifests are used for both.

Also would it be possible to remove bootupd from the ostree base layer? What is the purpose.

We are currently over capacity, if possible we need to minimize our work.

The goal of https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd is to fix:

We've reverted the change so it's not in right now, but it's a core part of the Ostree Native Containers work.

@jkonecny12
Copy link

@AdamWill could you please provide us the ISO which were failing? We need something to debug the issue.

@AdamWill
Copy link
Author

I already did, in the initial issue description, where it says "Get or build a Silverblue installer image containing an ostree with bootupd enabled, like this one".

@jkonecny12
Copy link

I see now, I guess I'm just blind :( . Thanks!

@travier
Copy link
Member

travier commented Jan 31, 2024

DEBUG:anaconda.modules.payloads.payload.rpm_ostree.rpm_ostree:Internal mounts are set to: ['/mnt/sysroot/usr', '/mnt/sysroot/dev', '/mnt/sysroot/proc', '/mnt/sysroot/run', '/mnt/sysroot/sys', '/mnt/sysroot/var', '/mnt/sysroot/boot', '/mnt/sysroot/home', '/mnt/sysroot/sysroot']
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-PrepareOSTreeMountTargetsTask-1 (140269177538240)                                                                                                                                                                                                                        
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/10.                                                          
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/11.                                                                                                                                                                                                                          
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/12.                                                                                                                                                                                                                          
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-CopyDriverDisksFilesTask-1 (140269177538240)                                                          
INFO:anaconda.modules.common.task.task:Copy driver disks files                                                                                                 
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-CopyDriverDisksFilesTask-1 (140269177538240)                                                             
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-ChangeOSTreeRemoteTask-2 (140269177538240)                                                                                                                                                                                                                            
INFO:anaconda.modules.common.task.task:Change OSTree remote                                                                                                     
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-ChangeOSTreeRemoteTask-2 (140269177538240)                                                                                                                                                                                                                               
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                
INFO:anaconda.modules.common.task.task:Configure OSTree bootloader
INFO:program:Running in chroot '/mnt/sysroot'... bootupctl backend install --auto --write-uuid --device /dev/vda /                                                                                                                                                                                                             
INFO:program:Installed: grub.cfg                                                                                                                                
DEBUG:program:Return code: 0                                                                                                                                    
INFO:program:Running in chroot '/mnt/sysroot'... ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw
INFO:program:error: Bootloader write config: grub2-mkconfig: Child process exited with code 1                                                                  
DEBUG:program:Return code: 1   
INFO:anaconda.core.threads:Thread Failed: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                                                                                                                                                                                
ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-ConfigureBootloader-1 has failed: Traceback (most recent call last):                              
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 280, in run                                                                                                                                                                                                                                      
    threading.Thread.run(self)                                                  
  File "/usr/lib64/python3.12/threading.py", line 1010, in run                                                                                                  
    self._target(*self._args, **self._kwargs)                                                                                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 94, in _thread_run_callback                                          
    self._task_run_callback()                                                   
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 107, in _task_run_callback                                           
    self._set_result(self.run())                                               
                     ^^^^^^^^^^                                                
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 505, in run                                                                                                                                                                                                  
    self._set_kargs()                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 568, in _set_kargs                                                                                                                                                                                           
    safe_exec_with_redirect("ostree", set_kargs_args, root=self._sysroot)                                                                                      
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 54, in safe_exec_with_redirect                                                                                                                                                                               
    raise PayloadInstallationError(                                            
pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw' exited with the code 1.

INFO:anaconda.core.threads:Thread Done: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                                                                                                                                                                                  
WARNING:dasbus.server.handler:The call org.fedoraproject.Anaconda.Task.Finish has failed with an exception:                                                                                                                                                                                                                    
Traceback (most recent call last):                                             
  File "/usr/lib/python3.12/site-packages/dasbus/server/handler.py", line 455, in _method_callback                                                                                                                                                                                                                             
    result = self._handle_call(                                                
             ^^^^^^^^^^^^^^^^^^                                                
  File "/usr/lib/python3.12/site-packages/dasbus/server/handler.py", line 265, in _handle_call                                                                                                                                                                                                                                 
    return handler(*parameters, **additional_args)                                                                                                             
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                             
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task_interface.py", line 114, in Finish                                                                                                                                                                                                             
    self.implementation.finish()                                               
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 173, in finish                                                                                                                                                                                                                       
    thread_manager.raise_if_error(self._thread_name)                                                                                                           
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 171, in raise_if_error                                                                                                                                                                                                                           
    raise exc_info[1]                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 280, in run                                                                                                                                                                                                                                      
    threading.Thread.run(self)                                                 
  File "/usr/lib64/python3.12/threading.py", line 1010, in run                                                                                                 
    self._target(*self._args, **self._kwargs)                                                                                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 94, in _thread_run_callback                                                                                                                                                                                                          
    self._task_run_callback()                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 107, in _task_run_callback                                                                                                                                                                                                           
    self._set_result(self.run())                                               
                     ^^^^^^^^^^                                                
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 505, in run                                                                                                                                                                                                  
    self._set_kargs()                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 568, in _set_kargs                                                                                                                                                                                           
    safe_exec_with_redirect("ostree", set_kargs_args, root=self._sysroot)                                                                                      
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 54, in safe_exec_with_redirect                                                                                                                                                                               
    raise PayloadInstallationError(                                            
pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw' exited with the code 1.

@travier
Copy link
Member

travier commented Jan 31, 2024

$ grub2-mkconfig
### BEGIN /etc/grub.d/15_ostree ###
/etc/grub.d/15_ostree: line 31: /etc/default/grub: No such file or directory

😕

@travier
Copy link
Member

travier commented Jan 31, 2024

[anaconda root@fedora /]# rpm -ql grub2-tools | grep default
/etc/default/grub
[anaconda root@fedora /]# rpm -qf /etc/default/grub
grub2-tools-2.06-110.fc40.x86_64

🤔

@travier
Copy link
Member

travier commented Jan 31, 2024

$ ostree ls --repo=repo 50ddc44f72905d9b463ffdf27119e670d7911629ae143644ab3809307a8746ff /usr/etc/default
d00755 0 0      0 /usr/etc/default
-00644 0 0    123 /usr/etc/default/useradd

🤔 🤔 🤔

@travier
Copy link
Member

travier commented Jan 31, 2024

OK, this is gone in latest Rawhide commits as well so this is not a bootupd issue.

@travier
Copy link
Member

travier commented Jan 31, 2024

Somehow, this file is not installed anymore: https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/grub2.spec#_436

@travier
Copy link
Member

travier commented Jan 31, 2024

Hum, it's not there on my F39 system either.

@travier
Copy link
Member

travier commented Jan 31, 2024

OK, it's anaconda writing this file in the GRUB2 installation path and likely not in the bootupd one: https://github.com/rhinstaller/anaconda/blob/c0d803edeeb8d7797492250353e52180f5ad6a33/pyanaconda/modules/storage/bootloader/grub2.py#L253

@travier
Copy link
Member

travier commented Jan 31, 2024

CC @jkonecny12

@travier
Copy link
Member

travier commented Jan 31, 2024

I suspect rhinstaller/anaconda#5350

@travier
Copy link
Member

travier commented Jan 31, 2024

OK, so we need to setup this config but only if the legacy ostree GRUB hooks/config are there (the ones that we want to remove after this change lands and people have time to update their bootloaders).

@travier
Copy link
Member

travier commented Jan 31, 2024

Or we can update this condition / check here: https://github.com/ostreedev/ostree/blob/main/src/boot/grub2/grub2-15_ostree#L31

travier added a commit to travier/ostree that referenced this issue Jan 31, 2024
With the new bootupd installation path in Anaconda, the
`/etc/default/grub` config file is not written anymore as we are only
using BLS configs with new enough bootloaders.

We thus don't need to generate (duplicated) legacy boot entries.

We still need to keep this logic in place in Atomic Desktops
(Silverblue, etc.) until we've actually landed bootupd there and forced
a bootloader update for everybody.

See: fedora-silverblue/issue-tracker#530
See: fedora-silverblue/issue-tracker#120
See: https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
@travier
Copy link
Member

travier commented Jan 31, 2024

Not 100% sure yet if my understanding is correct but made: ostreedev/ostree#3150

@AdamWill
Copy link
Author

you're correct that anaconda writes it, here. GRUB2.write_defaults() is called by GRUB2.write_config(), which is called by GRUB2.write(), which is called by InstallBootloaderTask.run(), using self._storage.bootloader as the bootloader class (so only when that is a GRUB2 instance). That's called by BootloaderModule.install_bootloader_with_tasks(), which has claimed to be "just a temporary method" since March 2019. That's called by BootloaderInterface.InstallBootloaderWithTasks(), which is called by RunInstallationTask._prepare_installation() (using an internal method). See? Simple...:P

The thing that is InstallBootloaderTask._storage is probably an instance of InstallerStorage, whose bootloader property is the result of BootLoaderFactory.create_boot_loader(), which via a slightly funky bit of logic usually winds up autodetecting the bootloader class via get_class_by_platform(): BootloaderModule winds up trying to do BootLoaderFactory.set_default_class("DEFAULT") via BootloaderType.DEFAULT, which the factory's get_class_by_name doesn't have a mapping for, so it returns None, which means BootLoaderFactory.get_class() falls through to get_class_by_platform()...all unless the user specifies extlinux or sdboot via configuration.

AAANYHOOO, what's actually happening here is that all of this is getting skipped because we hit a check in InstallBootloaderTask.run() (remember, that's the thing that calls write(), that calls write_config(), that writes the file). There's a check if self._payload_type == PAYLOAD_TYPE_RPM_OSTREE and have_bootupd(self._sysroot): which bails out of run() before it calls self._bootloader.write().

@travier
Copy link
Member

travier commented Jan 31, 2024

Thanks for digging 🕳️ into this!

@travier
Copy link
Member

travier commented Feb 1, 2024

I've verified that ostreedev/ostree#3150 fixes the installation

@travier
Copy link
Member

travier commented Feb 1, 2024

And I've tested basic bootupd function as well! Will try an update now.

@cgwalters
Copy link

Adding bootupd to a payload currently also defaults to --with-static-configs which I think is really the correct way to do things by default for dedicated Linux boxes. However...there may be people relying on more esoteric features of dynamic grub configs. For example, grub does this thing where it scans all your block devices and is like "Oh I see a Haiku install here, I'll make an entry for that" etc. And similarly for Windows I think.

But IMO, the right way to handle Windows at least is via the EFI bootloader, not grub.

Having new desktop (silverblue, etc.) installs use bootupd in this way will be awesome because it will align everything and work the same way as FCOS and how bootc does it.

But...transitioning old installs seems tricky and potentially risky.

@travier
Copy link
Member

travier commented Feb 1, 2024

Considering that we "already" have dual booting issues on Silverblue (#284), I'm tempted to say that we'll not fix this for Fedora 40. We should be able to document adding a Windows boot entry to the static GRUB config for new installations.

As we're not force upgrading users (yet), systems that were installed before will keep the dynamic GRUB configs for now.

@travier
Copy link
Member

travier commented Feb 5, 2024

I've tested installing Fedora Silverblue 37, updating all the way to 39 then rebasing to Rawhide and adopting using bootupd adopt-and-update.

@jkonecny12
Copy link

Great work @travier! Sorry that I wasn't able to help you more :(.

I wonder what is the correct behavior here... Should Anaconda create the /etc/default/grub or not? For OSTree based installations I would guess that this file shouldn't be created or it should be handled by bootupd?

@travier
Copy link
Member

travier commented Feb 6, 2024

For bootupd installations, we don't need anaconda to create this file right now as we will use a static GRUB config and BLS to get the ostree boot entries.

This will not generate boot entries for Windows, etc. for dual boot but we already don't have that working well so it's not a great loss.

I'll try to document how to manually add boot entries for Windows & Co.

@travier travier self-assigned this Feb 7, 2024
@travier travier changed the title Install of Silverblue installer image with bootupd enabled fails due to grub2-mkconfig failure Dual boot support for bootup & static GRUB config installations Feb 26, 2024
@travier travier changed the title Dual boot support for bootup & static GRUB config installations Dual boot support for bootupd with static GRUB config installations Feb 26, 2024
@travier travier added f41 Related to Fedora 41 and removed f40 Related to Fedora 40 labels Mar 18, 2024
@travier travier removed their assignment Apr 23, 2024
@travier
Copy link
Member

travier commented May 30, 2024

Related to ostreedev/ostree#3198 for GRUB static config + dual boot with composefs enabled.

@osalbahr
Copy link

osalbahr commented Oct 31, 2024

Similar error happened to me by following QA:Testcase dualboot with macOS with the Silverblue 41 ISO (using Fedora Media Writer) on an Intel-based MacBook Pro (2017).

The following error occurred while installing the payload. This is a fatal error and installation will be aborted.


The command 'ostree admin --sysroot=/mnt/sysimage deploy --os=fedora fedora:fedora/41/x86_64/silverblue' exited with the code 1.
Screenshot 2024-10-31 at 11 35 00 PM

@travier
Copy link
Member

travier commented Nov 5, 2024

Make sure to use a distinct, empty /boot & /efi partitions for the Silverblue installation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request f41 Related to Fedora 41 fedora-change Needs a Fedora Change kinoite Also affect Fedora Kinoite rawhide
Projects
None yet
Development

No branches or pull requests

5 participants