systemd-boot support

[refi64]

From what I can tell, this is basically present (since the loader syntax is identical), except systemd-boot needs the loaders and kernels to be on the ESP (/boot/efi), not where OSTree places them currently (/boot)...

[cgwalters]

That sounds likely yes. I have no opposition to supporting sd-boot - the main question is how we detect it (actually the fact that we dynamically detect bootloaders today is a bit of a mess).

[refi64]

Maybe just check for its EFI file? GRUB is a little easily to have automatically installed accidentally forget about it, but systemd-boot is usually manually (and therefore consciously) installed.

[AdrianVovk]

I've run into this issue also. I'm trying to develop a distro around OSTree and I'd much rather use systemd-boot than grub for efi

[AdrianVovk]

@cgwalters Can the bootloader detection issue be solved with a simple config file? OSTree, when running ostree admin deploy can simply look for a file in /usr/etc or whatever and see which bootloader it should use.

Another possibility is to have the installer tell OSTree which bootloader to use. I'd prefer this option, since this gives the installer script the flexibility to decide on the bootloader, not the tree. This way, boot installation/config all happens in one place. Something like: ostree admin init-boot --sysroot=/mnt --os=foo systemd-boot

If the config file doesn't exist/command wasn't run, OSTree can (temporarily) fall back to the current behavior

[cgwalters]

For people who are building libostree from source (as it sounds like you are?), one thing we could pretty easily do is add a build-time option, like --with-bootloader=sd-boot or so. Would that be sufficient?

My main hesistation on this is where to put the config file, but thinking about it as I'm typing, we could add it as an entry in the repo config? Something like

[sysroot]
bootloader=<grub|sd-boot>

[AdrianVovk]

@cgwalters Yes, I am building libostree from source. An option like this would be sufficient. The problem with it is that it will restrict the entire system to a single bootloader. If the user prefers Grub and replaces sd-boot, suddenly OSTree deployments break. You'd need to recompile OSTree to boot with Grub. For minimal distros, though, it would be nice to trim down ostree by restricting bootloader support

Putting it in the repo config would be perfect! This also automatically gives a cli command, since ostree config exists. I would suggest maybe naming the option boot.loader or system.bootloader. ostree-sysroot.bootloader seems unwieldy...

[refi64]

FWIW ostree also does stuff like chmod the kernel file's permissions when it saves it to /boot, but that can't be done with systemd-boot (since it only supports kernels on the ESP, which is fat).

[AdrianVovk]

@kirbyfan64 can it not just skip that step? I've been manually copying things from /boot to /boot/efi without issue

[refi64]

It could, I'm just pointing out that it would also need to be changed

[AdrianVovk]

Any update on this?

[AdrianVovk]

@cgwalters I've been able to mess about with my distro and get some insight about the relationship between /boot, the ESP, and systemd

systemd, by default, really wants the ESP to be mounted to either /boot directly or /efi. If /efi exists, systemd will mount the ESP there. Otherwise, it will try to mount it to /boot. This is the behaviour of the systemd-gpt-auto-generator, which my distro will use to find and mount the rootfs. bootctl checks those two mounts points for an ESP also. It's safe to assume that any systemd-related tool will probably assume the EFI exists in either of these two directories (short of looking at /etc/fstab, which I don't intend to use)

I've been able to temporarily get OSTree working like so: I have a boot.mount unit which bind-mounts /sysroot/boot to /boot. OSTree can then operate on this directory normally. I created an /efi mount point (and systemd automagically mounts the ESP there) that I can copy files to as necessary. This, of course, is not an atomic layout. If something goes wrong while my distro's scripts are copying from /boot to /efi, I'm screwed. The system would no longer boot.

OSTree depends on symlinks in /boot, so it cannot operate directly on the ESP. If this dependency can be broken, however, it could be possible to work with the ESP in /boot. I think this can be done with the renameat2 syscall to atomically swap two folders, instead of using a symlink and rename. More info and initial patch. The new flow could go something like this: empty out /boot/loader.inactive/, generate the ostree config files, copy in non-ostree config files from /boot/loader, renameat2 to swap the two folders. This would give an atomic swap without relying on symlinks. If I'm not missing something, this would allow OSTree to work directly on the ESP. Steps like setting permissions on the files can be skipped or can error out silently, since vfat doesn't support permissions anyway AFAIK

If, for compatability, you need to keep the old behavior, you can translate all of this new behavior to the /efi directory. This new atomic swap would happen in /efi, and the old behavior would happen in /boot.

What are your thoughts?

[AdrianVovk]

PS: It would also be nice is OSTree could format its filenames for boot counting

[AdrianVovk]

@cgwalters I'm unsure of you got pinged. What are your thoughts on my suggested reimplementation of /boot handling?

[cgwalters]

There's extensive prior discussion of EFI and BLS and ostree - see e.g. #717

Let's keep this issue just about having the bootloader be configurable.

[AdrianVovk]

@cgwalters Certainly. That issue seems more grub oriented and less EFI but I moved the ESP discussion to #1649. Moreover, isn't the issue of supporting sd-boot really just an issue of supporting the ESP?

I don't see any problems with having a simple repo config option for determining which files to generate on deploy. Maybe you could even support having a list, and if absent the value would be grub,bls for backwards compatibility reasons

You could support sd-boot's bootcounting the same way. A config option would determine the number appended to the end of generated boot options

Like so:

[boot]
# Use the Boot Loader Standard
loader=bls
# Boot couting (appends +3 to names of generated bls files)
retry=3

[klausenbusk]

@cgwalters Can the bootloader detection issue be solved with a simple config file? OSTree, when running ostree admin deploy can simply look for a file in /usr/etc or whatever and see which bootloader it should use.

sd-boot can be detected by checking the EFI variable LoaderInfo, but using a config file is probably better.

$ cat /sys/firmware/efi/efivars/LoaderInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f 
systemd-boot 240

[fansari]

I used systemd-boot ("gummitboot") with "normal" Fedora in the past. As I remember this worked fine - but if you want to use "Secure Boot" you had to use this solution (also /boot must have vfat format which anaconda will not allow - so you have to do this as a manual step):

https://blog.hansenpartnership.com/linux-foundation-secure-boot-system-released/

As far as I understood secure boot has to been done in a complete different way wiht systemd-boot. You cannot just sign your kernel but have always add a hash signature to the MokList. I had one PC where this did not work. So I don't know whether this is a solution wwill be maintained in future.

And if systemd-boot is only usable if you disable Secure Boot then I would stay with grub. But only for this reason - otherwise I would always prefer systemd-boot.

[AdrianVovk]

@fansari you should research shim. It allows two different ways to do secure boot: the MokList, basically a list of hashes of EFI binaries that are safe to run, and with signed binaries. I think systemd-boot has little to do with it to be honest

Also, unless I'm missing something this is seemingly unrelated to the issue.

[bam80]

Just wanted to clarify the status. Can we have milestone for this or something, to track it better?

[gdonval]

I think this can be done with the renameat2 syscall to atomically swap two folders, instead of using a symlink and rename.

Looks like the prerequisites for #1967, proper atomic renameat2 on vfat, are released (Linux 6.0).

Also, this is not necessary if boot counting is leveraged: there can be always one known-good version present if done correctly. The install-kernel mechanics can be used to ensure that it is "done correctly".

Concurrently, unified kernels made their ways into the fray with the exact same boot counting semantics. One way ostree could become compatible with sd-boot at low programming cost without breaking legacy BIOS would be by generating (and optionally signing) unified kernels in the right place in the EFI partition.

[godvino]

I was able to switch to systemd-boot on my Fedora Silverblue 37 installation by changing the partition type of /boot to Extended Boot Partition from GNOME Disks and then running bootctl install. Seems to work fine without much problems.

EDIT: I had to also install the ext4 EFI file system driver from efifs into my EFI partition to get sd-boot to recognize my /boot partition.

[Apacelus]

I was able to switch to systemd-boot on my Fedora Silverblue 37 installation by changing the partition type of /boot to Extended Boot Partition from GNOME Disks and then running bootctl install. Seems to work fine without much problems.

EDIT: I had to also install the ext4 EFI file system driver from efifs into my EFI partition to get sd-boot to recognize my /boot partition.

@godvino can you please explain what exact files you got from efifs(they only have ext2 files?) and where exactly you put them?

[fwilhe]

Hey everyone, I'm looking to build ostree-based Garden Linux system, and for GL we absolutely require systemd-boot instead of grub. In this POC stage, building ostree myself (also from a non-main branch) would be fine, but in the long run it would be great if this was a standard feature of ostree.

Can someone give me some pointers on the current status of this? I'm not yet familiar with ostree internals, but I'd be okay to spend time looking into this.

[ericcurtin]

@fwilhe if Garden Linux does non-UKI bls-based booting, the contributions required to get sd-boot working should be minimal... UKI support is in progress...

Here are some build logs of how an ostree-based image is composed:

$ grep -r ^org /home/ecurtin/cs9-qemu-minimal-ostree.aarch64.qcow21689004435.txt
org.osbuild.rpm: b70f530d6b1be8917582b0a0954a794a362c7a9ecb9bf382b104de8d89392eb4 {
org.osbuild.selinux: d5923e844fb73034488de7afc9272faa012eaa37ef787e537482b86b993690a1 {
org.osbuild.kernel-cmdline: 7ca36414afb6caecc589d7cfc45df554f4fc564bae19b37583603fcee2ce93e0 {
org.osbuild.rpm: 6f84c8eb4c1b0fa2acd87a4a5b914c0003580cbccb36cf11b519e081ff6e3be9 {
org.osbuild.users: 9810dd71c449c9024bd66d3669409fba0423699f9c9416e6385327955161a5bb {
org.osbuild.copy: 0584fb734bc19cf1980b2b95cab83fa4e5b4f74c4318268ced3560e681697cac {
org.osbuild.copy: 91a00b60b379021facb209b992d527786546d9c20cde619b1ab49fda6f466b61 {
org.osbuild.copy: 2df84c1607f60fd3c8b8fa0f0fc2d72bc17e7c5fe9bca65334edb13f508d91a3 {
org.osbuild.copy: 7123cca9ef7d480074890ab67cfadee8a56509919ff77f27cbbf4c0a726033f9 {
org.osbuild.copy: 3caa68f347728ad7b97d21e42c3dde334e4faf98f2bb0ef965dc211b3c0602da {
org.osbuild.systemd: 1ac22bbe28e5c91fd01e10fa2063c4c5fce70b7ed3b5cd83ba8fdc20dcad156a {
org.osbuild.locale: ca757f358d5f774a24e3d937a8664a3b06ad139f7d791725ea43235f6c0f39c5 {
org.osbuild.timezone: 323d1cd75b91656395f5ca71278fe0074bfc77bf2d7e28a061db444854cc5104 {
org.osbuild.systemd: d4959ef15f9404ff3703624a7704385d247e4ed077d471051f59042e1d801481 {
org.osbuild.selinux: a928d2ed64eec73415f173d5d63aab4ca865df9b2bd6c6c228f157be45fc46af {
org.osbuild.ostree.preptree: 67f375c48c25dc80db72aa96316a599442f181644eb2fe4fdd41f401444f1e79 {
org.osbuild.ostree.init: 7f07c8e3e9a16005be1d6fd4aa96e145ae8df5778802f4866084f0bed400eee6 {
org.osbuild.ostree.commit: 12c3714f00d80841b37c8714557ae1c71bd5ef27f4030f1d8646d6527a817745 {
org.osbuild.ostree.init-fs: 86f86ed655e924a72a225bc733ff672e8c4ffd4084ca42baa5b0d2b1f1179828 {}
org.osbuild.ostree.pull: 3e1e1663ad3d3b96f441033b0c99a1782aec4bce151ccb9e0e9eb278374deef6 {
org.osbuild.ostree.os-init: 128baaecec7caa58bbd5d4451c829b4a069068ea769de7d595d1463b177a11da {
org.osbuild.ostree.config: f28de0a093fcd0097cfdcd6bb9d51c835ea9789d5d18b155c8bb1b26cc45cf80 {
org.osbuild.ostree.remotes: 3912cf37c5645ddd11f01bc4b0ab699dd012a5ad047e19df9d6d3be3e942265d {
org.osbuild.mkdir: 42006671b2cbf798742ae6f23ea95f3d9c7e53d5232a4c0d3e5a911b036f57a6 {
org.osbuild.ostree.deploy: 5eae0ddf0e97fcb919557b22338fe01085af937e40c6af491cdb03c8c910ea6c {
org.osbuild.ostree.fillvar: 088ee6e604baf27b0646449c824d6304059e3e739e3a6cbd685f87c2ec749eaa {
org.osbuild.users: 41a742ba8f0e38b4f5dc925a20a941afb373ccca74b10898fb59464b18e517fc {
org.osbuild.fstab: 2d2854bd7768cc5f1258071b14e2911984bdb727eb9f51347b8e3d448dae5fc9 {
org.osbuild.ostree.selinux: 2e2d2d50a3a4bf59c508c2269de87f4996b454ea49822f87bab8a79d379679f7 {
org.osbuild.grub2: ac084b7113588350a8743df4bc91d0d79ae89e98ba391b373e93fb680bcec5eb {
org.osbuild.truncate: b23f125b3b9f59f4baa8c99dc03a4e6cf54a60cf55c567bbe3e7042b5d0fb112 {
org.osbuild.sfdisk: 1fe455d5f6de781e5119eca4a407158a3060ea01a409eccbfda14eda8ee20085 {
org.osbuild.mkfs.fat: 69e92821135df3e140340ac5c5a454b2cc494bee6fbe8026581beff44aa84e13 {
org.osbuild.mkfs.ext4: 69e7c6d790e2a386882822cf14bd7a733601273447598060dee9822c5ef3fc10 {
org.osbuild.mkfs.ext4: 77f975f7a4b8227ea8a61a1f025bea1ed810f0ccf88d2a76b6d354d1069427d7 {
org.osbuild.copy: 7a7f46ae0d64cb09180608d0f221f9ff68d00ef804895a37e4b231892ef7f455 {
org.osbuild.qemu: 0af3461f90eaaad96b739e84e159b59c81414b116410ecff3e514c6009b65128 {

so I recommend looking at the ostree osbuild python script stages to get an idea of how to compose an ostree-based OS.

[fwilhe]

Hi @ericcurtin,

thanks for your hint, I did make quite some progress meanwhile and I got a booting system using systemd-boot. I did not expect this to work and it might actually not be working properly, at least that's my suspicion for the cause of issue #3022.

Assuming non-UKI setup (which is good enough for now), can you give me some pointers of what kind of contributions you mean? I did look into the bootloader specific code of ostree, but I'm not sure what systemd-boot specific code would need to do.

Best,
Florian

[ericcurtin]

@fwilhe I don't actually know what specifically needs to change, just gotta keep building ostree based images with sd-boot and see what isn't working I guess...

[GrabbenD]

I recommend looking at the ostree osbuild python script stages to get an idea of how to compose an ostree-based OS.

Which file are you referencing to @ericcurtin?

[ericcurtin]

The ostree python scripts here:

https://github.com/osbuild/osbuild/tree/main/stages

that's how we build ostree based OS's, the log I posted above describes the order in which they are called.

[ericcurtin]

There's probably grub assumptions you are gonna come across that need to be sorted out.

[ericcurtin]

This would need to be completed also #1967, not sure if @valentindavid wants to complete or if he wants to hand it off to someone else.

[valentindavid]

I will try to remember it. I will see if I can find time to rebase and address the directory version concern. But if someone else takes over, I am fine with it.

[fwilhe]

@fwilhe I don't actually know what specifically needs to change, just gotta keep building ostree based images with sd-boot and see what isn't working I guess...

Are there any existing tests/PoCs/example for that?

Not sure what it's worth, but I've put my debian/sd-boot based experiment here. It's booting, but it is not really functioning.

[p1u3o]

I was able to switch to systemd-boot on my Fedora Silverblue 37 installation by changing the partition type of /boot to Extended Boot Partition from GNOME Disks and then running bootctl install. Seems to work fine without much problems.
EDIT: I had to also install the ext4 EFI file system driver from efifs into my EFI partition to get sd-boot to recognize my /boot partition.

@godvino can you please explain what exact files you got from efifs(they only have ext2 files?) and where exactly you put them?

I haven't seen anyone reply to this. So I will document how I got systemd-boot working on Silverblue 39. So far, it seems everything works. Make sure secure boot is disabled.

# # Set your /boot partition to "Extended Boot Partition" in GNOME Disks (see https://i.imgur.com/Tr4Z41o.png)
# rpm-ostree install systemd-boot-unsigned
# rpm-ostree apply-live
# mkdir -p /boot/efi/EFI/systemd/drivers
# cd /boot/efi/EFI/systemd/drivers
# wget https://github.com/pbatard/efifs/releases/download/v1.9/ext2_x64.efi
# bootctl install

This has the benefit of also not putting kernel images on the efi partition, which is usually smaller than /boot. Grub will continue to work as well.

[bam80]

# # Set your /boot partition to "Extended Boot Partition" in GNOME Disks

After transitioning to a separate /boot partition I started to see this error:

$ bootctl 
Failed to open '/boot//loader/entries': Remote address changed
...

this is with systemd-boot or grub, regardless.
The system boots, though.

If I return to my previous /boot placement (on btrfs subvolume), I see no such error.

I have no idea where this error comes from.
Thoughts?

[gdonval]

Make sure secure boot is disabled.

I am personally not keen on adding attack surface to sd-boot by disabling a crucial component of verified (to-the-admin) boot and add complex code in the form of a ext2 driver.

I've had success mounting the ESP on /efi and use kernel-install to automatically create (signed) UKIs from ostree-controlled /boot binaries.

The problem is that it's very clunky at the moment and the boot-blessing mechanism is different from the bootloader specs'.

Actually, if fcos could create UKIs and use grub's own (undocumented) support for the bootloader spec, switching bootloaders would become nearly trivial.

[bam80]

# # Set your /boot partition to "Extended Boot Partition" in GNOME Disks

After transitioning to a separate /boot partition I started to see this error:

$ bootctl 
Failed to open '/boot//loader/entries': Remote address changed
...

this is with systemd-boot or grub, regardless. The system boots, though.

Any advice on this?

PS:
I'm not the only one with the error:
https://unix.stackexchange.com/questions/785355/persistent-systemd-logind-error-in-fedora-silverblue-failed-to-open-boot-loa

PPS:
I learned if I make /boot/loader/ as a directory, not a symlink - the error goes away.
The rpm-ostree doesn't accept /boot/loader/ as a directory, though - so we need a change here.

[bam80]

After transitioning to a separate /boot partition I started to see this error:

$ bootctl 
Failed to open '/boot//loader/entries': Remote address changed
...

systemd/systemd#35293

However, it might be an issue with OSTree itself, too.