ignition: systemd: fix systemd 256 usage

[ader1990]

Systemd 256 mounts /usr as read-only, but Ignition setup requires to have some files written depending on the live image or the ignition contents.

The are multiple options we can consider:

• this option, where we make /usr read-write and do the necessary changes
• another option would be to setup symlinks which redirect to /tmp or a writable file we can write
• another option would be to have the ignition binaries pre-baked by build_image

Testing done

[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[ader1990]

Just found out from @tormath1 (thanks!) that Fedora CoreOS had the same issue and implemented similar fixes: coreos/fedora-coreos-config@44d0ff8 coreos/ignition#1891

[chewi]

I'm confused. Doesn't this break verity? We therefore must have mounted /usr read-only before? I recall hearing about this before, but I've forgotten what was said.

[ader1990]

I'm confused. Doesn't this break verity? We therefore must have mounted /usr read-only before? I recall hearing about this before, but I've forgotten what was said.

dm-verity protects the /usr partition, not the initrd mounts. This PR is not doing anything special that was not done before aka now (the initrd /usr is writable), it just aligns with the new systemd expected workflow (the initrd /usr to not be writable).

But we still need to have a working ignition, so that's that. There is no other way until Ignition decides on another place to store that information.

[chewi]

If CoreOS has done it, then I guess it makes sense, but I'm still confused. /usr isn't a separate partition within the initrd, is it?

[ader1990]

If CoreOS has done it, then I guess it makes sense, but I'm still confused. /usr isn't a separate partition within the initrd, is it?

The /usr in this context is the initrd image partition (vmlinuz one). Not the /usr partition A or B (3 or 4) from the Flatcar disk image, that partition is read-only at filesystem layer and dm-verity protected.

[chewi]

Right, but the initrd covers all of /, not just /usr. If I extract it, then I see this:

lrwxrwxrwx  1 root root    7 Sep  4 12:26 bin -> usr/bin
drwxr-xr-x  2 root root    6 Sep  4 12:26 dev
drwxr-xr-x 10 root root 4096 Sep  4 12:26 etc
lrwxrwxrwx  1 root root   23 Sep  4 12:26 init -> usr/lib/systemd/systemd
lrwxrwxrwx  1 root root    7 Sep  4 12:26 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Sep  4 12:26 lib64 -> usr/lib64
drwxr-xr-x  2 root root    6 Sep  4 12:26 proc
drwxr-xr-x  2 root root    6 Sep  4 12:26 root
drwxr-xr-x  2 root root    6 Sep  4 12:26 run
lrwxrwxrwx  1 root root    8 Sep  4 12:26 sbin -> usr/sbin
-rwxr-xr-x  1 root root 4780 Sep  4 12:26 shutdown
drwxr-xr-x  2 root root    6 Sep  4 12:26 sys
drwxr-xr-x  2 root root    6 Sep  4 12:26 sysroot
drwxr-xr-x  2 root root    6 Sep  4 12:26 tmp
drwxr-xr-x  8 root root   81 Sep  4 12:26 usr
drwxr-xr-x  4 root root   51 Sep  4 12:26 var

And if I boot with rd.break=pre-mount then /usr is not a mount point and running mount -o remount,rw /usr gives an error. What am I missing?

[ader1990]

Right, but the initrd covers all of /, not just /usr. If I extract it, then I see this:

lrwxrwxrwx  1 root root    7 Sep  4 12:26 bin -> usr/bin
drwxr-xr-x  2 root root    6 Sep  4 12:26 dev
drwxr-xr-x 10 root root 4096 Sep  4 12:26 etc
lrwxrwxrwx  1 root root   23 Sep  4 12:26 init -> usr/lib/systemd/systemd
lrwxrwxrwx  1 root root    7 Sep  4 12:26 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Sep  4 12:26 lib64 -> usr/lib64
drwxr-xr-x  2 root root    6 Sep  4 12:26 proc
drwxr-xr-x  2 root root    6 Sep  4 12:26 root
drwxr-xr-x  2 root root    6 Sep  4 12:26 run
lrwxrwxrwx  1 root root    8 Sep  4 12:26 sbin -> usr/sbin
-rwxr-xr-x  1 root root 4780 Sep  4 12:26 shutdown
drwxr-xr-x  2 root root    6 Sep  4 12:26 sys
drwxr-xr-x  2 root root    6 Sep  4 12:26 sysroot
drwxr-xr-x  2 root root    6 Sep  4 12:26 tmp
drwxr-xr-x  8 root root   81 Sep  4 12:26 usr
drwxr-xr-x  4 root root   51 Sep  4 12:26 var

And if I boot with rd.break=pre-mount then /usr is not a mount point and running mount -o remount,rw /usr gives an error. What am I missing?

If you want to see what is actually happening, you need to add debug logs here: https://github.com/flatcar/bootengine/pull/101/files#diff-3c5d226e3099ab036b0015f5a5e8eb1bfcfff1537fd1d32e6b1b3f0799ea752eR10, build Flatcar and run it.
The mounts are managed by the dracut/systemd hooks.

I think rd.break=pre-mount is not early enough in the process to see the status of the system when the dracut/30ignition/ignition-setup.sh is run, but not familiar where it should be starting or ending, as the whole initrd unit order is a beautiful mess, that's why we have the ldconfig bug too. You can get a better picture on what happens at every boot using only systemd-analize plot and you will see that every boot is slightly different in the order of units stop-end, with just a few non-moving parts.

What is important to know is that /bin/is-live-image gets written in the RAM to be used by the ignition service running just afterwards. It is not written on the partition 3 or 4 (USR-A or USR-B).

[chewi]

I'll take your word for it, but I'll try to get a better understanding when I get back to tinkering with Dracut soon. Approving because CoreOS has made the same change.

[tormath1]

Right, but the initrd covers all of /, not just /usr. If I extract it, then I see this:

lrwxrwxrwx  1 root root    7 Sep  4 12:26 bin -> usr/bin
drwxr-xr-x  2 root root    6 Sep  4 12:26 dev
drwxr-xr-x 10 root root 4096 Sep  4 12:26 etc
lrwxrwxrwx  1 root root   23 Sep  4 12:26 init -> usr/lib/systemd/systemd
lrwxrwxrwx  1 root root    7 Sep  4 12:26 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Sep  4 12:26 lib64 -> usr/lib64
drwxr-xr-x  2 root root    6 Sep  4 12:26 proc
drwxr-xr-x  2 root root    6 Sep  4 12:26 root
drwxr-xr-x  2 root root    6 Sep  4 12:26 run
lrwxrwxrwx  1 root root    8 Sep  4 12:26 sbin -> usr/sbin
-rwxr-xr-x  1 root root 4780 Sep  4 12:26 shutdown
drwxr-xr-x  2 root root    6 Sep  4 12:26 sys
drwxr-xr-x  2 root root    6 Sep  4 12:26 sysroot
drwxr-xr-x  2 root root    6 Sep  4 12:26 tmp
drwxr-xr-x  8 root root   81 Sep  4 12:26 usr
drwxr-xr-x  4 root root   51 Sep  4 12:26 var

And if I boot with rd.break=pre-mount then /usr is not a mount point and running mount -o remount,rw /usr gives an error. What am I missing?

If you want to see what is actually happening, you need to add debug logs here: https://github.com/flatcar/bootengine/pull/101/files#diff-3c5d226e3099ab036b0015f5a5e8eb1bfcfff1537fd1d32e6b1b3f0799ea752eR10, build Flatcar and run it. The mounts are managed by the dracut/systemd hooks.

I think rd.break=pre-mount is not early enough in the process to see the status of the system when the dracut/30ignition/ignition-setup.sh is run, but not familiar where it should be starting or ending, as the whole initrd unit order is a beautiful mess, that's why we have the ldconfig bug too. You can get a better picture on what happens at every boot using only systemd-analize plot and you will see that every boot is slightly different in the order of units stop-end, with just a few non-moving parts.

What is important to know is that /bin/is-live-image gets written in the RAM to be used by the ignition service running just afterwards. It is not written on the partition 3 or 4 (USR-A or USR-B).

Giving my 2 cents here, but if is-live-image is the only breaking stuffs with this systemd change, what about moving it directly to the initramfs in the Ignition dracut module definition?

[ader1990]

Right, but the initrd covers all of /, not just /usr. If I extract it, then I see this:

lrwxrwxrwx  1 root root    7 Sep  4 12:26 bin -> usr/bin
drwxr-xr-x  2 root root    6 Sep  4 12:26 dev
drwxr-xr-x 10 root root 4096 Sep  4 12:26 etc
lrwxrwxrwx  1 root root   23 Sep  4 12:26 init -> usr/lib/systemd/systemd
lrwxrwxrwx  1 root root    7 Sep  4 12:26 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Sep  4 12:26 lib64 -> usr/lib64
drwxr-xr-x  2 root root    6 Sep  4 12:26 proc
drwxr-xr-x  2 root root    6 Sep  4 12:26 root
drwxr-xr-x  2 root root    6 Sep  4 12:26 run
lrwxrwxrwx  1 root root    8 Sep  4 12:26 sbin -> usr/sbin
-rwxr-xr-x  1 root root 4780 Sep  4 12:26 shutdown
drwxr-xr-x  2 root root    6 Sep  4 12:26 sys
drwxr-xr-x  2 root root    6 Sep  4 12:26 sysroot
drwxr-xr-x  2 root root    6 Sep  4 12:26 tmp
drwxr-xr-x  8 root root   81 Sep  4 12:26 usr
drwxr-xr-x  4 root root   51 Sep  4 12:26 var

And if I boot with rd.break=pre-mount then /usr is not a mount point and running mount -o remount,rw /usr gives an error. What am I missing?

If you want to see what is actually happening, you need to add debug logs here: https://github.com/flatcar/bootengine/pull/101/files#diff-3c5d226e3099ab036b0015f5a5e8eb1bfcfff1537fd1d32e6b1b3f0799ea752eR10, build Flatcar and run it. The mounts are managed by the dracut/systemd hooks.
I think rd.break=pre-mount is not early enough in the process to see the status of the system when the dracut/30ignition/ignition-setup.sh is run, but not familiar where it should be starting or ending, as the whole initrd unit order is a beautiful mess, that's why we have the ldconfig bug too. You can get a better picture on what happens at every boot using only systemd-analize plot and you will see that every boot is slightly different in the order of units stop-end, with just a few non-moving parts.
What is important to know is that /bin/is-live-image gets written in the RAM to be used by the ignition service running just afterwards. It is not written on the partition 3 or 4 (USR-A or USR-B).

Giving my 2 cents here, but if is-live-image is the only breaking stuffs with this systemd change, what about moving it directly to the initramfs in the Ignition dracut module definition?

That would be similar to the option 3, basically create the image with the vmlinuz that contains the correct switch executable /bin/is-live-image.

Maybe

bootengine/dracut/30ignition/ignition-generator

Line 89 in c9de038

if [[ -z "${usr}" && -f /usr.squashfs ]]; then

is the place to do such a thing, if the generator runs at build time.

[ader1990]

Merging this PR as is to have systemd 256 in this cycle.