WIP: Enable initramfs-etc.img as an appended initramfs #364
Conversation
|
Note: this is missing updating the Throwing this up for an early review. I can see strong arguments against this, and I can see strong arguments to having Dracut as the enforcement mechanism. This is a bit more complicated than I'd like. To support this we'll need to change:
The envisioned user story is:
Finally, this is in rough shape, so it needs more work. |
|
I can see promise in this. Thank you for putting this together @darkmuggle. My initial question is around how |
|
/cc @imcleod too :-) |
|
It still seems a lot simpler to me to have as a kernel argument: And all our dracut module does is verify the checksum and unpack that file early on in boot. Today we're not verifying the kernel arguments, but we need to do so as part of securing the boot process, and there are already tools to do that. Inventing our own signing system seems like a large burden to carry. I guess to me what's important is that we come up a model where we think we can come by later and add signing, not that we do it right now. Having a separate "bootconfig.cpio" file that's separate from our code that we sign is part of ensuring we lay the groundwork for a future design there - it means e.g. that when we ship an OS update, the initramfs only needs to be verified with our key, any configuration (kargs, bootconfig.cpio) needs to be verified with the user key. Though in practice maybe we will end up with a model where we do something like this: https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images |
|
A tricky aspect to this is when we talk about signing, that is only relevant for UEFI, and we could e.g. actually use UEFI variables as Lennart was suggesting. But...we need a model for configuring multipath without signing since I am sure we need that for multipath/s390x which I am sure has something totally not UEFI. |
OK so we're rolling together the discussion on multipath (which is configuration) with kernel modules (which is code). I don't think most people will need to change the multipath config after install. But it's highly likely kernel modules will need to change - at least if they're not fully covered by kABI. We can clearly use an "overlay a cpio blob stored in /boot on the initramfs" model for both...I guess I'd hope that we at least make it harder for people to inject completely arbitrary code into our initramfs in the first case (e.g. I was imagining a whitelisted set of files so they don't inject It gets tricky if we want to cleanly support an "overlay initramfs" that's per kernel version in a clean fashion. Today ostree creates a |
|
Or to say this another way; the initramfs config and "rootfs kmods" are clearly related but also differ a lot in scope. For example, one can do a multipath install without caring at all about what the kernel version of the target OS is. But...trying to do early kmods for OpenShift will quickly get into the early OS upgrade and the fact there are (potentially) two kernel versions involved. The necessary code to inject for that is just much more complex - at least as long as one has a module not covered by kABI. |
Conceptually, I don't have a strong opinion about this either way. But there's a really interesting argument that if you're going to change the karg to The strongest argument for this approach is that there is no logic in Dracut -- the config is just there to start and there is no logic. And if we provide the sugar to make it, then we have a bit more control to drive the user behavior that we recommend. By using signed archives, we make it a bit harder for arbitrary code changes.
I think that's really orthogonal. If we're going to roll it all up and sign with their key, then this model can be easily adapted to support that and might be superior since the whole payload is there and signed by the user key.
The POC for
I started with Config only. But it quickly became apparent that it's extensible. I definitely think the two should be treated separately. With
For kmods, I 💯 think that we need to have a binding to it per OSTree Commit. I envision for RHCOS use-cases that we could use the SRO to drop the cpio file and then something else comes along and updates the
For config only (as this PR demonstrates) it can be version-less. For kmods, I think that it might be a bit easier -- we drop the |
| # Generate a real initrd, but we'll throw it away later | ||
| # This ensures that we get a "filtered" config, plus the | ||
| # extra's that we need. | ||
| dracut "${work_d}/work.img" $(uname -r) \ |
There was a problem hiding this comment.
@cgwalters I specifically use Dracut here to limit the arbitrary logic and just get the configuration that was needed. We get a config cpio that's going to be compatible with the machine and the major series.
Notice, the rm -rf /etc/systemd to clean out the logic.
There was a problem hiding this comment.
Note, this will break if/when we enable coreos/rpm-ostree#1789 - would need to directly invoke /usr/libexec/rpm-ostree/wrapped/dracut.
Definitely agreed with this. We should be re-using the infrastructure in place for trusted booting when we get there.
Hmm, this seems to be leaning in the opposite direction though, no? We're taking on the responsibility of checking the payload instead of re-using existing mechanisms to do that. For example, GRUB uses
I agree this is a strong argument for this approach. It may be multipath today, but there are lots more configs in there, and If we had to have only one mechanism for adding things without rebuilding the initrd, then IMO something transparent to dracut/systemd is cleanest. This is exactly what Ignition solves for the real root case too, so there's some beauty in symmetry there. ;) Re. |
Its worth noting that we're A) not inventing our own signing system. Grub has the support already for this; B) there are plenty of examples in the wild where people are doing secure boot signed Grub with GPG validation on the initrd. (https://wiki.archlinux.org/index.php/Secure_Boot#shim_with_key_and_GRUB) Switching to using a hash validation is trivial. https://www.gnu.org/software/grub/manual/grub/html_node/hashsum.html |
Ah thanks, I missed that! Definitely worth adding that
With the checksum we're not taking responsibility for any signing or key management - those are hard problems. Computing a checksum isn't. One aspect to this I'm thinking is that the later in the operating system boot we load any 3rd party code, the easier it is for us to e.g. report failures. If we use GRUB to check signatures, when it fails...what happens? The machine just stays dead at the console? Whereas if e.g. we can delay loading 3rd party initramfs config/code until the middle of our initramfs, ideally we have networking and can report failures to the cluster. |
In the WIP code here, here the user will see a message about the checksum failure. If we add
This is an interesting idea, but without precedence in the initramfs. And it's moving the goalposts. Today we don't have a clear secure boot story, we don't have networking except on the second boot, and we don't have an initramfs module for reporting back to the cluster failures. The linked issue would only apply to first boot for reporting failures. That's not to say that we can't move towards those goals, but they are orthogonal to this problem. For the 3rd part initramfs config/code, we could have a "failure" cpio that handles reporting back. For example, checksum fails, failure handler cpio is loaded and reports it back to the cluster. |
|
|
I see discussion, a link, and then closure. What is the outcome of this discussion? |
We did land coreos/ignition-dracut#146 and it's already useful in our CI and locally, and does set precedent for this. I wouldn't say the failure discussion is completely orthogonal once we talk about signing; today e.g. the UEFI Secure Boot failure scenario sucks. See also this tech-list thread. Now one can't easily have UEFI report failures somewhere because that gets into e.g. how you configure it to report somewhere. But if we can at least boot our verified kernel and initrd without user customizations, we can easily load a file in One further argument against doing this with GRUB is that we need to handle !GRUB bootloaders, namely But to counter that, I think it's also perfectly fine by me if our signing story only works on UEFI to start and we think about porting later. |
|
Back to the top though, I am trying to understand the expected flow here for the multipath case. All we need for multipath is the What I'm uncertain about here is how What if this logic lived in |
|
Let's keep the signing code but make it optional or commented out or something? When I first looked at this PR I just had difficulty thinking through the ramifications of the signing from the UX perspective; basically I don't think we can force everyone who wants to do a multipath install to deal with GPG key/passphrase management, particularly if we're not signing anything else. (I also went down a rabbit hole for a bit of understanding the signing infrastructure - it seems like the linked GRUB design requires re-generating the GRUB binary with the user key and enrolling it in the Secure Boot chain, which is basically AIUI how it has to work - e.g. with this code nothing stops someone from replacing both the |
In previous discussions there was concern for too much being in the initrd. Per https://github.com/coreos/fedora-coreos-config/pull/364/files#diff-ab205697e5620ab84e4e269a5708e943R41-R51 a currated white-list is added to support additional things.
From the live ISO, the user would:
The open question is whether or not the installer would pick up
There are three paths that have been proposed:
IMO, I would rather see it go into |
The tension from previous feedback is that we don't want to make it too easy for the user to use this, and @haraldh had suggested that we bind to the resulting configuration to the machine. Switching over hashes is trivial. Or we don't do anything. I'd just rather kill the code if there is no utility.
https://github.com/coreos/fedora-coreos-config/pull/364/files#diff-306b7a9e42fcd7f8e16823fa070a2c07R20 is the bit that results in trust. |
The idea here is to enable configuration needed during the initramfs without requiring the re-generation of it. Previous models including [1] were considered. As counter-idea, it was suggested to add to Dracut the ability to source in a signed CPIO archive or at least checksum it (e.g. boot-etc=<file>:sha256) and then rely on signed kargs. The chief concerns is that we need to ensure that the content is machine specific and not enable an easy boot-foot gun. This solution used Grub as the enforcement mechanism for both authenticity and authentication of the change and bypasses Dracut all-together. The core advantage is that admin could enable secure boot, set the Grub password (to prevent menu editing) and then Grub would only set "initramfs-etc='...'" if the file checked out. [1] dracutdevs/dracut#792 Signed-off-by: Ben Howard <[email protected]>
|
Repushed with changes for using checksumming instead. This is untested, though |
|
One random thought I had about this yesterday was that we could punt on this entirely and instead solve a different problem, which is to make it easier to run rpm-ostree to target the freshly installed system after doing an install but before reboot. That way, one can still use For example, coreos-installer could learn to mount the deployment root at
(And of course, this can be extended for whatever other firstboot thing one really needs.) |
Right, Ben mentioned that option above in passing too. To be clear I'm not opposed to it...but regenerating the initramfs locally feels like a step "backwards" towards being less of an image system by default. We went to some effort to get systemd patched to add a karg for the CPU affinity thing so we didn't need to regenerate the initramfs for that. On the flip side if we go the |
Ah hmm...I see. There may be some disconnect with what Harald is thinking here because we need to support multipath/iSCSI - it can't be too hard to use right? |
|
We did a video chat about this topic and the decision is was to satisfy the needs of the generic case (i.e. DASD) by using the The tricky part, though is that the growpart code needs to become device mapper aware in at least two places. In parallel we're going to look at doing the appended initramfs's in |
The idea here is to enable configuration needed during the initramfs
without requiring the re-generation of it.
Previous models including [1] were considered. As counter-idea, it was
suggested to add to Dracut the ability to source in a signed CPIO
archive or at least checksum it (e.g. boot-etc=/boot/foo.cpio:sha256) and then
rely on signed kargs. The chief concerns is that we need to ensure that
the content is machine specific and not enable an easy boot-foot gun.
This solution used Grub as the enforcement mechanism for both
authenticity and authentication of the change and bypasses Dracut
all-together. The core advantage is that admin could enable secure boot,
set the Grub password (to prevent menu editing) and then Grub would only
set "initramfs-etc='...'" if the file checked out.
[1] dracutdevs/dracut#792