*: add LUKS #960

arithx · 2020-04-08T21:14:45Z

Still will need some coordination with ignition-dracut to re-open said devices (both on first-boot before Ignition files & on subsequent boots).

Additionally needs a udev rule similar to https://github.com/lucab/bootengine/blob/40a7918874ced2a6a4008dcab556f1eba61309a3/dracut/99cryptsetup/99-xx-coreos-systemd-cryptsetup.rules as systemd ignores unformatted crypto devices.

cgwalters · 2020-04-09T17:09:43Z

Still will need some coordination with ignition-dracut to re-open said devices (both on first-boot before Ignition files & on subsequent boots).

We could have this inject kernel arguments right? Then we can reuse the standard dracut crypt module.

Basically we just inject the usual root=UUID=<uuid of new rootfs> rd.luks=1 rd.luks.uuid=<uuid of crypt blockdev>.

Though of course this links together Ignition with dracut, but right now ignition-dracut is a separate project. But I don't know of any Ignition users that aren't using dracut today, and in the end any true successor is probably going to need to parse at least a subset of the same kernel arguments anyways to gain adoption.

cgwalters · 2020-04-09T17:10:41Z

As soon as we inject a root= kernel argument, that turns off all of our "standard" units: https://github.com/coreos/fedora-coreos-config/blob/e8abbe2b5e9c4c139937c450dc7d2d81bc907e5a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-mount-subsequent-sysroot.service#L8

cgwalters · 2020-04-17T14:36:22Z

Though of course this links together Ignition with dracut, but right now ignition-dracut is a separate project

Does anyone oppose merging ignition-dracut into ignition? Today it's pretty weird because the other projects we maintain like ostree and afterburn have them merged, and we really want to be able to do things like "add an ignition stage" as an atomic commit. Plus, in this PR I don't see how to avoid at least somewhat tying Ignition rootfs reprovisioning into dracut logic.

darkmuggle · 2020-04-17T14:48:06Z

Though of course this links together Ignition with dracut, but right now ignition-dracut is a separate project

Does anyone oppose merging ignition-dracut into ignition? Today it's pretty weird because the other projects we maintain like ostree and afterburn have them merged, and we really want to be able to do things like "add an ignition stage" as an atomic commit. Plus, in this PR I don't see how to avoid at least somewhat tying Ignition rootfs reprovisioning into dracut logic

I am a strong advocate of this idea.

jlebon · 2020-04-17T14:59:18Z

Does anyone oppose merging ignition-dracut into ignition?

This came up a few times already: coreos/fedora-coreos-config#70 (comment). We just didn't have the cycles at the time to weed out the few remaining CoreOS-specific bits from the repo. (To be clear: I do still think this is where we should be heading.)

darkmuggle

So if I'm following this right, then design is:

Ignition setups the LUKS volumes
The opening will rely on Clevis-Dracut and Dracut to do the opening? Or is there another Ignition-Dracut piece?

Overall this looks goods. I'm just unclear of the mechanics of how the volume gets opened.

internal/exec/stages/disks/luks.go

arithx · 2020-05-29T16:59:50Z

Ignition setups the LUKS volumes

The opening will rely on Clevis-Dracut and Dracut to do the opening? Or is there another Ignition-Dracut piece?

The initramfs on first boot works like so:

Ignition Disks creates the LUKS volumes, formats them, and closes the volumes.
Igniton Mount re-opens the devices (when we have OSTree logic for moving root devices there will be a separate dracut unit that will handle unlocking it and it will stay unlocked).
Ignition Files runs (it writes entries for each LUKS device to /etc/crypttab)
Ignition Umount closes the LUKS devices after umounts.

Inside of the real root non-root devices will be unlocked by clevis-systemd & dm-crypt based on entries in /etc/crypttab.

Inside of the initramfs on subsequent boots the real root will be unlocked before sysroot is mounted. Only clevis devices would really make sense here (as non-clevis would imply that we're storing a key-file in non-encrypted /boot).

darkmuggle · 2020-05-29T17:33:54Z

Inside of the real root non-root devices will be unlocked by clevis-systemd & dm-crypt based on entries in /etc/crypttab.

Inside of the initramfs on subsequent boots the real root will be unlocked before sysroot is mounted. Only clevis devices would really make sense here (as non-clevis would imply that we're storing a key-file in non-encrypted /boot).

Okay, that's what I was afraid of. I think we need to be very explicit about the Clevis path here.

clevis-dracut uses the initique:

    inst_hook initqueue/online 60 "$moddir/clevis-hook.sh"                      
    inst_hook initqueue/settled 60 "$moddir/clevis-hook.sh"

So the boot path is:

Ignition Disks writes an /etc/crypttab
https://www.freedesktop.org/software/systemd/man/systemd-cryptsetup-generator.html creates a the udev rules for prompting on the console for a passphrase. We need to make sure that systemcd-cryptsetup-generator runs after Igniton Disks (or at least have Ignition Disks run it)
The hook then runs /usr/libexec/clevis-luks-askpass in an infinite loop waiting for either a Clevis Pin to unlock (network or TPM2) or a passphrase.

So that means that we are going to implicitly allow user-input on boot. During the RHCOS implementation of LUKs, the group decided that this path was not desirable. I'm not saying that its not the path that we should take, just making the point. My concern to this point is that the failure of LUKS to open will have users asking "what is the passphrase?" (And why the RHCOS implementation just used its own util to open up the device.

I think that we'll need to make sure that ignition-disks.service runs before cryptsetup-pre.target and before the dracut-iniqueue.service in order to avoid racing conditions for /sysroot rewrites (when we get there).

arithx · 2020-05-29T18:02:00Z

From OOB discussions it sounds like the plan going forward for real root clevis based unlocks will be something like:

drop clevis-systemd
make sure that we're not writing udev rules requiring user input automatically
write a real root systemd generator that looks for entries in /etc/crypttab that are in the form of %name UUID=%UUID none luks (this format implies they are clevis based devices) which will run clevis luks unlock

Some additional clarifications about /sysroot unlocks that came up in the OOB discussion:

First boot, there will likely be a new unit in ignition-dracut that runs sandwiched in between Ignition disks & the new ostree unit that would do the rootfs move.
Subsequent boots will be most likely be done inside of ignition-ostree-mount-sysroot

cgwalters · 2020-06-02T14:35:29Z

I don't think in any cases with this we should support prompting for a traditional LUKS passphrase. It basically just doesn't make sense for servers.

The MVP here is binding to a TPM 2 device right? Clevis/Tang/NBDE is definitely a very nice to have too.

arithx · 2020-06-03T06:47:00Z

I don't think in any cases with this we should support prompting for a traditional LUKS passphrase. It basically just doesn't make sense for servers.

Yeah, this implementation won't have any prompting.

The MVP here is binding to a TPM 2 device right? Clevis/Tang/NBDE is definitely a very nice to have too.

The MVP here is key-file based devices & TPM/Tang (via clevis).

cgwalters

Overall is looking sane! Thanks so much for working on this.

internal/exec/stages/disks/luks.go

internal/exec/stages/files/filesystemEntries.go

cgwalters · 2020-06-03T13:22:29Z

doc/configuration-v3_1-experimental.md

+    * **device** (string): the absolute path to the device. Devices are typically referenced by the `/dev/disk/by-*` symlinks.
+    * **_cipher_** (string): the cipher specification string.
+    * **_hash_** (string): the hash used in LUKS key setup scheme & volume key digest.
+    * **_keyFile_** (string): the contents to use as a key file.


This probably needs some warnings/docs. For the root volume, it really doesn't make sense right? For non-root volumes...hmm, I guess one could store key files in e.g. /etc/luks/$volumename or so, i.e. unlocking subsequent volumes is bound to the rootfs binding. That could be nicer than binding each volume to the tpm2 maybe in that if e.g. one chooses to later add a passphrase in addition to the tpm2 binding to allow moving the disk later, one would only need to do it for the root?

(There may be a "how to use keyfiles well" doc upstream but I didn't see one in a brief search)

config/v3_1_experimental/types/schema.go

arithx · 2020-06-10T06:21:21Z

Ignition LUKS Overview:

PR Links:
Ignition
Ignition-Dracut
fedora-coreos-config

Ignition now supports configuring LUKS devices that can be unlocked via either key-files, TPM2, or tang. TPM2 & tang based devices are handled internally via clevis while key-file based devices are entirely handled via cryptsetup.

First boot workflow:

Ignition Disks creates LUKS devices
- Every device has a key-file, if one was not provided to Ignition then one will be created (NOTE: this can only happen for clevis based devices). If the key-file was created via Ignition it will be removed, provided key-files will be persisted to the real root at /etc/luks/<device_name>
- Devices are closed when the Disks stage ends
A new script ignition-copy-keyfiles runs copying the key-files to the real root
Ignition Mount unlocks all LUKS devices before attempting mounts
Ignition Files runs, appends content based on LUKS devices into /etc/crypttab (for key-file based devices) & /etc/clevistab (for clevis based devices).
- /etc/crypttab format: <name> UUID=<device_uuid> <key_file_path> luks
- /etc/clevistab format: <name> UUID=<device_uuid> <is_net>, if <is_net> is _netdev then the resulting unlock unit will be generated targeting running before remote-cryptsetup.target instead of the default cryptsetup.target (NOTE: the mount unit for the filesystem will also need to contain Before=remote-fs.target, After=network-online.target and DefaultDependencies=no)
Ignition Umount runs and closes LUKS devices after unmounting filesystems

This set of changes will require updates to the Ignition specfile. In the ignition-dracut change set the ignition-firstboot-complete unit has it's directory moved (from systemd/ to systemd/system/) and a new generator unit ignition-clevis-generator (in systemd/system-generators/). Example change from my local setup:

%files
%license LICENSE LICENSE.dracut
%doc README.md doc/
%{dracutlibdir}/modules.d/*
%{_prefix}/lib/systemd/system/*.service
%{_prefix}/lib/systemd/system-generators/*

&

%install
# ignition-dracut
install -d -p %{buildroot}/%{dracutlibdir}/modules.d
install -d -p %{buildroot}/%{_prefix}/lib/systemd/system
install -d -p %{buildroot}/%{_prefix}/lib/systemd/system-generators
pushd %{dracutrepo}-%{dracutcommit} >/dev/null
cp -r dracut/* %{buildroot}/%{dracutlibdir}/modules.d/
install -m 0644 -t %{buildroot}/%{_prefix}/lib/systemd/system/ systemd/system/*
install -m 0755 -t %{buildroot}/%{_prefix}/lib/systemd/system-generators/ systemd/system-generators/*
popd >/dev/null

arithx · 2020-06-10T06:22:03Z

Updates pushed, marked ready for review.

arithx · 2020-07-08T22:11:15Z

Yeah, I think all testing should probably be in kola to avoid having to require a ton of crypto binaries on the blackbox host and we'll be blocked until either 3.2 stabilizes or we land experiemental Ignition versions into kola.

bgilbert

Nice work!

doc/configuration-v3_2_experimental.md

config/v3_2_experimental/translate/translate.go

config/v3_2_experimental/schema/ignition.json

doc/configuration-v3_2_experimental.md

arithx

Update pushed.

config/v3_2_experimental/schema/ignition.json

internal/exec/stages/disks/luks.go

arithx · 2020-07-10T03:30:07Z

internal/exec/stages/disks/luks.go

+		args = append(args, luks.Device)
+
+		if _, err := s.Logger.LogCmd(
+			exec.Command(distro.CryptsetupCmd(), args...),


Yes, at the moment it does not re-use devices. My thought was to add that in a follow-up.

For keyfile based devices it should be pretty simple to unlock the device and check it, would need to dive further in for clevis based devices.

doc/configuration-v3_2_experimental.md

config/v3_2_experimental/types/luks.go

doc/configuration-v3_2_experimental.md

internal/distro/distro.go

internal/exec/stages/disks/luks.go

config/v3_2_experimental/types/luks.go

internal/exec/stages/files/filesystemEntries.go

arithx · 2020-07-10T17:07:55Z

Updated!

config/shared/errors/errors.go

jlebon

LGTM, nice work!

jlebon · 2020-07-10T18:25:37Z

config/v3_2_experimental/types/schema.go

@@ -2,6 +2,12 @@ package types

 // generated by "schematyper --package=types config/v3_2_experimental/schema/ignition.json -o config/v3_2_experimental/types/schema.go --root-type=Config" -- DO NOT EDIT

+type Clevis struct {
+	Tang      []Tang `json:"tang,omitempty"`


Let's not block on this, but I think fetch-offline should probably also check whether any Tang pins are present and SignalNeedNet() in that case. We can do that in a follow-up though!

Hmm, and then the rootmap code could also check if LUKS devices in the root block device tree are Tang-pinned and adds rd.neednet=1 to the BLS for subsequent boots.

Whoops, yeah, forgot about fetch-offline.

The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503

The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503 Closes: coreos#1589

The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503 Closes: #1589

cgwalters mentioned this pull request Apr 17, 2020

multipath: racing condition coreos/ignition-dracut#154

Closed

gicmo mentioned this pull request May 14, 2020

LVM+LUKS: initial research into enabling support for it osbuild/osbuild#369

Closed

arithx force-pushed the luks branch 2 times, most recently from c6e3f17 to f069f2f Compare May 29, 2020 01:48

darkmuggle mentioned this pull request May 29, 2020

Revert "ignition-ostree-mount-sysroot" coreos/fedora-coreos-config#434

Merged

darkmuggle reviewed May 29, 2020

View reviewed changes

internal/exec/stages/disks/luks.go Show resolved Hide resolved

darkmuggle mentioned this pull request Jun 1, 2020

RFC: Merge coreos/ignition-dracut into coreos/ignition coreos/fedora-coreos-tracker#511

Closed

cgwalters reviewed Jun 3, 2020

View reviewed changes

sergio-correia reviewed Jun 9, 2020

View reviewed changes

config/v3_1_experimental/types/schema.go Outdated Show resolved Hide resolved

arithx force-pushed the luks branch 5 times, most recently from cdd77ce to 9402c81 Compare June 10, 2020 06:14

arithx marked this pull request as ready for review June 10, 2020 06:18

This was referenced Jun 10, 2020

*: add Ignition LUKS support coreos/fedora-coreos-config#466

Closed

*: add LUKS coreos/ignition-dracut#192

Merged

arithx force-pushed the luks branch from 9402c81 to f3adaeb Compare June 10, 2020 16:38

arithx force-pushed the luks branch from f4396d1 to c560b32 Compare July 9, 2020 02:45

bgilbert reviewed Jul 9, 2020

View reviewed changes

arithx force-pushed the luks branch from c560b32 to 0d09b88 Compare July 10, 2020 05:50

arithx commented Jul 10, 2020

View reviewed changes

arithx force-pushed the luks branch from 0d09b88 to 530b9a6 Compare July 10, 2020 05:54

jlebon mentioned this pull request Jul 10, 2020

Move to conditional networking coreos/fedora-coreos-config#426

Merged

bgilbert reviewed Jul 10, 2020

View reviewed changes

arithx force-pushed the luks branch from 530b9a6 to 62fc01e Compare July 10, 2020 17:06

bgilbert approved these changes Jul 10, 2020

View reviewed changes

config/shared/errors/errors.go Outdated Show resolved Hide resolved

arithx added 3 commits July 10, 2020 12:48

config: add LUKS to schema

882a65a

internal: add LUKS support

5bd63f3

doc: add LUKS fields to configuration document

b866d91

arithx force-pushed the luks branch from 62fc01e to b866d91 Compare July 10, 2020 17:48

bgilbert approved these changes Jul 10, 2020

View reviewed changes

jlebon approved these changes Jul 10, 2020

View reviewed changes

arithx merged commit ff94f87 into coreos:master Jul 10, 2020

This was referenced Jul 10, 2020

Encryption: All disks are belong to us coreos/fedora-coreos-tracker#287

Closed

LUKS: support custom clevis configurations #1019

Closed

jlebon mentioned this pull request Jul 13, 2020

fetch-offline detection for Tang based LUKS devices #1017

Closed

jlebon mentioned this pull request Jul 20, 2020

config/internal: add clevis override support #1031

Merged

arithx mentioned this pull request Aug 4, 2020

There are no docs for how to incorporate Ignition in a distro #649

Open

shosti mentioned this pull request Jan 29, 2021

[ignition] Add luks support flatcar/Flatcar#243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add LUKS #960

*: add LUKS #960

arithx commented Apr 8, 2020

cgwalters commented Apr 9, 2020 •

edited

Loading

cgwalters commented Apr 9, 2020

cgwalters commented Apr 17, 2020

darkmuggle commented Apr 17, 2020

jlebon commented Apr 17, 2020 •

edited

Loading

darkmuggle left a comment

arithx commented May 29, 2020

darkmuggle commented May 29, 2020 •

edited

Loading

arithx commented May 29, 2020

cgwalters commented Jun 2, 2020

arithx commented Jun 3, 2020

cgwalters left a comment

cgwalters Jun 3, 2020

arithx commented Jun 10, 2020

arithx commented Jun 10, 2020

arithx commented Jul 8, 2020

bgilbert left a comment

arithx left a comment

arithx Jul 10, 2020

arithx commented Jul 10, 2020

jlebon left a comment

jlebon Jul 10, 2020

bgilbert Jul 10, 2020

*: add LUKS #960

*: add LUKS #960

Conversation

arithx commented Apr 8, 2020

cgwalters commented Apr 9, 2020 • edited Loading

cgwalters commented Apr 9, 2020

cgwalters commented Apr 17, 2020

darkmuggle commented Apr 17, 2020

jlebon commented Apr 17, 2020 • edited Loading

darkmuggle left a comment

Choose a reason for hiding this comment

arithx commented May 29, 2020

darkmuggle commented May 29, 2020 • edited Loading

arithx commented May 29, 2020

cgwalters commented Jun 2, 2020

arithx commented Jun 3, 2020

cgwalters left a comment

Choose a reason for hiding this comment

cgwalters Jun 3, 2020

Choose a reason for hiding this comment

arithx commented Jun 10, 2020

arithx commented Jun 10, 2020

arithx commented Jul 8, 2020

bgilbert left a comment

Choose a reason for hiding this comment

arithx left a comment

Choose a reason for hiding this comment

arithx Jul 10, 2020

Choose a reason for hiding this comment

arithx commented Jul 10, 2020

jlebon left a comment

Choose a reason for hiding this comment

jlebon Jul 10, 2020

Choose a reason for hiding this comment

bgilbert Jul 10, 2020

Choose a reason for hiding this comment

cgwalters commented Apr 9, 2020 •

edited

Loading

jlebon commented Apr 17, 2020 •

edited

Loading

darkmuggle commented May 29, 2020 •

edited

Loading