Native encryption - request to allow raw keys on devices, (like USB flash)

[Lady-Galadriel]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	Rolling
Linux Kernel	4.9.34
Architecture	AMD64
ZFS Version	0.7.0-33_g08de8c16
SPL Version	0.7.0-12_g9df9692

Describe the problem you're observing

It would be nice if we could use a USB flash drive as the raw keysource. Meaning if the file was a device file, read only the first 32 bytes for the raw key.

This would be helpful when using a physical USB flash drive as a raw key. You only need it inserted during boot, pool import or dataset mount to un-lock the ZFS encryption. Afterwards, SysAdmin or Operator can remove the USB flash drive and lock it up. Any theft of the equipment, (disks or entire server), would then not include the raw key.

Of course the USB flash drive could have a real file system on it, with either a raw key file or a hex key file. Except that this then requires extra security to make sure normal users can't read that file system, and can't read the {raw/hex} key file. It would be more secure to use a partition and raw data from it.

Further, the same USB flash drive can have dozens of keys on it, for different datasets, pools or servers. All with their own partition.

Describe how to reproduce the problem

dd if=/dev/urandom bs=32 count=1 of=/dev/sdb1

zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///dev/disk/by-id/usb-USB_2.0_Flash_Disk_12345678-part1 rpool/encrypt_test

Include any warning/errors/backtraces from the system logs

cannot create 'rpool/encrypt_test': Raw key too long (expected 32).

[tcaputi]

This can probably be fixed pretty easily. I have another follow-up patch with bug fixes and additional tests and such which could definitely include this. I'm not really sure what the security advantage of using partitions for the keys instead of files might be, however. From a cryptographic standpoint this really doesn't matter and from a Linux permissions standpoint, you should be able to just chown it over to root and chmod it so other users can't read it. That might be a bit more practical, but I should be able to fix this regardless.

[Lady-Galadriel]

Since hex / raw keys are always 32 bytes, perhaps simply truncating any additional hex / raw byte values greater than 32?

This would then be documented behavior in the manual page. And take care of all cases equally.

[tcaputi]

Since hex / raw keys are always 32 bytes

This is true for now, but is not actually a limitation of the implementation. At the moment all wrapping keys are 256 bits simply because that is the strongest version of both the AES-CCM and AES-GCM encryption ciphers. In the future, when 512 bit encryption becomes more widely used we will want to be able to add those ciphers in without breaking existing code.

The more I think about this, the less sure I am that its a good idea.... For raw and hex keys, the only thing we really check to ensure the user is providing a valid key is the length. Theoretically, we could do some kind of entropy analysis to make sure they are using a proper key, but I don't think there is any plan for that at the moment. I'm a bit afraid of people who are new to encryption doing something like generating an RSA keypair and providing the public key to ZFS. This would be particularly bad because the first thing in an RSA public key is the extremely predictable -----BEGIN PUBLIC KEY----- header and the rest is in base64 which severely limits the valid character set for the remaining bytes.

I'm also still not sure what exactly the use case is here. The "correct" way to do this should just be to create and mount a filesystem on the thumb drive and point ZFS at a file on that mountpoint. Maybe if I understood this better we could come up with a better solution than just truncating the file.

[Lady-Galadriel]

@tcaputi, I understand. There is no hurry to make any change.

---

Basically the use case is in a data center. From a Unix System Administrator's point of view, whence I have to set up a procedure for operations staff, like a scheduled reboot, I need it to be as straight forward as possible. The procedure I outlined using a USB flash drive with un-mountable partitions requires little intervention from operations staff. They simply install the USB flash drive before the reboot and then check after the reboot that all is good.

Some places I have worked, had remote data centers. We did have people available on site, but they were either field engineer type, or operations type. Not SysAdmins. More than capable of installing a USB flash drive. But, not mounting a file system as they were un-likely to be able to login to the server(s). Let alone have priviledged access.

---

All that said, Linux, (and other OSes), do include auto-mounting of USB flash drives. So, at boot time, the server could automount a EXT4 file system from a USB flash drive, and then when the ZFS Pool with encrypted datasets on it is imported, it could automatically decrypt the datasets, making them available for the application.

The difficultly lies with removing the USB flash drive. It would have to be manually un-mounted, (or scripted to be un-mounted after the ZFS Pool & datasets are decrypted). This is so the USB flash drive can be secured elsewhere, (like a locked box), or potentially used by another server.

One last problem with a mounted file system is that even if the directory and key file have restricted permissions, if this USB flash drive is still mounted when backups kick off, then a copy of the encryption keys is now in the backups. Again, we can work around that problem by excluding the mount point from backups. But, mistakes can be made.

---

Basically I want the key to be easy to use, but restricted in access. I may have dozens of copies, (say 2 or 3 USB flash drives per data center to account for failures and simultanous reboots), but keep the master copies at the corporate office.

I have worked at sites that included simple confidential data, as well as US DOD secret data. Both can have requirements that decryption keys, (whether it's a passphrass or key file), be secured when not in use.

---

To be clear on a couple of points:
- Each USB flash drive can have multiple partitions / raw keys, (upto 128 using GPT)
- Some raw keys could be shared between servers, (HA cluster or similar apps.)
- A single server may use several different raw keys, (for different apps.)
- Devices & partitions can be found by UUID, thus simplifying ZFS source key path

---

Anyway, this is something that I thought would make OpenZFS native encryption easier to use in some production cases.

[tcaputi]

Are you aware of the capabilities of the "prompt" keylocation? This allows you to pipe the key to zfs load-key and its companion commands. For instance you can do something like this:

dd if=/dev/sdb10 bs=1 count=32 | zfs load-key pool/encrypted

It seems that in either implementation you will need to have a script (probably hooked into udev) to load the key upon inserting the usb drive (as long as you don't want the operations people to have to log in). This would allow you to have the script work exactly the way you need it to without zfs having to get rid of its sanity checks. Would that be a good solution?

[Lady-Galadriel]

Yes, I am aware of the prompt keylocation feature.

I think you are missing the main point. If the USB flash drive is installed and the keylocation references a partition device, no interaction is required during a boot sequence. Meaning someone plugs in the USB flash drive before reboot, waits til reboot is complete, then removes the USB flash drive to secure it. (Or re-use it on another server boot...)

No logging in by operations or application support staff to make the ZFS encrypted dataset available. No script or modifications to udev required.

The "wait til reboot is complete" may end up with a check of a monitoring tool to see if the application came up. (For applications that require ZFS encrypted datasets.) Or whatever other test may be required before removing the USB flash drive.

All that said, I am just trying to point out a use case with reduced interaction.

If it ends up being too problematic either now or in the future, SysAdmins will adapt to what is available.

---

Another way to deal with the issue could be to allow an option as this;

keyformat=raw|raw32|hex|passphrase

That would allow reading longer raw keys but also meet the use case I describe. Or even a more general purpose case of;

keyformat=raw{:SIZE}|hex|passphrase

Where SIZE is documented as 32 for the current implementation. I don't know if SIZE would be relevant to hex type keyformat.

[Lady-Galadriel]

The more I thought about this solution, the more I like it;

keyformat=raw{:SIZE}|hex{:SIZE}|passphrase

Tom's comment that we may have additional encryption key lengths would actually benefit from this above method.

Simple example. Assume someone starts using ZFS native encryption today with 32 byte keys. Later they upgrade ZFS which then supports newer encryption and 64 byte keys. Except the company wants to use the same key file for both. They don't have time to copy the existing encrypted dataset to one using newer 64 byte keys, (and perhaps no business case to do so). But, that's not the point.

With the ability to specify the length of the key, the above example could simply add an additional random 32 bytes to the existing 32 byte key file, (in binary or hex as appropriate). Then, have the old existing dataset use;

keyformat=raw:32    or    keyformat=hex:32

This gives the future users some flexability.

But again, no hurry to make any decision, let alone change.

[ikozhukhov]

hi, what is status of this update?

[tcaputi]

@ikozhukhov I never got around to working on this. As per the conversation above, I was never really sure what the best way to deal with wrapping key size would be.

[jameslikeslinux]

This already basically works as implemented because passphrases can be up to 512 bytes and the minimum partition size on a typical USB flash drive is 512 bytes. So you can effectively make a 1-sector key partition like:

> parted -s /dev/usbdevice mklabel gpt mkpart key 2048s 2048s

> parted -s /dev/usbdevice unit s print
Model: Unknown (unknown)
Disk /dev/zd112: 2097152s
Sector size (logical/physical): 512B/8192B
Partition Table: gpt
Disk Flags:

Number  Start  End    Size  File system  Name  Flags
 1      2048s  2048s  1s                 key

> dd if=/dev/urandom of=/dev/disk/by-partlabel/key
dd: writing to '/dev/disk/by-partlabel/key': No space left on device
2+0 records in
1+0 records out
512 bytes copied, 0.0428454 s, 11.9 kB/s

> zfs create -o encryption=on -o keyformat=passphrase -o keylocation=file:///dev/disk/by-partlabel/key rpool/crypt

[abbaad]

> dd if=/dev/urandom of=/dev/disk/by-partlabel/key

Be careful with this, as keyformat=passphrase uses getline() which will stop reading your file after encountering a newline character. Just putting random bytes into the raw partition might give you less key material than you think if there happens to be a newline early on.

Instead you might try stripping out the newlines:

# tr -d '\n' < /dev/urandom | dd of=/dev/disk/by-partlabel/key

[Malvineous]

Except that this then requires extra security to make sure normal users can't read that file system, and can't read the {raw/hex} key file. It would be more secure to use a partition and raw data from it.

Perhaps I'm missing something, but what extra security do you gain from storing the key as a raw partition as opposed to a filesystem owned by root and chmod 700? Even if you don't unmount it once the encryption key has loaded, nobody except the root user could read the keys anyway. And if you are the root user then you'll have access to the raw disks anyway (they likely have the same owner/group/perms). You also have access to read the ZFS properties to tell you exactly where the key is, so reading it would be trivial for the root user whether the key is in a file or a raw partition.

When you consider that root's SSH private keys and the authorized_keys file on most servers are protected behind chmod 700, if those permissions don't stop someone from getting your ZFS key file, then it also won't stop them from adding their own key to your authorized_keys file. That will allow them to SSH into your machine and copy your decrypted data, even if you completely remove the ZFS key from the machine after mounting the datasets!

So I just wonder whether it's worth going to the trouble of adding this functionality when it doesn't seem to provide any additional security.

But possibly I am just missing something here - please feel free to correct me if I've overlooked something.

[Lady-Galadriel]

@Malvineous, part of the advantage of using a raw partition for the encryption key, is that you can then use a USB flash drive. After boot & ZFS key installation, the USB flash drive can be removed and locked away. You can have several of the "key" USB flash drives, all with identical data. Different partitions for different servers or groups of servers.

One comment from someone said we could achieve the same thing by using a file system, (with 700 permissions and root owned), on the USB flash drive, with a key file. But, their is a problem with that. If it's mounted, the key file data could end up in backups. Plus, it has to be mounted and un-mounted when it's used on a USB flash drive.

Anyway, just minor details.

[alexsmartens]

Did anyone end up implementing native encryption with a key on a device?
Could you share your script if so

[Malvineous]

I don't think it's something you can easily share because it would be distro specific. All you have to do is run zfs load-key and supply the key, but the trick is doing it at the right point in the boot sequence.

If your root filesystem isn't encrypted then a start up script or systemd unit can do it, but if the root filesystem needs a key then you have to get your script to run as part of the initramfs system, which exists before the root filesystem has been mounted. In Arch Linux you can put scripts like this into /etc/initcpio/install/ that will get packaged up and run from initramfs, but probably every major distribution does this differently.

The reason for creating this GitHub issue was to provide a way where ZFS could read keys off certain devices like USB keys, without needing to alter the boot process, making it more robust when kernel updates are applied to a system - and allowing it to work the same way regardless of the Linux distribution in use.

[alexsmartens]

@Malvineous, thanks for your extended explanation.

The distro that I'm using is Ubuntu 20.04.

Although my use case is slightly different from using a usb, I'm seeing the same problem with configuring the boot sequence. In my case, I'd want to keep the key from my rpool on boot partition, whereas boot is ext4 encrypted with LUKS and it is decrypted with the key from TPM.

The reason for creating this GitHub issue was to provide a way where ZFS could read keys off certain devices like USB keys, without needing to alter the boot process, making it more robust when kernel updates are applied to a system - and allowing it to work the same way regardless of the Linux distribution in use.

Having this mechanism would be awesome!

[Lady-Galadriel]

In theory, the ZFS encryption key can automatically be loaded at boot, using something like this;

zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///dev/disk/by-id/usb-USB_2.0_Flash_Disk_12345678-part1 rpool/encrypt_test

The problem arises because the key length is limited to 32 characters. Since a partition is at least 1 sector in size, (512 bytes or larger), this does not work.

However, thinking about this further, perhaps using raw key format is not the way to go. Assuming we can take a hex string and write it with end of line to a raw partition, then perhaps that will work. Like this;

echo "" >newline
dd if=/dev/zero bs=1 count=1 seek=1 of=newline

dd if=/dev/urandom bs=32 count=1 2>/dev/null | \
  od -xw | \
  cut -b9- | \
  tr -d ' ' | \
  tr -d '\n' | \
  cat - newline >/dev/nvme0n1p6

zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///dev/nvme0n1p6 amypond/encrypt_test

Let me reboot and see if that will automatically load the key.

Note that there is most certainly an easier way to create an 32 byte random hex string with end of line. I just threw this together in seconds to test proof of concept.

Edit: Seems to work, though does not automatically load the key on pool import. Have to use -l with the Zpool import command.

[Malvineous]

Seems to work, though does not automatically load the key on pool import. Have to use -l with the Zpool import command.

What distro are you running? Arch Linux has a script that runs in the initramfs that will call zfs load-key for an encrypted root pool during boot, which makes the process automatic (or requires you to type in your password very early in the boot process).

That doesn't work for non-root pools however, as they aren't picked up until later in the boot process. In that case I had to add a systemd unit to load the keys - there are instructions on the Arch wiki which probably apply to any distro using systemd. The second example (for loading all keys) worked perfectly for me.