Enable CephFS encryption for CephFS subvolumes

[humblec]

Describe the feature you'd like to have

CephFS make use of fscrypt based encryption or it will be available soon.
Ref# https://lwn.net/Articles/829448/

We have to think about the possibilities of integration for the subvolumes we provision and manage from CSI.

CSI requirements can be summarized to below:

*) Each PVC is mapped to a subvolume in the backend
*) Each PVC has to be encrypted with a different key
     I believe in the `fscrypt` case, we will have a custom protector for “CSI” which is common for all the PVCs but will have a “seperate” protector key for each subvolume

*) The keys will be stored in a KMS engine and retrieved at time of mounting the volume to a workload/POD
*) The key retrieved from the KMS engine will be used for encrypting and decrypting the volume

*) Operations like “expansion”, “snapshot” , “Cloning to a new subvolume” on an existing PVC ..etc has to work without hiccups. 
*) Once the snapshot or clone is in place the parent/source volume can be deleted at any time. IOW, no dependency exist to the source/original volume used for snapshot/clone .

I would like to capture the thoughts or would like to discuss on fscrypt+ cephfs implmentation against satisfying above requirements from Ceph CSI side.

@batrick @dillaman @vasyl-purchel @ShyamsundarR @JohnStrunk

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[laptimus]

Thanks Madhu for the prompt reply and reopening this issue.

We are at a point of making critical design choices for our application. We will be going with CepfFS hoping that KMS encryption support will be added soon to the CephFS provisioning and storageclass.

regards

[laptimus]

Hello Madhu

Any update on this issue

regards

[Madhu-1]

@jtlayton @batrick @vshankar @kotreshhr CephFS already supports fscrypt based encryption?

[jtlayton]

No, it's still a work in progress. We're hoping to have it ready to ship for Quincy release.

[laptimus]

Thanks @Madhu-1 @jtlayton

[laptimus]

@Madhu-1 @jtlayton - What is the bug number for this issue on tracker.ceph.com

thanks

[Madhu-1]

https://tracker.ceph.com/issues/46690 is the one, lets get confirmation from Jeff.

[irq0]

Hello!

I looked into how CephFS and FSCrypt could be integrated a bit and believe much of what is already there for RBD could be made to work for CephFS+FSCrypt as well. Biggest exception would be DEK storage.

I'd love to get started on a patch and like to hear if there is already something planned?

What I have in mind for a first version would be:

• Enable FSCrypt always on the volume root level.
• Focus on the 'metadata' KSM. Use Xattrs as DEK storage
• Either leverage https://github.com/google/fscrypt or https://github.com/google/fscryptctl to set keys and policies
• Where encryption is set up during CreateVolume, do this in NodeStateVolume to allow easy access to Xattrs

[humblec]

Hello!

I looked into how CephFS and FSCrypt could be integrated a bit and believe much of what is already there for RBD could be made to work for CephFS+FSCrypt as well. Biggest exception would be DEK storage.

I'd love to get started on a patch and like to hear if there is already something planned?

@irq0 That would be awesome 👍 , please give a try, we can also target this feature's first version in release 3.6. 👍

What I have in mind for a first version would be:

* Enable FSCrypt always on the volume root level.

Do you mean, subvolume or subvolumegroup in this context ?

* Focus on the 'metadata' KSM. Use Xattrs as DEK storage

That looks reasonable.

* Either leverage https://github.com/google/fscrypt or https://github.com/google/fscryptctl to set keys and policies

I would prefer fscrypt library than cli though or was exploring on that more than CLI.

* Where encryption is set up during CreateVolume, do this in NodeStateVolume to allow easy access to Xattrs

Yep.

Also, please feel free to have a design doc along with first POC implementation, that helps.

Please let us know if you need any help on this.

[irq0]

Hello!
I looked into how CephFS and FSCrypt could be integrated a bit and believe much of what is already there for RBD could be made to work for CephFS+FSCrypt as well. Biggest exception would be DEK storage.
I'd love to get started on a patch and like to hear if there is already something planned?

@irq0 That would be awesome +1 , please give a try, we can also target this feature's first version in release 3.6. +1

My WIP branch is here: https://github.com/irq0/ceph-csi/commits/wip/fscrypt. Needs k8s config from my dev env repo https://github.com/irq0/dev-ceph-csi-fscrypt-config plus custom kernel and MDS as the feature isn't merged yet.

It is currently more or less a PoC. The main feature, unlocking volumes, is there and stores encrypted DEKs in an xattr.
It shells out to fscryptctl quite a bit. I'll polish it up and replace with the fscrypt go library next. I'll open a PR once that is done.

What I have in mind for a first version would be:

* Enable FSCrypt always on the volume root level.

Do you mean, subvolume or subvolumegroup in this context ?

Subvolume / k8s PV

* Focus on the 'metadata' KSM. Use Xattrs as DEK storage

That looks reasonable.

I'm not sure if there is a good alternative. I don't think there is something akin to the RBD volume metadata for subvolumes, or did I miss anything?

* Either leverage https://github.com/google/fscrypt or https://github.com/google/fscryptctl to set keys and policies

I would prefer fscrypt library than cli though or was exploring on that more than CLI.

Me too. I started with fscryptctl though, because it is way simpler to use.

* Where encryption is set up during CreateVolume, do this in NodeStateVolume to allow easy access to Xattrs

Yep.

Also, please feel free to have a design doc along with first POC implementation, that helps.

Please let us know if you need any help on this.

Thanks! I'm a bit unsure on how to best test this, as most of the functionality crosses the system border towards the OS. Any ideas?

[humblec]

Thanks a lot @irq0 for the revert and sharing the gist of what you have been experimenting. We got to discuss about this in 3.6 release triage call yesterday and few thoughts on how to proceed further. It would be good to start with a design doc of this feature and get opinions/suggestions about the approaches from CSI and Ceph Team . It was also told that, we may have to wait for CephFS merge and kernel releases to make sure we are taking right approach towards whats going to land finally as a cephfs encryption solution. With all that, would it be possible to propose a design doc, so that we can discuss further and validate the possibilities of this feature in 3.6.
@nixpanic anything else to add which I missed here?

[irq0]

Will do. Should be ready mid next week.

I'll also want to explore integration possibilities with https://github.com/google/fscrypt some more. google/fscrypt and Ceph-CSI share quite a bit functionality around key management and there is probably some merit to making them compatible somehow. (The PoC code I wrote would require Ceph-CSI key management to unlock; google/fscrypt could allow multiple ways to unlock (e.g PAM, passwords, Ceph-CSI))

There aren't actually any hard CephFS dependencies here, since all the integration would do is key management. The nice thing about fscrypt is that this is independent of the filesystem below (see https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html). (So, what Ceph-CSI would do to unlock fscrypt on CephFS would work on ext4 as well)

[humblec]

Considering we have to wait for the dependency support in place and also some discussions on the design has to be concluded, removing this from the release 3.6 tracker. Thanks for the great attempt to take it forward.. we will continue the effort and will try to get this in release 3.7..

[humblec]

@irq0 Thanks for all the meetings or disussions on this topic and continuously making progress on the same. We will continue reviewing the new PR in place #3310 and track this for upcoming Ceph CSI release. I am untracking this feature from release 3.7 . 👍