Encryption at Rest (EAR) using e4crypt breaks after update to 2605.8.0

[ryanm101]

Updating from 2512.5.0 to 2605.8.0 breaks encryption at rest using e4crypt.

e4crypt get_policy /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/\[MYDatastore\]\ kubevols/kubernetes-dynamic-pvc-axx32d6b-xxx-xxxx-xxxx-0xxx45dc2.vmdk/encrypted
returns
Error getting policy for /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/\[MYDatastore\]\ kubevols/kubernetes-dynamic-pvc-axx32d6b-xxx-xxxx-xxxx-0xxx45dc2.vmdk/encrypted: Operation not supported

I'll provide more information hopefully over the next week as i narrow down what exactly happened and if we can go into 26XX at all. This is a placeholder so I dont lose the info over the weekend and incase anyone else has encountered the issue.

Rolling back the update fixes the issue.

[vbatts]

Thanks for opening the issue. We'll look to reproduce, so any additional information is appreciated

[ryanm101]

This breaks as soon as we go to 2605.5.0, so the break seems to be related to the change of kernel (or maybe docker)
We are using https://www.kernel.org/doc/html/v4.18/filesystems/fscrypt.html

Requirements:

• Seccomp -- pod policy needs to allow keyring-related syscalls.
• Key needs to be in session keyring for the processes that wish to use encrypted filesystem.

Below is the function we use to create the encryption:

set_key_and_link() {
    e4crypt add_key "$ENCPATH"
    # e4crypt get_policy always returns success (even when no policy)
    EXT4KEY=$(e4crypt get_policy "$ENCPATH" | awk -v vn="$ENCPATH" '$1 == vn ":" {print $2}')
    EXT4KEYID=$(keyctl show -x | awk -v key="$EXT4KEY" '$NF == "ext4:" key {print $1}')
    #
    # keyctl setperm <key> 0x<mask>
    #
    #    Possessor     UID      GID    Other Permission Granted
    #    ======== ======== ======== ======== ==================
    #    01000000 00010000 00000100 00000001 View
    #    02000000 00020000 00000200 00000002 Read
    #    04000000 00040000 00000400 00000004 Write
    #    08000000 00080000 00000800 00000008 Search
    #    10000000 00100000 00001000 00000010 Link
    #    20000000 00200000 00002000 00000020 Set Attribute
    #    3f000000 003f0000 00003f00 0000003f All
    #
    # - View permits the type, description and other parameters of a key to be
    #   viewed.
    # - Read permits the payload (or keyring list) to be read if supported by
    #   the type.
    # - Write permits the payload (or keyring list) to be modified or updated.
    # - Search on a key permits it to be found when a keyring to which it is
    #   linked is searched.
    # - Link permits a key to be linked to a keyring.
    # - Set Attribute permits a key to have its owner, group membership,
    #   permissions mask and timeout changed.
    #
    # Set permissions (0x1b = link, search, read, view) and link key to @u
    # keyring.  Pod containers that with to use encryption needs to link user
    # keyring to session keyring.
    keyctl setperm "$EXT4KEYID" 0x1b1b1b1b || { echo "^^^ - OK to fail if permissions already set."; }
    keyctl link "$EXT4KEYID" "@u" || { echo "^^^ - Troubles ahead. Missing/wrong key?"; }
}

Basically we have an init container that starts, mounts the PV, loads the key into the session on the node to decrypt the data and then the main container starts and only sees the decrypted data.

[ryanm101]

can anyone confirm the new kernel is built with CONFIG_FS_ENCRYPTION=y?

[pothos]

No, CONFIG_FS_ENCRYPTION is not set yet

[ryanm101]

Was it ever?

As the way of interacting with the encryption has changed between 2512 and 26xx, I wonder if this is either a new requirement as of kernel 5.xx or if you were building with that flag prior (my guess is more likely the former).

Could I request this flag be enabled? as it makes encryption more efficient and for me is a dependency preventing me from upgrading flatcar.

https://cateee.net/lkddb/web-lkddb/FS_ENCRYPTION.html

[jepio]

The 4.19 kernel in 2512.5.0 appears to have had these options enabled (zcat /proc/config.gz on a running system):

CONFIG_EXT4_FS_ENCRYPTION=y
CONFIG_FS_ENCRYPTION=m

[t-lo]

So the task at hand is to add the kernel config options outlined above (see kernel module section in the custom image docs), then file a PR to https://github.com/kinvolk/coreos-overlay/blob/main/sys-kernel/coreos-modules/files/commonconfig-5.10.

Labelling as "good first issue".

[ryanm101]

Reading: https://github.com/google/fscrypt#runtime-dependencies

For 4.1-5.0:

• https://cateee.net/lkddb/web-lkddb/EXT4_FS_ENCRYPTION.html
• https://cateee.net/lkddb/web-lkddb/EXT4_ENCRYPTION.html

For 5.1 onwards the correct setting is:

• https://cateee.net/lkddb/web-lkddb/FS_ENCRYPTION.html

My guess is that when it was cut over from 4.19 to 5.1+ the old deprecated CONFIG entries where removed but it was missed to add in FS_ENCRYPTION to replace them

[jepio]

@sayanchowdhury do you flatcar-archive/coreos-overlay#1212 is something we could cherry-pick into all channels?

[ryanm101]

@jepio Do you know when this is likely to make it to 'stable' ?

[jepio]

Should be in the stable release coming out this/next week.

[ryanm101]

Just grabbed it will be testing properly next week, thank you