disk usage with discard option in encrypted rbd

[tareksha]

Hi,
We are trying to cope with a growth in ceph OSD usage that we have been experiencing since enabling encryption for all rbd-backend volumes that are provisioned via ceph-csi (rook installation).
Storage class:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: testsc
mountOptions:
- discard  # <-- IMPORTANT!
parameters:
  # .. omitted "csi.storage.k8s.io/*" params
  encrypted: "true"
  encryptionKMSID: appstudio-cmk
  imageFeatures: layering
  imageFormat: "2"
  pool: replicapool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Example volume claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: testpvc
  namespace: testns
spec:
  resources:
    requests:
      storage: 10Gi
  storageClassName: myencrypted
  volumeMode: Filesystem

I mount the PVC into a simple pod and check disk usage. First the volume is empty:

$ df -h /data
Filesystem                                                                                      Size  Used Avail Use% Mounted on
/dev/mapper/luks-rbd-0001-0009-rook-ceph-0000000000000001-243c0786-7181-11ed-af4a-6ee366128b14  9.8G   24K  9.8G   1% /data

but it already consumes 240MB in the OSDs:

$ rbd du replicapool/csi-vol-243c0786-7181-11ed-af4a-6ee366128b14
NAME                                          PROVISIONED  USED
csi-vol-243c0786-7181-11ed-af4a-6ee366128b14       10 GiB  240 MiB

it is acceptable penalty for sake of some ext4 metadata blocks.

next, I write a random file and check that the extra size is reflected in both the mapped volume and the underlying rbd:

# write a file
cd /data
head -c 50M /dev/urandom > file1
# check size on mapped device
df -h /data
Filesystem                                                                                      Size  Used Avail Use% Mounted on
/dev/mapper/luks-rbd-0001-0009-rook-ceph-0000000000000001-243c0786-7181-11ed-af4a-6ee366128b14  9.8G   51M  9.7G   1% /data
# check size in rbd
$ rbd du replicapool/csi-vol-243c0786-7181-11ed-af4a-6ee366128b14
NAME                                          PROVISIONED  USED
csi-vol-243c0786-7181-11ed-af4a-6ee366128b14       10 GiB  292 MiB

good. 50 MB added in both devices.

however , when I delete the file created earlier the space is reclaim in the mapped device but it is still allocated in the underlying rbd:

# delete file
$ cd /data
$ rm file1
# check size in mapped device - reclaimed!
Filesystem                                                                                      Size  Used Avail Use% Mounted on
/dev/mapper/luks-rbd-0001-0009-rook-ceph-0000000000000001-243c0786-7181-11ed-af4a-6ee366128b14  9.8G   24K  9.8G   1% /data
# check size in in rbd - still allocated!
$ rbd du replicapool/csi-vol-243c0786-7181-11ed-af4a-6ee366128b14
NAME                                          PROVISIONED  USED
csi-vol-243c0786-7181-11ed-af4a-6ee366128b14       10 GiB  292 MiB

our runtime is somewhat chaotic and the pods that mount these volume continuously read and write a large number of files, far worse than this isolated test. as a result we are observing a steady increase in disk usage inside the OSDs.

the "discard" mount option worked perfectly before we had to switch to encrypted volumes. now it is practically useless.

is there a way to sparsify encrypted ceph csi rbd volumes at mapped device layer and reflect the reclaimed blocks down in the encrypted rbd layer?

thank you

[nixpanic]

The answer is much related to issue #3563. For OSDs the same counts as for RBD-images when using encryption. Extra options for cryptsetup influence the behavior. It is not clear to me if using the options in subsequent opening/mapping of an encrypted device will result in freeing the allocated-but-unused blocks of the OSDs.

If you have tried it by now, could you share the results? A pointer to where to set extra cryptsetup options for the OSDs will be appreciated by others that come across this discussion. Thanks!