About snapshot creation of RBD images with exclusive-lock feature

[satoru-takeuchi]

My team manages several Rook/Ceph clusters for RBD, and we use rbd export/import to handle backup data. We're looking to speed up these processes by enabling the fast-diff feature on our RBD images. I have a question regarding the creation of snapshots for RBD images that use the exclusive-lock feature, which is necessary for fast-diff.

ceph-csi supports fast-diff.

   # (optional) RBD image features, CSI creates image with image-format 2 CSI
   # RBD currently supports `layering`, `journaling`, `exclusive-lock`,
   # `object-map`, `fast-diff`, `deep-flatten` features.
   # Refer https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#image-features
   # for image feature dependencies.
   # imageFeatures: layering,journaling,exclusive-lock,object-map,fast-diff
   imageFeatures: "layering"

A Rook's example of StorageClass describes exclusive-lock can be used with kernel >= 5.4 and my kernel is 6.x.

  # RBD image features
  # Available for imageFormat: "2". Older releases of CSI RBD
  # support only the `layering` feature. The Linux kernel (KRBD) supports the
  # full complement of features as of 5.4
  # `layering` alone corresponds to Ceph's bitfield value of "2" ;
  # `layering` + `fast-diff` + `object-map` + `deep-flatten` + `exclusive-lock` together
  # correspond to Ceph's OR'd bitfield value of "63". Here we use
  # a symbolic, comma-separated format:
  # For 5.4 or later kernels:
  #imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock
  # For 5.3 or earlier kernels:
  imageFeatures: layering

However, I'm still unsure whether I can enable exclusive-lock due to the following message in #578.

> as we do map --exclusive we cannot take a snapshot, right? is this expected

It's a limitation of the fact that krbd doesn't perform any maintenance ops. If you used rbd-nbd --exclusive, you would still be able to create snapshots because it would handle the request from a remote rbd client.

I assume we can map RBD images with the exclusive-lock feature without actually acquiring the lock (using the --exclusive option), and still create snapshots without causing metadata corruption. Is my understanding correct?

I'm concerned that creating snapshots of such RBD images might lead to inconsistencies between the in-memory object map and the on-disk one.

[idryomov]

I assume we can map RBD images with the exclusive-lock feature without actually acquiring the lock (using the --exclusive option), and still create snapshots without causing metadata corruption. Is my understanding correct?

Correct, you don't need to use --exclusive option to take advantage of fast-diff. Without --exclusive option, krbd would acquire the lock as needed (for I/O) and release it as needed (for a maintenance operation, such as creating a snapshot).

I'm concerned that creating snapshots of such RBD images might lead to inconsistencies between the in-memory object map and the on-disk one.

This shouldn't happen. There was a very nasty bug in this area which affected both krbd and librbd (i.e. rbd-nbd), but it got fixed last year and fixes got backported all the way to Ceph Pacific and many LTS kernels:

https://tracker.ceph.com/issues/61472
https://tracker.ceph.com/issues/61616

[satoru-takeuchi]

@idryomov Thank you very much!