How to achieve mirrirong cloned PVCs

[satoru-takeuchi]

As shown in #2426 and #2427, it's impossible to mirror cloned PVC (RBD image) only by ceph-csi for now. Although all ancestor RBD images must also enable RBD mirroring, there is no way to do it in ceph-csi's layer. I'd like to discuss how to overcome this limitation.

One possible way is toggling the mirror settiong of ancestor RBD images according to the existence of VolumeReplication resource corresponding to cloned PVC. However, this logic will be hard to maintain and the major change will be necessary in the cloning logic.

I believe that providing the way to flatten PVC in ceph-csi layer is the nice solution. What do you think?

If the flattening is regarded as a reasonable solution, we have several options as follows:

a. Provide the way to flatten an arbitrary PVC explicitly (e.g. by adding an annotation) and write a document that this step is the prerequisite of mirroring cloned PVC.
b. Write a document that the manual flattening operation is necessary to mirror cloned PVC
c. Flatten a PVC automatically when the corresponding VolumeReplication is created.

IMO, (a) is the best solution. (b) is annoying for users to find the correct RBD image corresponding to the cloned PVC. (c) is OK as a perspective from only RBD mirror user. However, it's convenient for users to be able to flatten RBD images explicitly if they concern the data redundancy and/or performance.

One more thing. If we decide flattening as the solution, we should also consider how to handle the temporary RBD image.

the parent RBD image: correspondind to the parent PVC
| clone
v
the temporary RBD image: corresponding to a VolumeSnapshotContent (on clone from snapshot) or no resource (no PVC-PVC clone)
|
v clone
the cloned RBD image: corresponding to the cloned PVC

If we delete the parent PVC after flattening the cloned PVC, the parent PVC's space is not freed. It's because the temporary RBD image is still alive.

the temporary RBD image(still alive): corresponding to a VolumeSnapshotContent (on clone from snapshot) or no resource (no PVC-PVC clone)
|
v clone
the cloned RBD image(flattened): corresponding to the cloned PVC

We should delete the temporary RBD image on PVC-PVC clone case.

[Rakshith-R]

Snapshot in following lines refers to k8s snapshot -> a rbd image + rbd snap.

a. Provide the way to flatten an arbitrary PVC explicitly (e.g. by adding an annotation) and write a document that this step is the prerequisite of mirroring cloned PVC.
b. Write a document that the manual flattening operation is necessary to mirror cloned PVC
c. Flatten a PVC automatically when the corresponding VolumeReplication is created.

IMO, (a) is the best solution. (b) is annoying for users to find the correct RBD image corresponding to the cloned PVC. (c) is OK as a perspective from only RBD mirror user. However, it's convenient for users to be able to flatten RBD images explicitly if they concern the data redundancy and/or performance.

Regarding (a), cephcsi is a server agnostic of container orchestrator. It maybe simple for an operator or controller to do this but that is not in realm of cephcsi server.

Incorporating (a) & (b) together, we definitely need to document the steps. We can also make use of https://github.com/rook/kubectl-rook-ceph to add an operation to flatten PVC to make it easier for users.

Regarding (c), We never run flatten operation on a rbd image that a client can do io on at cephcsi. This is the least favorable one.

---

Currently, We've discussed to add support for mirroring of VolumeSnapshot, several points are yet to be discussed.
This would enable the following scenarios:

• Primary mirror enabled PVC + Snapshots at regular intervals which can also be mirrored
• Snapshot with a unmirrored/deleted parent
  • Such a snapshot will undergo flattening on enable mirror call
  • multiple child restore PVC can be created and mirrored.
  • This has huge benefit of space optimization since all the child PVCs will still point to parent data in the snasphot.

Handling PVC-PVC clone is indeed complex because of the temporary image. For now, I think we can have logic to flatten this in rook-kubectl-plugin and handle proper mirroing flow of PVC-PVC clone in next releases.

[satoru-takeuchi]

Thank you for the detailed explanation.

Handling PVC-PVC clone is indeed complex because of the temporary image. For now, I think we can have logic to flatten this in rook-kubectl-plugin and handle proper mirroing flow of PVC-PVC clone in next releases.

Does the ceph-csi team have enough resources to implement this? If you don't mind, I will do it because I need mirrirong PVC-PVC clone for my system.

[Rakshith-R]

Thank you for the detailed explanation.

Handling PVC-PVC clone is indeed complex because of the temporary image. For now, I think we can have logic to flatten this in rook-kubectl-plugin and handle proper mirroing flow of PVC-PVC clone in next releases.

Does the ceph-csi team have enough resources to implement this? If you don't mind, I will do it because I need mirrirong PVC-PVC clone for my system.

Yes, you can implement the functionality. Thanks for the help, its greatly appreciated !

I think the following steps should be sufficient:

• Check if the PVC is in Use. Something like this

  ceph-csi/internal/rbd/rbd_util.go

  Line 526 in 51decb0

  func (ri *rbdImage) isInUse() (bool, error) {

  or checking volumeattachments.
• If it is in use, error out
• Check for temporary image with name "+'-temp' ". If it exists, move it to trash and add a task to remove it from trash.
• Flatten the main rbd image underlying the PVC.

[satoru-takeuchi]

Check if the PVC is in Use. Something like this
ceph-csi/internal/rbd/rbd_util.go

Line 526 in 51decb0

func (ri *rbdImage) isInUse() (bool, error) {
or checking volumeattachments.

@Rakshith-R Is it necessary to verify whether PVC is in use or not? I didn't found the limitation like flattening in-use RBD image is prohibited.

[Rakshith-R]

It is not prohibited but it may have an impact on client IO performance so we never flatten a rbd image that maybe mapped for IO.

[satoru-takeuchi]

OK, I see. It it acceptable to have an option to skip this check? It's convenient for admins who want to flatten images without restarting their pods. Since the flattening wil be triggered not by ceph-csi automatically, but by kubectl-rook-ceph manually, admins can control the timing of running this command.

[Rakshith-R]

Yes, that sounds good to me.

[satoru-takeuchi]

kubectl-rook-ceph v0.9.0 supports flattening RBD PVC.

f003e54 core: add flatten-rbd-pvc command

@Rakshith-R Thank you for giving me many hints!