rbd: add support for adjusting the number of RBD mirror image watchers

[togoschi]

Since ceph-csi release v3.3.1 the workflow of RBD volume atttachment considers 2 watchers to check on the image when mirroring primary is true (#1993).

If you operate your ceph cluster environment with more than one rbd-mirror daemon (e.g. for high availibility and/or the cluster is streched over more than one datacenter) you maybe want to adjust the number of image watchers to check on the image.

It's required to run at least 2 rbd-mirror daemons if you want to ensure non-disruptive RBD mirroring. This is not a problem for the "in use" logic of mirrored images because only 1 rbd-mirror client watches the image usually.
But we observed volume attachment issues with 2 rbd-mirror clients when a mirror daemon leader becomes temporarily unresponsive (for example due to heavy load in the peering cluster).
That's not an serious issue for the rbd mirroring because of redundancy but meanwhile for the volume attachment of re-scheduled stateful pods.

In addition in some cases you even want to run more than 2 rbd-mirror daemons to balance the leaders over multiple RBD pools. This setup is not possible with the current implementation because you have 2 watchers on every image with 3 daemons for example (under normal operation).

Current implementation tolerates only one watcher if mirroring primary is true (https://github.com/ceph/ceph-csi/blob/v3.10.0/internal/rbd/rbd_util.go#L553-L559). I think it would be very helpful to make it possible adjusting the number of rbd-mirror watchers in more complex ceph environments.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[simon-wessel]

Support for this would be great. The hardcoded value does not seem to be a clean solution.

[Madhu-1]

@simon-wessel @togoschi sure will make it configurable

[togoschi]

@Madhu-1 thx for take care of it. A viable approach could be include the ability to configure an ipaddress array of all RBD mirror dameons

[Madhu-1]

@Madhu-1 thx for take care of it. A viable approach could be include the ability to configure an ipaddress array of all RBD mirror dameons

@togoschi i was thinking of adding a new key like mirrorDaemonCount (something like that) where the user can specify the count of daemons running

ceph-csi/deploy/csi-config-map-sample.yaml

Line 66 in 200fbdb

"radosNamespace": "<rados-namespace>",

. am not yet sure including ipaddresses will have any benefit as in case of container world the ip always changes dynamically, feel free to send a patch with suggestion As the community is always a priority :)

[togoschi]

@Madhu-1 of course you're completely right, setting the number of mirror daemons is the better approach

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[simon-wessel]

@Madhu-1 Is this still on the roadmap?

[Madhu-1]

@Madhu-1 Is this still on the roadmap?

@simon-wessel will try to fix it in coming week

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[simon-wessel]

Looking forward to the fix 🙂

[Madhu-1]

@simon-wessel Sorry, I wasn't able to work on it earlier. I have created a PR to make it configurable.