rbd: add support for flattenMode option for replication #4678
Conversation
|
This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏 |
Rakshith-R
force-pushed
the
flattenMode
branch
8 times, most recently
from
0839781
to
5588b73
Compare
|
I've added testing results in pr description. |
@Rakshith-R, I don't see the testing results in the PR description. can you re-check if you have missed it? |
forgot to press update button. |
| return corerbd.FlattenModeNever | ||
| } | ||
|
|
||
| switch corerbd.FlattenMode(val) { |
There was a problem hiding this comment.
| switch corerbd.FlattenMode(val) { | |
| mode := corerbd.FlattenMode(val); | |
| switch mode { | |
| case corerbd.FlattenModeForce, corerbd.FlattenModeNever: | |
| return mode,nil | |
| default: | |
| log.DebugLog(ctx, "flattenMode %v not supported",mode) | |
| return mode,err | |
| } |
lets return error in case of its a user mistake instead of choosing FlattenModeNever
| if err != nil { | ||
| return nil, err | ||
| } |
There was a problem hiding this comment.
this check seems to be wrong?
internal/rbd/mirror.go
Outdated
| const ( | ||
| // FlattenModeNever indicates that the image should never be flattened. | ||
| FlattenModeNever FlattenMode = "never" | ||
| // FlattenModeForce indicates that the image with parent should definitely |
There was a problem hiding this comment.
FlattenModeForce indicates that the image with the parent must be flattened
internal/rbd/mirror.go
Outdated
| // if the flattenMode is FlattenModeForce, it flattens the image itself. | ||
| // if the parent image is in trash, it returns an error. | ||
| // if the parent image exists and is not enabled for mirroring, it returns an error. | ||
| func (rv *rbdVolume) HandleParentImage( |
There was a problem hiding this comment.
should we rename this to MirrorImageIfParentExists or something else as we are not handling the ParentImage rather we are flattening of the image only if the parent exists ?
There was a problem hiding this comment.
HandleParentImage can also be read as handling the cases where parent image may exist and the method to handle such parent depends.
Currently we either error out or flatten the live image.
There was a problem hiding this comment.
we are acting/handling on the actual image not on the parent, IMO we don't handle anything in parentImage we are having few checks on it, the main idea is to flatten the main/live image. For me, it make sense to have something like MirrorImageIfParentExists and HandleParentImage looks to be generic and doesn't mention what exactly we are handing here. please feel free to rename to anything else that gives a sense of what we are doing in this function.
There was a problem hiding this comment.
Renamed it to HandleParentImageExistence to imply this function contains logic to handle existence of a parent
image
| } | ||
|
|
||
| if flattenMode == FlattenModeForce { | ||
| err := rv.DeleteTempImage(ctx) |
There was a problem hiding this comment.
please add a comment why we are deleting the temporary clone here
internal/rbd/mirror.go
Outdated
| "failed to get mirroring info of parent %q of image %q: %w", | ||
| rv, parent, err) |
There was a problem hiding this comment.
| "failed to get mirroring info of parent %q of image %q: %w", | |
| rv, parent, err) | |
| "failed to get mirroring info of parent %q of image %q: %w", | |
| parent, rv, err) |
internal/rbd/mirror.go
Outdated
| rv, parent, err) | ||
| } | ||
|
|
||
| if parentMirroringInfo.State == librbd.MirrorImageEnabled { |
There was a problem hiding this comment.
| if parentMirroringInfo.State == librbd.MirrorImageEnabled { | |
| if parentMirroringInfo.State != librbd.MirrorImageEnabled { | |
| fmt.Errorf("%w:failed to enable mirroring on image %q: "+ | |
| "parent image %q is not enabled for mirroring", | |
| ErrFailedPrecondition, rv, parent) | |
| } | |
| return nil |
internal/rbd/rbd_util.go
Outdated
| @@ -703,6 +703,27 @@ func (ri *rbdImage) trashRemoveImage(ctx context.Context) error { | |||
| return nil | |||
| } | |||
|
|
|||
| // DeleteTempImage deletes the temporary image created for pvc-pvc clone. | |||
There was a problem hiding this comment.
| // DeleteTempImage deletes the temporary image created for pvc-pvc clone. | |
| // DeleteTempImage deletes the temporary image created for volume datasource. |
internal/rbd/rbd_util.go
Outdated
| log.ErrorLog(ctx, "failed to delete rbd image: %s with error: %v", | ||
| tempClone, err) |
There was a problem hiding this comment.
lets not log here, dont pass the ctx and log it in caller if there is any error?
There was a problem hiding this comment.
lets not log here, dont pass the ctx and log it in caller if there is any error?
ctx is definitely required for other function calls in this function.
I'll move the log upwards.
There was a problem hiding this comment.
it looks like its not addressed, can you please check this one again
Rakshith-R
force-pushed
the
flattenMode
branch
3 times, most recently
from
e582ff7
to
6c74635
Compare
| case corerbd.FlattenModeForce, corerbd.FlattenModeNever: | ||
| return mode, nil | ||
| } | ||
| log.DebugLog(ctx, "%q=%q is not supported", flattenModeKey, val) |
There was a problem hiding this comment.
lets log it as in error message in caller not as debug here
internal/rbd/mirror.go
Outdated
| // if the flattenMode is FlattenModeForce, it flattens the image itself. | ||
| // if the parent image is in trash, it returns an error. | ||
| // if the parent image exists and is not enabled for mirroring, it returns an error. | ||
| func (rv *rbdVolume) HandleParentImage( |
There was a problem hiding this comment.
we are acting/handling on the actual image not on the parent, IMO we don't handle anything in parentImage we are having few checks on it, the main idea is to flatten the main/live image. For me, it make sense to have something like MirrorImageIfParentExists and HandleParentImage looks to be generic and doesn't mention what exactly we are handing here. please feel free to rename to anything else that gives a sense of what we are doing in this function.
internal/rbd/mirror.go
Outdated
| log.ErrorLog(ctx, "failed to delete temporary rbd image: %v", err) | ||
|
|
||
| return err |
There was a problem hiding this comment.
| log.ErrorLog(ctx, "failed to delete temporary rbd image: %v", err) | |
| return err | |
| return fmt.Errorf("failed to delete temporary rbd image: %w", err) |
internal/rbd/rbd_util.go
Outdated
| err = tempClone.ensureImageCleanup(ctx) | ||
|
|
||
| return err |
There was a problem hiding this comment.
| err = tempClone.ensureImageCleanup(ctx) | |
| return err | |
| return tempClone.ensureImageCleanup(ctx) |
internal/rbd/rbd_util.go
Outdated
| log.ErrorLog(ctx, "failed to delete rbd image: %s with error: %v", | ||
| tempClone, err) |
There was a problem hiding this comment.
it looks like its not addressed, can you please check this one again
|
@Mergifyio queue |
✅ The pull request has been merged automaticallyThe pull request has been merged automatically at d166229 |
This commit adds support for flattenMode option for replication. If the flattenMode is set to "force" in volumereplicationclass parameters, cephcsi will add a task to flatten the image if it has parent. This enable cephcsi to then mirror such images after flattening them. The error message when the image's parent is in trash or unmirrored is improved as well. Signed-off-by: Rakshith R <[email protected]>
|
/test ci/centos/upgrade-tests-cephfs |
|
/test ci/centos/upgrade-tests-rbd |
|
/test ci/centos/k8s-e2e-external-storage/1.28 |
|
/test ci/centos/k8s-e2e-external-storage/1.29 |
|
/test ci/centos/mini-e2e-helm/k8s-1.28 |
|
/test ci/centos/k8s-e2e-external-storage/1.27 |
|
/test ci/centos/mini-e2e-helm/k8s-1.29 |
|
/test ci/centos/mini-e2e/k8s-1.28 |
|
/test ci/centos/k8s-e2e-external-storage/1.30 |
|
/test ci/centos/mini-e2e-helm/k8s-1.27 |
|
/test ci/centos/mini-e2e/k8s-1.29 |
|
/test ci/centos/mini-e2e/k8s-1.27 |
|
/test ci/centos/mini-e2e-helm/k8s-1.30 |
|
/test ci/centos/mini-e2e/k8s-1.30 |
This commit adds support for flattenMode option
for replication.
If the flattenMode is set to "force" in
volumereplicationclass parameters, cephcsi will
add a task to flatten the image if it has parent.
This enable cephcsi to then mirror such images after flattening them.
The error message when the image's parent is
in trash or unmirrored is improved as well.
Checklist:
guidelines in the developer
guide.
Request
notes
updated with breaking and/or notable changes for the next major release.
Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:
/retest ci/centos/<job-name>: retest the<job-name>after unrelatedfailure (please report the failure too!)
test results
Restored PVC
I0619 09:20:05.482558 1 utils.go:198] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication I0619 09:20:05.485272 1 utils.go:199] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"} I0619 09:20:05.506278 1 omap.go:89] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph] W0619 09:20:05.845510 1 replication.go:153] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 09:20:05.845583 1 replication.go:129] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 flattenMode is not set in parameters, setting to default (never) E0619 09:20:06.336891 1 replication.go:314] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18":parent image "replicapool/csi-snap-ad971f5b-61a2-4bf4-8c0b-4ec61cec216e" is not enabled for mirroring E0619 09:20:06.340126 1 utils.go:203] ID: 36 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC error: rpc error: code = Internal desc = failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18": parent image "replicapool/csi-snap-ad971f5b-61a2-4bf4-8c0b-4ec61cec216e" is not enabled for mirroringI0619 09:21:45.816334 1 utils.go:198] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication I0619 09:21:45.819942 1 utils.go:199] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"} I0619 09:21:45.828980 1 omap.go:89] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph] W0619 09:21:45.958255 1 replication.go:153] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 09:21:45.958630 1 replication.go:129] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 flattenMode is not set in parameters, setting to default (never) E0619 09:21:46.093419 1 replication.go:314] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 system is not in a state required for the operation's execution: failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18": parent is in trash E0619 09:21:46.093596 1 utils.go:203] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC error: rpc error: code = FailedPrecondition desc = system is not in a state required for the operation's execution: failed to enable mirroring on image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18": parent is in trashI0619 09:24:53.228433 1 utils.go:198] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication I0619 09:24:53.228743 1 utils.go:199] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"} I0619 09:24:53.237278 1 omap.go:89] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph] W0619 09:24:53.387365 1 replication.go:153] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 09:24:53.538514 1 rbd_util.go:829] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 rbd: adding task to flatten image "replicapool/csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18" E0619 09:24:53.736260 1 replication.go:314] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 flatten in progress: flatten is in progress for image csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 E0619 09:24:53.736433 1 utils.go:203] ID: 53 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC error: rpc error: code = Aborted desc = flatten in progress: flatten is in progress for image csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 ... I0619 09:24:55.058061 1 utils.go:198] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC call: /replication.Controller/EnableVolumeReplication I0619 09:24:55.058184 1 utils.go:199] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18"} I0619 09:24:55.090814 1 omap.go:89] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 got omap values: (pool="replicapool", namespace="", name="csi.volume.ae66b7a1-21d7-4205-bc0a-d620e1fbab18"): map[csi.imageid:11c0103d7da8 csi.imagename:csi-vol-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 csi.volname:pvc-cf11e3a2-50c8-438a-8911-958e046772f8 csi.volume.owner:rook-ceph] W0619 09:24:55.170256 1 replication.go:153] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 09:24:56.210444 1 utils.go:205] ID: 57 Req-ID: 0001-0009-rook-ceph-0000000000000002-ae66b7a1-21d7-4205-bc0a-d620e1fbab18 GRPC response: {}PVC-PVC clone
I0619 09:29:13.460421 1 utils.go:198] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 GRPC call: /replication.Controller/EnableVolumeReplication I0619 09:29:13.468826 1 utils.go:199] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 GRPC request: {"parameters":{"schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7"} I0619 09:29:13.491370 1 omap.go:89] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 got omap values: (pool="replicapool", namespace="", name="csi.volume.f3f494a9-c9f9-4734-b9e4-b04cd58cfac7"): map[csi.imageid:187bfd24f7c6 csi.imagename:csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 csi.volname:pvc-12fe2ae9-cb90-4bee-9247-6146f11bf8e2 csi.volume.owner:rook-ceph] W0619 09:29:13.620327 1 replication.go:153] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 09:29:13.620533 1 replication.go:129] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 flattenMode is not set in parameters, setting to default (never) E0619 09:29:14.225457 1 replication.go:314] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 failed to enable mirroring on image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7":parent image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7-temp" is not enabled for mirroring E0619 09:29:14.226126 1 utils.go:203] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000002-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7 GRPC error: rpc error: code = Internal desc = failed to enable mirroring on image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7": parent image "replicapool/csi-vol-f3f494a9-c9f9-4734-b9e4-b04cd58cfac7-temp" is not enabled for mirroringI0619 10:17:48.127012 1 utils.go:198] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC call: /replication.Controller/EnableVolumeReplication I0619 10:17:48.127174 1 utils.go:199] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93"} I0619 10:17:48.139129 1 omap.go:89] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 got omap values: (pool="replicapool", namespace="", name="csi.volume.d73378c1-62b3-454e-8646-dae3207e7f93"): map[csi.imageid:1d39997d6758 csi.imagename:csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93 csi.volname:pvc-95b1e8ca-590d-420e-9c93-d6686e857d2a csi.volume.owner:rook-ceph] W0619 10:17:48.284313 1 replication.go:153] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 10:17:48.468021 1 rbd_util.go:634] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: delete csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93-temp using mon 192.168.122.191:6789, pool replicapool I0619 10:17:48.494034 1 rbd_util.go:676] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: adding task to remove image "replicapool/csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93-temp" with id "1d39206cc57d" from trash I0619 10:17:48.503025 1 rbd_util.go:700] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: successfully added task to move image "replicapool/csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93-temp" with id "1d39206cc57d" to trash I0619 10:17:48.503080 1 rbd_util.go:850] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 rbd: adding task to flatten image "replicapool/csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93" E0619 10:17:48.599921 1 replication.go:314] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 flatten in progress: flatten is in progress for image csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93 E0619 10:17:48.600632 1 utils.go:203] ID: 33 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC error: rpc error: code = Aborted desc = flatten in progress: flatten is in progress for image csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93 ... 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC call: /replication.Controller/EnableVolumeReplication I0619 10:17:57.469112 1 utils.go:199] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC request: {"parameters":{"flattenMode":"force","schedulingInterval":"1m"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93"} I0619 10:17:57.481057 1 omap.go:89] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 got omap values: (pool="replicapool", namespace="", name="csi.volume.d73378c1-62b3-454e-8646-dae3207e7f93"): map[csi.imageid:1d39997d6758 csi.imagename:csi-vol-d73378c1-62b3-454e-8646-dae3207e7f93 csi.volname:pvc-95b1e8ca-590d-420e-9c93-d6686e857d2a csi.volume.owner:rook-ceph] W0619 10:17:57.530084 1 replication.go:153] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot) I0619 10:17:57.603214 1 utils.go:205] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000002-d73378c1-62b3-454e-8646-dae3207e7f93 GRPC response: {}