feat(local-snapshot-restore, velero) : support to restore local snapshot to different namespace using velero #1575

mynktl · 2020-01-06T07:56:52Z

Changes:

Added support to restore local snapshot to different namespace using velero-plugin. Restored PV (and cstorVolume) will have cstor-clone- prefix.

Sample output:

kubectl get pv 
NAME                                               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS           REASON   AGE
cstor-clone-37458540-3bc4-42ae-b0ba-57ba116bb72f   5G         RWO            Delete           Bound    ns2/demo-cstor-vol-claim-1    openebs-cstor-sparse            5m55s
cstor-clone-7db69671-1243-4d96-99f7-7041ad9f3c2b   5G         RWO            Delete           Bound    ns1/demo-cstor-vol-claim-1    openebs-cstor-sparse            10s
pvc-109d7f73-699c-11ea-9326-42010a9a0064           5G         RWO            Delete           Bound    test/demo-cstor-vol-claim-1   openebs-cstor-sparse            3h52m

In above output, first two PVs are created from pvc-109d7f73-699c-11ea-9326-42010a9a0064 PV.

PVC output:

kubectl get pvc -A 
NAMESPACE   NAME                     STATUS   VOLUME                                             CAPACITY   ACCESS MODES   STORAGECLASS           AGE
ns1         demo-cstor-vol-claim-1   Bound    cstor-clone-7db69671-1243-4d96-99f7-7041ad9f3c2b   5G         RWO            openebs-cstor-sparse   18s
ns2         demo-cstor-vol-claim-1   Bound    cstor-clone-37458540-3bc4-42ae-b0ba-57ba116bb72f   5G         RWO            openebs-cstor-sparse   6m3s
test        demo-cstor-vol-claim-1   Bound    pvc-109d7f73-699c-11ea-9326-42010a9a0064           5G         RWO            openebs-cstor-sparse   3h52m

CStorVolume output:

kubectl get cstorvolume -n openebs
NAME                                               STATUS    AGE     CAPACITY
cstor-clone-37458540-3bc4-42ae-b0ba-57ba116bb72f   Healthy   13m     5G
cstor-clone-7db69671-1243-4d96-99f7-7041ad9f3c2b   Healthy   8m13s   5G
pvc-109d7f73-699c-11ea-9326-42010a9a0064           Healthy   4h      5G

For documentation:

For local snapshot(or remote backup) CV, related to PV, must be in healthy state.
If local snapshot( or remote backup) failed then check the backup logs for the error:
error taking snapshot of volume: rpc error: code = Unknown desc = Failed to send backup request: Error calling REST api: Status error{Bad Request}

Signed-off-by: mayank <[email protected]>

…estore Signed-off-by: mayank <[email protected]>

Signed-off-by: mayank <[email protected]>

cmd/maya-apiserver/app/server/restore_endpoint_v1alpha1.go

Signed-off-by: mayank <[email protected]>

mynktl · 2020-03-19T09:30:43Z

@prateekpandey14 Can you verify https://github.com/openebs/maya/pull/1575/files#diff-861ad0cc410461be20924c447e61bc86R281. I am creating volume through pkg/volume/volume.go. If i remember correctly, you are calling this volume creation API multiple times through provisioner. Do i need to add similar behavior here?

Signed-off-by: mayank <[email protected]>

pkg/apis/openebs.io/v1alpha1/cstor_backup.go

pkg/apis/openebs.io/v1alpha1/cstor_restore.go

Signed-off-by: mayank <[email protected]>

cmd/maya-apiserver/app/server/backup_endpoint_v1alpha1.go

vishnuitta · 2020-03-24T14:36:36Z

cmd/maya-apiserver/app/server/restore_endpoint_v1alpha1.go

@@ -62,7 +64,7 @@ func (rOps *restoreAPIOps) create() (interface{}, error) {
 	}

 	// namespace is expected
-	if len(strings.TrimSpace(restore.Namespace)) == 0 {
+	if !restore.Spec.Local && len(strings.TrimSpace(restore.Namespace)) == 0 {


should we throw error if Local and NameSpace is NOT 0?

In case of local We are not using namespace. We are ignoring it so skipped the check.

vishnuitta

Few things like adding comments, testcases are required. Overall flow looks to be good, however one more round of review need to be done.
APIs are changed however backup compatibility is maintained.
This involves good amount of changes with respect to approach as well. Now, storage classes with 'WatiForFirstConsumer' may not work. So, its better to get into the early cycles of testing.
Approving this in order to get into early cycles of testing.

kmova · 2020-03-25T18:12:55Z

@mynktl - can you verify that mayactl volume list works with cloned volumes being present.

Though there is not a lot of information obtained by looking at mayactl volume list for cstor volumes, it should not result in - inability to fetch the jiva volumes list, as it is the only way to know the status of jiva volumes.

If there are issue listing the cStor volumes - we can raise another PR to deprecate including cStor volumes in the mayactl volume list

kmova · 2020-03-25T18:17:33Z

In addition to @vishnuitta comments, this PR also will need to document the following:

Cases / Pre-requisites for this feature to work.
For instance
- If the volume is not healthy, how is the snapshot failure communicated?
- If volume is healthy with two replicas, and snapshot is taken, what will happen when the a new Replica gets added and a clone request comes before the Replica had a chance to rebuild.

cmd/maya-apiserver/app/server/backup_endpoint_v1alpha1.go

kmova

Changes are good.

A few follow-up items are mentioned as review comments, once they are verified and backlogs are created, the merge can proceed.

vishnuitta · 2020-03-25T19:02:54Z

cmd/maya-apiserver/app/server/restore_endpoint_v1alpha1.go

+			return nil, errors.Wrapf(err, "Failed to get restore volume details")
+		}
+	} else {
+		return cvol, nil


is this possible in case of cloud backup?

yes, In case of cloud backups we are creating volume through PVC from velero-plugin. So this will return the existing volume. Added comment.

vishnuitta · 2020-03-25T19:08:38Z

cmd/maya-apiserver/app/server/restore_endpoint_v1alpha1.go

+		return cvol, nil
+	}
+
+	return vOps.Create()


is this supposed to be for local backup?

It serves for local and cloud backup. As of now we are creating volume through PVC from velero-plugin in case of remote restore So this will not execute for remote backup. #1575 (comment)

mynktl · 2020-03-26T08:33:56Z

@kmova

For jiva volumes with mayactl We already have a test case which uses mayactl to fetch volumes.
For CStor volumes I verified locally.

bash-4.3# /tmp/mayactl  volume list --volname cstor-clone-d1a8fdce-30e0-4df5-b616-97a05192ed71
Namespace  Name                                              Status   Type   Capacity  StorageClass          Access Mode
---------  ----                                              ------   ----   --------  -------------         -----------
openebs    cstor-clone-d1a8fdce-30e0-4df5-b616-97a05192ed71  Running  cstor  5G        openebs-cstor-sparse  ReadWriteOnce
openebs    pvc-559e2dc4-6f3b-11ea-86cd-d89c67d5e4a1          Running  cstor  5G        openebs-cstor-sparse  ReadWriteOnce

bash-4.3# /tmp/mayactl  volume stats --volname pvc-559e2dc4-6f3b-11ea-86cd-d89c67d5e4a1 --namespace openebs
 
Portal Details :
---------------
Volume  :   pvc-559e2dc4-6f3b-11ea-86cd-d89c67d5e4a1
Size    :   5.000000

Performance Stats :
--------------------
r/s      w/s      r(MB/s)      w(MB/s)      rLat(ms)      wLat(ms)
----     ----     --------     --------     ---------     ---------
0        0        0.000        0.000        0.000         0.000    

Capacity Stats :
---------------
LOGICAL(GB)      USED(GB)
------------     ---------
0.000            0.000    
bash-4.3# /tmp/mayactl  volume stats --volname cstor-clone-d1a8fdce-30e0-4df5-b616-97a05192ed71 --namespace openebs 
 
Portal Details :
---------------
Volume  :   cstor-clone-d1a8fdce-30e0-4df5-b616-97a05192ed71
Size    :   5.000000

Performance Stats :
--------------------
r/s      w/s      r(MB/s)      w(MB/s)      rLat(ms)      wLat(ms)
----     ----     --------     --------     ---------     ---------
0        0        0.000        0.000        0.000         0.000    

Capacity Stats :
---------------
LOGICAL(GB)      USED(GB)
------------     ---------

Signed-off-by: mayank <[email protected]>

mynktl · 2020-03-26T09:37:10Z

For documentation I've covered it in openebs/velero-plugin#53.

mynktl · 2020-03-26T09:37:54Z

@kmova @vishnuitta PTAL. I've addressed the review comments.

mayank added 6 commits December 30, 2019 13:58

Removing PVC from PV creation path

2092d2f

Signed-off-by: mayank <[email protected]>

Removing check for PVC and updating error messages

666480b

Signed-off-by: mayank <[email protected]>

Adding missing changes

c2708a7

Signed-off-by: mayank <[email protected]>

Using volume API of maya-apiserver to create a volume during velero r…

3e431f8

…estore Signed-off-by: mayank <[email protected]>

Fixing build

ab39c1b

Signed-off-by: mayank <[email protected]>

Defining new CASKey AnnPVCASVolKey for velero CStor volume restore

cb7ad75

Signed-off-by: mayank <[email protected]>

mynktl mentioned this pull request Jan 6, 2020

[WIP]feat(backup,restore): backup/restore of local snapshot using velero #1576

Closed

9 tasks

kmova added this to the 1.7.0 milestone Jan 8, 2020

kmova mentioned this pull request Jan 10, 2020

CI/CD and cloning - how to make use of? openebs/openebs#2844

Closed

shubham14bajpai reviewed Jan 29, 2020

View reviewed changes

cmd/maya-apiserver/app/server/restore_endpoint_v1alpha1.go Outdated Show resolved Hide resolved

mynktl changed the title ~~fix(restore, velero) : using volume API to create a volume during velero restore~~ [WIP]fix(restore, velero) : using volume API to create a volume during velero restore Feb 3, 2020

Merge remote-tracking branch 'openebs/master' into backup_w_pvc

4431561

mynktl modified the milestones: 1.7.0, 1.9 Mar 17, 2020

Adding support to restore local snapshot through velero-plugin

452263c

Signed-off-by: mayank <[email protected]>

mynktl changed the title ~~[WIP]fix(restore, velero) : using volume API to create a volume during velero restore~~ fix(local-snapshot-restore, velero) : support to restore local snapshot to different snapshot using velero Mar 19, 2020

mynktl requested a review from vishnuitta March 19, 2020 08:42

Updating generated code through kubegen

15a70ff

Signed-off-by: mayank <[email protected]>

mynktl changed the title ~~fix(local-snapshot-restore, velero) : support to restore local snapshot to different snapshot using velero~~ feat(local-snapshot-restore, velero) : support to restore local snapshot to different snapshot using velero Mar 19, 2020

mynktl requested a review from prateekpandey14 March 19, 2020 09:29

mayank added 2 commits March 19, 2020 15:29

Merge remote-tracking branch 'openebs/master' into backup_w_pvc

ef5aa70

fixing travis build

d319740

Signed-off-by: mayank <[email protected]>

mynktl requested a review from kmova March 20, 2020 06:49

vishnuitta reviewed Mar 20, 2020

View reviewed changes

pkg/apis/openebs.io/v1alpha1/cstor_backup.go Show resolved Hide resolved

kmova reviewed Mar 21, 2020

View reviewed changes

pkg/apis/openebs.io/v1alpha1/cstor_restore.go Outdated Show resolved Hide resolved

Addressing review comments

ce5a516

Signed-off-by: mayank <[email protected]>

prateekpandey14 changed the title ~~feat(local-snapshot-restore, velero) : support to restore local snapshot to different snapshot using velero~~ feat(local-snapshot-restore, velero) : support to restore local snapshot to different namespace using velero Mar 24, 2020

vishnuitta reviewed Mar 24, 2020

View reviewed changes

cmd/maya-apiserver/app/server/backup_endpoint_v1alpha1.go Show resolved Hide resolved

vishnuitta reviewed Mar 24, 2020

View reviewed changes

vishnuitta approved these changes Mar 25, 2020

View reviewed changes

kmova reviewed Mar 25, 2020

View reviewed changes

cmd/maya-apiserver/app/server/backup_endpoint_v1alpha1.go Outdated Show resolved Hide resolved

kmova added pr/documentation-pending pr/release-note PR should be included in release notes pr/hold-merge Hold on the merge to complete few activities. labels Mar 25, 2020

kmova approved these changes Mar 25, 2020

View reviewed changes

vishnuitta reviewed Mar 25, 2020

View reviewed changes

Addressing review comments

ff10549

Signed-off-by: mayank <[email protected]>

kmova merged commit 7daa3c6 into openebs-archive:master Mar 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(local-snapshot-restore, velero) : support to restore local snapshot to different namespace using velero #1575

feat(local-snapshot-restore, velero) : support to restore local snapshot to different namespace using velero #1575

mynktl commented Jan 6, 2020 •

edited by kmova

Loading

mynktl commented Mar 19, 2020

vishnuitta Mar 24, 2020

mynktl Mar 24, 2020

vishnuitta left a comment

kmova commented Mar 25, 2020

kmova commented Mar 25, 2020

kmova left a comment

vishnuitta Mar 25, 2020

mynktl Mar 26, 2020

vishnuitta Mar 25, 2020 •

edited

Loading

mynktl Mar 26, 2020

mynktl commented Mar 26, 2020

mynktl commented Mar 26, 2020

mynktl commented Mar 26, 2020

feat(local-snapshot-restore, velero) : support to restore local snapshot to different namespace using velero #1575

feat(local-snapshot-restore, velero) : support to restore local snapshot to different namespace using velero #1575

Conversation

mynktl commented Jan 6, 2020 • edited by kmova Loading

mynktl commented Mar 19, 2020

vishnuitta Mar 24, 2020

Choose a reason for hiding this comment

mynktl Mar 24, 2020

Choose a reason for hiding this comment

vishnuitta left a comment

Choose a reason for hiding this comment

kmova commented Mar 25, 2020

kmova commented Mar 25, 2020

kmova left a comment

Choose a reason for hiding this comment

vishnuitta Mar 25, 2020

Choose a reason for hiding this comment

mynktl Mar 26, 2020

Choose a reason for hiding this comment

vishnuitta Mar 25, 2020 • edited Loading

Choose a reason for hiding this comment

mynktl Mar 26, 2020

Choose a reason for hiding this comment

mynktl commented Mar 26, 2020

mynktl commented Mar 26, 2020

mynktl commented Mar 26, 2020

mynktl commented Jan 6, 2020 •

edited by kmova

Loading

vishnuitta Mar 25, 2020 •

edited

Loading