feat(cspc): migrate pools from spc to cspc #1531

shubham14bajpai · 2019-11-19T11:57:22Z

Signed-off-by: shubham [email protected]

What this PR does / why we need it:
This PR adds the functionality to migrate a pool from SPC to CSPC.
Refer openebs/openebs#2840 for design.

Example Migration Job yaml

---
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-spc-sparse-claim-auto
  namespace: openebs
spec:
  backoffLimit: 4
  template:
    spec:
      #VERIFY the value of serviceAccountName is pointing to service account
      # created within openebs namespace. Use the non-default account.
      # by running `kubectl get sa -n <openebs-namespace>`
      serviceAccountName: openebs-maya-operator
      containers:
      - name:  migrate
        args:
        - "pool"
        # name of the spc that is to be migrated
        - "--spc-name=sparse-claim-auto"

        #Following are optional parameters
        #Log Level
        - "--v=4"
        #DO NOT CHANGE BELOW PARAMETERS
        env:
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        tty: true

        image: shubham1494/migrate:ci
      restartPolicy: OnFailure
---

Logs for the job executed for 3 pool SPC migration.

mayadata:setup$ kubectl -n openebs logs -f migrate-spc-sparse-claim-auto-79tmp 
I1210 12:10:12.633883       1 pool.go:73] Migrating spc sparse-claim-auto to cspc
I1210 12:10:12.657852       1 pool.go:256] Updating bdc bdc-26c12fb6-1b45-11ea-b9b2-42010aa00037 with cspc labels & finalizer.
I1210 12:10:12.666131       1 pool.go:256] Updating bdc bdc-26c4a58c-1b45-11ea-b9b2-42010aa00037 with cspc labels & finalizer.
I1210 12:10:12.671711       1 pool.go:256] Updating bdc bdc-2866fa95-1b45-11ea-b9b2-42010aa00037 with cspc labels & finalizer.
I1210 12:10:12.678051       1 pool.go:256] Updating bdc bdc-286a0ca1-1b45-11ea-b9b2-42010aa00037 with cspc labels & finalizer.
I1210 12:10:12.684106       1 pool.go:256] Updating bdc bdc-2a07a9ee-1b45-11ea-b9b2-42010aa00037 with cspc labels & finalizer.
I1210 12:10:12.690697       1 pool.go:256] Updating bdc bdc-2a082f89-1b45-11ea-b9b2-42010aa00037 with cspc labels & finalizer.
I1210 12:10:12.698127       1 pool.go:72] Creating equivalent cspc for spc sparse-claim-auto
I1210 12:10:17.758073       1 pool.go:286] Updating bdc bdc-26c12fb6-1b45-11ea-b9b2-42010aa00037 with cspc ownerRef.
I1210 12:10:17.768552       1 pool.go:286] Updating bdc bdc-26c4a58c-1b45-11ea-b9b2-42010aa00037 with cspc ownerRef.
I1210 12:10:17.778365       1 pool.go:286] Updating bdc bdc-2866fa95-1b45-11ea-b9b2-42010aa00037 with cspc ownerRef.
I1210 12:10:17.787820       1 pool.go:286] Updating bdc bdc-286a0ca1-1b45-11ea-b9b2-42010aa00037 with cspc ownerRef.
I1210 12:10:17.796191       1 pool.go:286] Updating bdc bdc-2a07a9ee-1b45-11ea-b9b2-42010aa00037 with cspc ownerRef.
I1210 12:10:17.807942       1 pool.go:286] Updating bdc bdc-2a082f89-1b45-11ea-b9b2-42010aa00037 with cspc ownerRef.
I1210 12:10:17.829258       1 pool.go:147] Migrating csp sparse-claim-auto-77dq to cspi sparse-claim-auto-vs62
I1210 12:10:17.829313       1 pool.go:208] Scaling down deployemnt sparse-claim-auto-vs62
I1210 12:12:13.084652       1 pool.go:310] Updating cvr pvc-5ad20014-1b45-11ea-b9b2-42010aa00037-sparse-claim-auto-vs62 with cspi sparse-claim-auto-77dq info.
I1210 12:12:13.110128       1 pool.go:147] Migrating csp sparse-claim-auto-hxbb to cspi sparse-claim-auto-wci1
I1210 12:12:13.110161       1 pool.go:208] Scaling down deployemnt sparse-claim-auto-wci1
I1210 12:12:58.306339       1 pool.go:310] Updating cvr pvc-5ad20014-1b45-11ea-b9b2-42010aa00037-sparse-claim-auto-wci1 with cspi sparse-claim-auto-hxbb info.
I1210 12:12:58.323553       1 pool.go:147] Migrating csp sparse-claim-auto-wgjz to cspi sparse-claim-auto-t6zm
I1210 12:12:58.323607       1 pool.go:208] Scaling down deployemnt sparse-claim-auto-t6zm
I1210 12:13:38.425799       1 pool.go:310] Updating cvr pvc-5ad20014-1b45-11ea-b9b2-42010aa00037-sparse-claim-auto-t6zm with cspi sparse-claim-auto-wgjz info.
I1210 12:13:38.444027       1 pool.go:79] Successfully migrated spc sparse-claim-auto to cspc

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Checklist:

Fixes #
Labelled this PR & related issue with documentation tag
PR messages has document related information
Labelled this PR & related issue with breaking-changes tag
PR messages has breaking changes related information
Labelled this PR & related issue with requires-upgrade tag
PR messages has upgrade related information
Commit has unit tests
Commit has integration tests

pkg/cstor/migrate/cspc_generator.go

pkg/cstor/migrate/migration.go

cmd/migrate/executor/setup_job.go

pkg/cstor/migrate/migration.go

pkg/cstor/migrate/pool.go

cmd/cspi-mgmt/app/handler.go

buildscripts/migrate/Makefile.mk

pkg/cstor/migrate/pool.go

cmd/cstor-pool-mgmt/pool/v1alpha2/import.go

mittachaitu · 2019-12-21T17:45:25Z

cmd/migrate/executor/setup_job.go

+}
+
+// PreRun will check for environement variables to be read and intialized.
+func PreRun(cmd *cobra.Command, args []string) {


There is the same check-in upgrade code also. Can we have a common code?

mittachaitu · 2019-12-21T17:57:50Z

cmd/migrate/executor/setup_job.go

+		NewMigratePoolJob(),
+	)
+
+	cmd.PersistentFlags().StringVarP(&options.openebsNamespace,


Are we overwriting the value of openebsNamespace after reading from env?

mittachaitu · 2019-12-22T05:33:31Z

pkg/cstor/migrate/pool.go

+		"replicas": 0
+	}
+}`
+	cspNameLabel           = "cstorpool.openebs.io/name"


Can we move these to APIs? So that we can use the same name wherever it required.

pkg/cstor/migrate/pool.go

mittachaitu · 2019-12-22T06:12:14Z

pkg/cstor/migrate/pool.go

+			WithNamespace(openebsNamespace).
+			Update(&bdcObj)
+		if err != nil {
+			return errors.Wrapf(err, "failed to update bdc %s with cspc label & finalizer", bdcObj.Name)


I think we have to add a note to README.md. Once the user tried the migration process user should try until it succeeds or else there will be chances of stale BD's if the user triggers the deletion of the SPC process(Since CSPC is not yet created).

Here, In the worst case, there will be a chance of network failures.

pkg/cstor/migrate/cspc_generator.go

pkg/apis/openebs.io/v1alpha1/cas_keys.go

pkg/cstor/migrate/cspc_generator.go

mittachaitu · 2019-12-22T11:38:12Z

pkg/cstor/migrate/cspc_generator.go

+	if err != nil {
+		return nil, err
+	}
+	cspcObj, err = cspc.NewKubeClient().


nit: Good to have a client in Hands instead of every time generating client. In CSPC kubernetes package we can use sync.Once

pkg/cstor/migrate/cspc_generator.go

pkg/cstor/migrate/pool.go

pkg/apis/openebs.io/v1alpha1/cstorpool_cluster.go

pkg/algorithm/nodeselect/v1alpha2/build_csp.go

mittachaitu · 2019-12-22T12:52:37Z

cmd/migrate/executor/env.go

@@ -0,0 +1,30 @@
+/*


Can we remove this package?

ghost

Found some fixes!

P.S. share your ideas, feedbacks or issues with us at https://github.com/fixmie/feedback (this message will be removed after the beta stage).

pkg/webhook/configuration.go

tests/localpv/hostdevice_test.go

pkg/debug/inject_client.go

cmd/cspi-mgmt/app/handler.go

Signed-off-by: shubham <[email protected]>

annotations: - removed oldCSPUID schema change - added two generic annotation which can be used in different scenarios - renamed one function according to the functionality Signed-off-by: shubham <[email protected]>

Signed-off-by: shubham <[email protected]>

pkg/cstor/migrate/pool.go

pkg/cstor/migrate/cspc_generator.go

pkg/cstor/migrate/pool.go

Signed-off-by: shubham <[email protected]>

pkg/cstor/migrate/pool.go

cmd/cstor-pool-mgmt/pool/v1alpha2/import.go

Signed-off-by: shubham <[email protected]>

pkg/cstor/migrate/pool.go

Signed-off-by: shubham <[email protected]>

mittachaitu

changes are good according to the current migration OEP. Enhancements are required based on CSPC & it's deployment changes.

shubham14bajpai · 2020-06-11T09:25:11Z

This PR is replaced by openebs-archive/upgrade#4

golangcibot reviewed Nov 19, 2019

View reviewed changes

pkg/cstor/migrate/cspc_generator.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/cspc_generator.go Show resolved Hide resolved

pkg/cstor/migrate/cspc_generator.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/cspc_generator.go Outdated Show resolved Hide resolved

shubham14bajpai force-pushed the migrate branch from 5b75709 to 0e14d17 Compare November 28, 2019 11:08

shubham14bajpai changed the title ~~[WIP] feat(migrate): generate CSPC form given SPC~~ feat(cspc): migrate pools from spc to cspc Nov 28, 2019

golangcibot reviewed Nov 28, 2019

View reviewed changes

shubham14bajpai force-pushed the migrate branch from af1e096 to 160c699 Compare November 28, 2019 11:35

golangcibot reviewed Nov 28, 2019

View reviewed changes

pkg/cstor/migrate/migration.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/migration.go Outdated Show resolved Hide resolved

shubham14bajpai force-pushed the migrate branch from 160c699 to e9c8c8d Compare November 28, 2019 11:47

golangcibot reviewed Nov 28, 2019

View reviewed changes

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

vishnuitta reviewed Nov 28, 2019

View reviewed changes

cmd/cspi-mgmt/app/handler.go Outdated Show resolved Hide resolved

vishnuitta reviewed Nov 28, 2019

View reviewed changes

buildscripts/migrate/Makefile.mk Show resolved Hide resolved

shubham14bajpai force-pushed the migrate branch from 01059c0 to 27e0fe4 Compare November 29, 2019 05:42

shubham14bajpai force-pushed the migrate branch from 0ddd9d5 to 99bc5a0 Compare December 10, 2019 11:06

golangcibot reviewed Dec 10, 2019

View reviewed changes

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

cmd/cstor-pool-mgmt/pool/v1alpha2/import.go Outdated Show resolved Hide resolved

mittachaitu reviewed Dec 22, 2019

View reviewed changes

mittachaitu requested a review from sonasingh46 December 23, 2019 03:31

shubham14bajpai force-pushed the migrate branch 2 times, most recently from 1fa726e to fcc401b Compare December 27, 2019 07:04

ghost reviewed Dec 27, 2019

View reviewed changes

pkg/webhook/configuration.go Outdated Show resolved Hide resolved

pkg/webhook/configuration.go Outdated Show resolved Hide resolved

golangcibot reviewed Dec 27, 2019

View reviewed changes

tests/localpv/hostdevice_test.go Outdated Show resolved Hide resolved

pkg/debug/inject_client.go Outdated Show resolved Hide resolved

pkg/debug/inject_client.go Outdated Show resolved Hide resolved

shubham14bajpai force-pushed the migrate branch 4 times, most recently from e05d79e to b5cf01a Compare December 27, 2019 07:46

kmova added this to the 1.6.0 milestone Dec 31, 2019

golangcibot reviewed Dec 31, 2019

View reviewed changes

cmd/cspi-mgmt/app/handler.go Outdated Show resolved Hide resolved

cmd/cspi-mgmt/app/handler.go Outdated Show resolved Hide resolved

shubham14bajpai added 3 commits December 31, 2019 17:13

feat(migrate): generate CSPC form given SPC

050b97d

Signed-off-by: shubham <[email protected]>

feat(cspc): migrate pools from spc to cspc

82e8d56

Signed-off-by: shubham <[email protected]>

fix linting issues

5b63cad

Signed-off-by: shubham <[email protected]>

shubham14bajpai added 12 commits December 31, 2019 17:14

added comments for the function and removed unnecessary checks

a6e1356

Signed-off-by: shubham <[email protected]>

This commit chnages the design from schema changes to use of

206c214

annotations: - removed oldCSPUID schema change - added two generic annotation which can be used in different scenarios - renamed one function according to the functionality Signed-off-by: shubham <[email protected]>

fixed golang-ci issues

ea82cd8

Signed-off-by: shubham <[email protected]>

addressed review comments for pkg

b590ef3

Signed-off-by: shubham <[email protected]>

fix scopelint issue and add priorityclass

976f301

Signed-off-by: shubham <[email protected]>

feat(migrate): generate CSPC form given SPC

99b37f0

Signed-off-by: shubham <[email protected]>

feat(cspc): migrate pools from spc to cspc

2559bc3

Signed-off-by: shubham <[email protected]>

fix linting issues

627c12b

Signed-off-by: shubham <[email protected]>

renamed one file

33f4d9c

Signed-off-by: shubham <[email protected]>

moved hardcoded strings in function calls to global constants

edc729d

Signed-off-by: shubham <[email protected]>

added comments for the function and removed unnecessary checks

4e771b1

Signed-off-by: shubham <[email protected]>

added aux resources to migrated pool

2a24183

Signed-off-by: shubham <[email protected]>

shubham14bajpai force-pushed the migrate branch from a077dbd to 2a24183 Compare December 31, 2019 11:55

resolved conflicts

d81cf5c

Signed-off-by: shubham <[email protected]>

mittachaitu reviewed Jan 6, 2020

View reviewed changes

addressed review comments

524897a

Signed-off-by: shubham <[email protected]>

mittachaitu reviewed Jan 14, 2020

View reviewed changes

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

cmd/cstor-pool-mgmt/pool/v1alpha2/import.go Outdated Show resolved Hide resolved

added object name to logs and fixed annotation deletion

aab7caa

Signed-off-by: shubham <[email protected]>

golangcibot reviewed Jan 17, 2020

View reviewed changes

pkg/cstor/migrate/pool.go Outdated Show resolved Hide resolved

fixed golang issue

e5f6235

Signed-off-by: shubham <[email protected]>

shubham14bajpai mentioned this pull request Jan 17, 2020

oep(cstor): proposal to migrate pools and volumes from old schema to new schema openebs/openebs#2840

Merged

mittachaitu approved these changes Jan 17, 2020

View reviewed changes

shubham14bajpai added the pr/hold-review label May 19, 2020

shubham14bajpai closed this Jun 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cspc): migrate pools from spc to cspc #1531

feat(cspc): migrate pools from spc to cspc #1531

shubham14bajpai commented Nov 19, 2019 •

edited

Loading

mittachaitu Dec 21, 2019

mittachaitu Dec 21, 2019

mittachaitu Dec 22, 2019

mittachaitu Dec 22, 2019

mittachaitu Dec 22, 2019

mittachaitu Dec 22, 2019

mittachaitu Dec 22, 2019

ghost left a comment

mittachaitu left a comment

shubham14bajpai commented Jun 11, 2020

feat(cspc): migrate pools from spc to cspc #1531

feat(cspc): migrate pools from spc to cspc #1531

Conversation

shubham14bajpai commented Nov 19, 2019 • edited Loading

mittachaitu Dec 21, 2019

Choose a reason for hiding this comment

mittachaitu Dec 21, 2019

Choose a reason for hiding this comment

mittachaitu Dec 22, 2019

Choose a reason for hiding this comment

mittachaitu Dec 22, 2019

Choose a reason for hiding this comment

mittachaitu Dec 22, 2019

Choose a reason for hiding this comment

mittachaitu Dec 22, 2019

Choose a reason for hiding this comment

mittachaitu Dec 22, 2019

Choose a reason for hiding this comment

ghost left a comment

Choose a reason for hiding this comment

mittachaitu left a comment

Choose a reason for hiding this comment

shubham14bajpai commented Jun 11, 2020

shubham14bajpai commented Nov 19, 2019 •

edited

Loading