doc: add migration design documentation

This commit adds migration design doc which carry information about
the required changes and design for rbd intree to csi migration.

Updates ceph#2509
Signed-off-by: Humble Chirammal <[email protected]>

docs/design/proposals/intree-migrate.md

@@ -0,0 +1,68 @@
+# In-tree storage plugin to CSI Driver Migration
+
+If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. After migration, Kubernetes
+users may continue to rely on all the functionality of in-tree storage plugins using the existing interface. That said,
+In-tree plugin API is untouched, user Pods and PVs continue working after upgrades, thus minimize user visible changes.
+The plan moving forward is to remove the in-tree plugins in kubernetes. CSI migration is the feature in k8s that can
+support a smooth transition from in-tree usage to CSI usage for volume related operations. Without CSI migration and if
+the intree driver is removed, It might cause some problem for them for their existing volume. The plan in kubernetes
+community is to eventually migrate all storage users to CSI. CSI migration will be an opt-in feature turned on at
+cluster creation time that will redirect in-tree plugin operations ( Provision, Attach, Detach, Mount, Unmount (
+including Inline Volumes))
+to a corresponding CSI Driver. The migration library will redirect in-tree plugin usage to appropriate CSI drivers,
+while supporting seamless upgrade and downgrade between new Kubernetes version that uses CSI drivers for in-tree volume
+plugins to an old Kubernetes version that uses old-fashioned volume plugins without CSI, thus facilitates gradual
+roll-over to CSI
+
+## RBD in tree plugin to CSI migration:
+
+This feature is tracked in kubernetes for v1.23 as alpha under
+[kubernetes/enhancements#2963](https://github.com/kubernetes/enhancements/pull/2963). Below are the changes identified
+in the design discussions to enable CSI migration for RBD plugin:
+
+### ClusterID field in the migration request:
+
+Considering the `clusterID` field is a required one for csi, but intree SC has `monitors` field as a required thing the
+clusterID will be sent from the migration library based on the monitors field, kubernetes storage admin supposed to
+create a clusterID based on the monitors hash ( md5sum) in the csi config map and keep the monitors under this
+configuration before enabling the migration. While CSI driver receive the volume handle it will look at the configmap
+and figure out the mons to do the operations. Thus CSI operations are unchanged or untouched wrt to the clusterID field.
+
+### New Volume Handle/ID for existing volume operations:
+
+The existing volume will have a volume handle passed in similar to the following format:
+
+```
+mig_mons-<monitors-hash>_image-<imagename>_<pool-hash>
+```
+
+Details on the hash values:
+
+* MonsHash: this carry a hash value (md5sum) which will be acted as the
+  `clusterID` for the operations in this context.
+
+* ImageUID: this is the unique UUID generated by kubernetes for the created volume.
+
+* PoolHash: this is an encoded string of pool name.
+
+The existing intree volume's node operations should work without any issues as those will be tracked as
+`static volumes` for the csi driver. From above volume handle, the CSI driver can also connect to the backend cluster
+and clean/delete the image. The migration volume ID carry the information like mons, pool and image name which is good
+enough for the driver to identify and connect to the backend cluster for its operations.
+
+### Migration secret for CSI operations:
+
+Considering taking out admin burden is the main ask from kubernetes community this change is introduced.
+
+The intree secret has a data field called "key" which is the base64 admin secret key. The ceph CSI driver currently
+expect the secret to contain data field "UserKey" for the equivalent. The CSI driver also expect the "UserID" field
+which is not available in the in-tree secret by default. This missing userID will be filled (if the username differ
+than 'admin') in the migration secret as 'adminId' field in the migration request, ie:
+
+"key" field value will be picked up from the migration secret to "UserKey"
+field.
+"adminId" field value will be picked up from the migration secret to "UserID"
+field
+
+if `adminId` field is nil or not set, `UserID` field will be filled with default value ie `admin`.The above logic get
+activated only when the secret is a migration secret, otherwise skipped to the normal workflow as we have today.