doc: add migration design documentation

This commit adds migration design doc which carry information about
the required changes and design for rbd intree to csi migration.

Updates ceph#2509
Signed-off-by: Humble Chirammal <[email protected]>

docs/design/proposals/intree-migrate.md

@@ -0,0 +1,95 @@
+# In-tree storage plugin to CSI Driver Migration
+
+Prior to CSI, Kubernetes provided a powerful volume plugin system. These volume
+plugins were “in-tree” meaning their code was part of the core Kubernetes code
+and shipped with the core Kubernetes binaries. The CSI migration effort enables
+the replacement of existing in-tree storage plugins such as kubernetes.io/rbd
+with a corresponding CSI driver. If CSI Migration is working properly,
+Kubernetes end users shouldn’t notice a difference. After migration, Kubernetes
+users may continue to rely on all the functionality of in-tree storage plugins
+using the existing interface. That said, In-tree plugin API is untouched, user
+Pods and PVs continue working after upgrades, thus minimize user visible
+changes. When a Kubernetes cluster administrator updates a cluster to enable CSI
+migration, existing stateful deployments and workloads continue to function as
+they always have; however, behind the scenes Kubernetes hands control of all
+storage management operations (previously targeting in-tree drivers) to CSI
+drivers.
+
+The plan moving forward is to remove the in-tree plugins in kubernetes. CSI
+migration is the feature in k8s that can support a smooth transition from
+in-tree usage to CSI usage for volume related operations. Without CSI migration
+and if the intree driver is removed, It might cause some problem for users for
+their existing volume. The plan in kubernetes community is to eventually migrate
+all storage users to CSI. CSI migration will be an opt-in feature turned on at
+cluster creation time that will redirect in-tree plugin operations (Provision,
+Attach, Detach, Mount, Unmount(including Inline Volumes))
+to a corresponding CSI Driver. The migration library will redirect in-tree
+plugin usage to appropriate CSI drivers, while supporting seamless upgrade and
+downgrade between new Kubernetes version that uses CSI drivers for in-tree
+volume plugins to an old Kubernetes version that uses old-fashioned volume
+plugins without CSI, thus facilitates gradual roll-over to CSI
+
+## RBD in tree plugin to CSI migration
+
+This feature is tracked in kubernetes for v1.23 as alpha under
+[kubernetes/enhancements#2963](https://github.com/kubernetes/enhancements/pull/2963)
+. Below are the changes identified in the design discussions to enable CSI
+migration for RBD plugin:
+
+### ClusterID field in the migration request
+
+Considering the `clusterID` field is a required one for csi, but intree SC
+has `monitors` field as a required thing the clusterID will be sent from the
+migration library based on the monitors field, kubernetes storage admin supposed
+to create a clusterID based on the monitors hash ( md5sum) in the csi config map
+and keep the monitors under this configuration before enabling the migration.
+While CSI driver receive the volume handle it will look at the configmap and
+figure out the mons to do the operations. Thus CSI operations are unchanged or
+untouched wrt to the clusterID field.
+
+### New Volume Handle/ID for existing volume operations
+
+The existing volume will have a volume handle passed in similar to the following
+format:
+
+```
+mig_mons-<monitors-hash>_image-<imageUID>_<pool-hash>
+```
+
+Details on the hash values:
+
+* Monitors Hash: this carry a hash value (md5sum) which will be acted as the
+  `clusterID` for the operations in this context.
+
+* ImageUID: this is the unique UUID generated by kubernetes for the created
+  volume.
+
+* PoolHash: this is an encoded string of pool name.
+
+The existing intree volume's node operations should work without any issues as
+those will be tracked as `static volumes` for the csi driver. From above volume
+handle, the CSI driver can also connect to the backend cluster and clean/delete
+the image. The migration volume ID carry the information like mons, pool and
+image name which is good enough for the driver to identify and connect to the
+backend cluster for its operations.
+
+### Migration secret for CSI operations
+
+Considering taking out admin burden is the main ask from kubernetes community
+this change is introduced.
+
+The intree secret has a data field called "key" which is the base64 admin secret
+key. The ceph CSI driver currently expect the secret to contain data field "
+UserKey" for the equivalent. The CSI driver also expect the "UserID" field which
+is not available in the in-tree secret by default. This missing userID will be
+filled (if the username differ than 'admin') in the migration secret as '
+adminId' field in the migration request, ie:
+
+"key" field value will be picked up from the migration secret to "UserKey"
+field.
+"adminId" field value will be picked up from the migration secret to "UserID"
+field
+
+if `adminId` field is nil or not set, `UserID` field will be filled with default
+value ie `admin`.The above logic get activated only when the secret is a
+migration secret, otherwise skipped to the normal workflow as we have today.