[ISSUE-1018] Refine Storage Group Feature (#1044)

* [ISSUE-1040] fix pip3 install PyYAML failed. (#1041) Signed-off-by: yimingwangdell <[email protected]> * [ISSUE-1018] Refine Storage Group Feature (#1031) * add StorageGroupStatus Signed-off-by: Shi, Crane <[email protected]> * support StorageGroupStatus in current workflows Signed-off-by: Shi, Crane <[email protected]> * refine log Signed-off-by: Shi, Crane <[email protected]> * refine error handling Signed-off-by: Shi, Crane <[email protected]> * trigger storage group resync if applicable in drive removal Signed-off-by: Shi, Crane <[email protected]> * A drive whose Usage is REMOVED will not be selected in any storage group and its existing sg label takes no effect Signed-off-by: Shi, Crane <[email protected]> * sg feature will not apply to drive physically removed Signed-off-by: Shi, Crane <[email protected]> * handle the drive removal case of drive with manual sg label Signed-off-by: Shi, Crane <[email protected]> * fix go lint error Signed-off-by: Shi, Crane <[email protected]> * add UT case for drive-removal-triggered sg sync Signed-off-by: Shi, Crane <[email protected]> * improve UT coverage Signed-off-by: Shi, Crane <[email protected]> * refine sg annotation for drive removal Signed-off-by: Shi, Crane <[email protected]> * handle case of invalid sg for drive removal Signed-off-by: Shi, Crane <[email protected]> * also exclude removing sg for trigger sg resync in drive removal Signed-off-by: Shi, Crane <[email protected]> * refine sg removal status handling Signed-off-by: Shi, Crane <[email protected]> * Revert "refine error handling" This reverts commit 06607e7. Signed-off-by: Shi, Crane <[email protected]> * refine log and some code logic Signed-off-by: Shi, Crane <[email protected]> * try to add immutability validation rule to storagegroup spec Signed-off-by: Shi, Crane <[email protected]> * upgrade controller-gen version to v0.9.2 Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT initial Signed-off-by: Shi, Crane <[email protected]> * Revert "add storagegroupcontroller UT initial" This reverts commit 1ea8660. Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * fix Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * refactor and add UT of storagegroupcontroller Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * fix storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * refine the logic of sg deletion Signed-off-by: Shi, Crane <[email protected]> * refine Signed-off-by: Shi, Crane <[email protected]> * fix bug Signed-off-by: Shi, Crane <[email protected]> * fix go-lint err Signed-off-by: Shi, Crane <[email protected]> * fix go-lint error Signed-off-by: Shi, Crane <[email protected]> * add drive IsClean support, decrease k8s api call, remove manual sg labeling support Signed-off-by: Shi, Crane <[email protected]> * fix Signed-off-by: Shi, Crane <[email protected]> * fix Signed-off-by: Shi, Crane <[email protected]> * fix UT Signed-off-by: Shi, Crane <[email protected]> * refine corner case handling Signed-off-by: Shi, Crane <[email protected]> * fix Signed-off-by: Shi, Crane <[email protected]> * refine and add UT to storagegroupcontroller Signed-off-by: Shi, Crane <[email protected]> * refine storagegroupcontroller and add UT Signed-off-by: Shi, Crane <[email protected]> * make controller svc's k8scache also sync sg and lvg objs' Signed-off-by: Shi, Crane <[email protected]> * use k8s cache, re-support sg label manual change and refine in sg ctrl Signed-off-by: Shi, Crane <[email protected]> * fix lint err Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * add storagegroupcontroller UT Signed-off-by: Shi, Crane <[email protected]> * storagegroup controller will not reconcile on drive delete event Signed-off-by: Shi, Crane <[email protected]> * not support Health, Status, Usage and IsClean as DriveSelector's MatchFields Signed-off-by: Shi, Crane <[email protected]> * in storagegroupcontroller's reconcile, only sync drive when reqName is uuid Signed-off-by: Shi, Crane <[email protected]> * refine the logic to avoid nil pointer error Signed-off-by: Shi, Crane <[email protected]> * revert the usage of k8scache Signed-off-by: Shi, Crane <[email protected]> * add custom storage group proposal draft Signed-off-by: Shi, Crane <[email protected]> * refine custom storage group proposal Signed-off-by: Shi, Crane <[email protected]> --------- Signed-off-by: CraneShiEMC <[email protected]> --------- Signed-off-by: yimingwangdell <[email protected]> Signed-off-by: CraneShiEMC <[email protected]> Co-authored-by: yimingwangdell <[email protected]>
dell · Jul 31, 2023 · fc9d4f4 · fc9d4f4
1 parent 2fa2836
commit fc9d4f4
Show file tree

Hide file tree

Showing 11 changed files with 1,293 additions and 271 deletions.
diff --git a/api/generated/v1/types.pb.go b/api/generated/v1/types.pb.go
diff --git a/api/v1/constants.go b/api/v1/constants.go
@@ -141,4 +141,14 @@ const (
 
 	// CSI StorageGroup label key
 	StorageGroupLabelKey = "drive.csi-baremetal.dell.com/storage-group"
+
+	// CSI StorageGroup Status
+	StorageGroupPhaseSyncing  = "SYNCING"
+	StorageGroupPhaseSynced   = "SYNCED"
+	StorageGroupPhaseRemoving = "REMOVING"
+	StorageGroupPhaseInvalid  = "INVALID"
+
+	// CSI StorageGroup annotations
+	StorageGroupAnnotationDriveRemovalPrefix = "drive-removal"
+	StorageGroupAnnotationDriveRemovalDone   = "done"
 )
diff --git a/api/v1/storagegroupcrd/storagegroup_types.go b/api/v1/storagegroupcrd/storagegroup_types.go
@@ -26,20 +26,21 @@ import (
 // StorageGroup is the Schema for the StorageGroups API
 // +kubebuilder:resource:scope=Cluster,shortName={sg,sgs}
 // +kubebuilder:printcolumn:name="DRIVES_PER_NODE",type="string",JSONPath=".spec.driveSelector.numberDrivesPerNode",description="numberDrivesPerNode of StorageGroup's DriveSelector"
-// +kubebuilder:printcolumn:name="TYPE",type="string",JSONPath=".spec.driveSelector.matchFields.Type",description="Drive Type of StorageGroup's DriveSelector"
-// +kubebuilder:printcolumn:name="SLOT",type="string",JSONPath=".spec.driveSelector.matchFields.Slot",description="Drive Slot of StorageGroup's DriveSelector"
-// +kubebuilder:printcolumn:name="PATH",type="string",JSONPath=".spec.driveSelector.matchFields.Path",description="Drive Path of StorageGroup's DriveSelector"
-// +kubebuilder:printcolumn:name="SYSTEM",type="string",JSONPath=".spec.driveSelector.matchFields.IsSystem",description="Whether StorageGroup's DriveSelector to Select System Drive"
+// +kubebuilder:printcolumn:name="DRIVE_FIELDS",type="string",JSONPath=".spec.driveSelector.matchFields",description="Match Fields of StorageGroup's DriveSelector to Select Drives on Field Values"
+// +kubebuilder:printcolumn:name="STATUS",type="string",JSONPath=".status.phase",description="status of StorageGroup"
 type StorageGroup struct {
 	metav1.TypeMeta   `json:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty"`
-	Spec              api.StorageGroupSpec `json:"spec,omitempty"`
+
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="updates to storagegroup spec are forbidden"
+	Spec   api.StorageGroupSpec   `json:"spec,omitempty"`
+	Status api.StorageGroupStatus `json:"status,omitempty"`
 }
 
 // +kubebuilder:object:root=true
 
 // StorageGroupList contains a list of StorageGroup
-//+kubebuilder:object:generate=true
+// +kubebuilder:object:generate=true
 type StorageGroupList struct {
 	metav1.TypeMeta `json:",inline"`
 	metav1.ListMeta `json:"metadata,omitempty"`

diff --git a/api/v1/types.proto b/api/v1/types.proto
@@ -108,3 +108,7 @@ message DriveSelector {
     int32 numberDrivesPerNode = 1;
     map<string, string> matchFields = 2;
 }
+
+message StorageGroupStatus {
+    string phase = 1;
+}
diff --git a/docs/proposals/custom-storage-group.md b/docs/proposals/custom-storage-group.md
@@ -0,0 +1,30 @@
+# Proposal: Custom Storage Group 
+
+Last updated: 15-05-2023
+
+
+## Abstract
+
+At present, for each PVC, CSI just chooses one of those physical drives whose available capacity meets the requested
+storage size of PVC to provision corresponding volume and doesn't support the functionality to provision volume on the physical drive specified by user.
+Here, we propose a custom-storage-group solution that user can select specific physical drives to provision their volumes.
+
+## Background
+
+To use CSI to create persistent volumes, you need to specify the storage classes managed by CSI in your k8s volume specifications. All the current CSI-managed storage classes are pre-defined storage classes created during CSI installation, based on local disk types including HDD, SSD, NVME and whether to use 1 entire local drive or create 1 logical volume for the k8s persistent volume.
+In our current Bare-metal CSI, users can only have PVC satisfying their basic request based on storage type and size and cannot select the specific drive for their PVC. Instead, for each PVC, CSI will select 1 among those local drives satisfying the request of PVC. Furthermore, for LVG PVCs, the current CSI’s drive selection strategy is that it tries to accumulate those PVCs on the drive with existing LVG PVCs and the least free capacity if applicable.
+This strategy aims to leave more entire free disks, which is useful if there would be requirements of non-LVG PVCs later, but it cannot satisfy the requirement of selecting specific drives in volume provisioning.
+
+## Proposal
+
+The main idea of custom storage group solution is that, to make PVCs land on some specific drives, we can firstly create StorageGroup with driveSelector selecting those specific drives and then just add the corresponding storage-group k8s label in the spec of volume claims.
+
+Here, StorageGroup is a new custom resource in Baremetal CSI.
+
+User can create custom storage group on specific drives by specifying some criteria to select drives on properties via driveSelector field of StorageGroup.
+
+Then, the corresponding custom-storage-group k8s label **drive.csi-baremetal.dell.com/storage-group: ${storage-group-name}** would be immediately synced to all the CSI Drives selected in this storage group by the storage group controller after each storage group created in kubernetes. Actually, all the corresponding custom-storage-group k8s label would then be also further immediately synced to the drives' corresponding AC objects.
+
+Here, we actually use the storage-group k8s label to symbolize the drives list in the corresponding storage group, which can be conveniently listed by k8s list api with labelSelector.
+
+Then, to make PVCs land on those specific drives selected, just add the corresponding custom storage group label **drive.csi-baremetal.dell.com/storage-group: ${storage-group-name}** in volume claim spec. Then, CSI will ensure the custom-storage-group label matching in selecting drives for PVCs, and PVC with k8s label **drive.csi-baremetal.dell.com/storage-group: ${storage-group-name}** would only reside on one of those specific drives with the same k8s label **drive.csi-baremetal.dell.com/storage-group: ${storage-group-name}**.