From 94c882ed6389b5f1284fb7581454646f271e713d Mon Sep 17 00:00:00 2001
From: Patrick Ohly <patrick.ohly@intel.com>
Date: Thu, 24 Feb 2022 22:28:26 +0100
Subject: [PATCH 1/3] storage capacity tracking: GA

---
 keps/prod-readiness/sig-storage/1472.yaml     |  2 +
 .../1472-storage-capacity-tracking/README.md  | 49 ++++++++++---------
 .../1472-storage-capacity-tracking/kep.yaml   | 16 +++---
 3 files changed, 38 insertions(+), 29 deletions(-)

diff --git a/keps/prod-readiness/sig-storage/1472.yaml b/keps/prod-readiness/sig-storage/1472.yaml
index b2d9a5b165f..97a41c92e0b 100644
--- a/keps/prod-readiness/sig-storage/1472.yaml
+++ b/keps/prod-readiness/sig-storage/1472.yaml
@@ -1,3 +1,5 @@
 kep-number: 1472
 beta:
   approver: "@wojtek-t"
+stable:
+  approver: "@wojtek-t"
diff --git a/keps/sig-storage/1472-storage-capacity-tracking/README.md b/keps/sig-storage/1472-storage-capacity-tracking/README.md
index cb4f4a85c4a..f6dc2f67de6 100644
--- a/keps/sig-storage/1472-storage-capacity-tracking/README.md
+++ b/keps/sig-storage/1472-storage-capacity-tracking/README.md
@@ -77,10 +77,10 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
 - [X] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
 - [X] (R) Graduation criteria is in place
 - [X] (R) Production readiness review completed
-- [ ] Production readiness review approved
-- [ ] "Implementation History" section is up-to-date for milestone
-- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
-- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+- [X] Production readiness review approved
+- [X] "Implementation History" section is up-to-date for milestone
+- [X] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [X] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
 
 <!--
 **Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
@@ -806,7 +806,7 @@ checks for events that describe the problem.
 - 5 installs
 - More rigorous forms of testing e.g., downgrade tests and scalability tests
 - Allowing time for feedback
-- Integration with [Cluster Autoscaler](https://github.com/kubernetes/autoscaler)
+- Design for support in [Cluster Autoscaler](https://github.com/kubernetes/autoscaler)
 
 ### Upgrade / Downgrade Strategy
 
@@ -842,15 +842,14 @@ enhancement:
 ### Feature enablement and rollback
 
 * **How can this feature be enabled / disabled in a live cluster?**
-  - [X] Feature gate
-    - Feature gate name: CSIStorageCapacity
-    - Components depending on the feature gate:
-      - apiserver
+  - [X] CSIDriver.StorageCapacity field can be modified
+    - Components depending on the field:
       - kube-scheduler
 
 * **Does enabling the feature change any default behavior?**
 
-  Enabling it only in kube-scheduler and api-server and not any of the
+  Enabling it only in kube-scheduler and api-server by updating
+  to a Kubernetes version where it is enabled and not in any of the
   running CSI drivers causes no changes. Everything continues as
   before because no `CSIStorageCapacity` objects are created and
   kube-scheduler does not wait for any.
@@ -861,12 +860,19 @@ enhancement:
 
 * **Can the feature be disabled once it has been enabled (i.e. can we rollback
   the enablement)?**
-  Yes.
 
-  In Kubernetes 1.19 and 1.20, registration of the
-  `CSIStorageCapacity` type was controlled by the feature gate. In
-  1.21, the type will always be enabled in the v1beta1 API
-  group. Depending on the combination of Kubernetes release and
+  Yes, by disabling it in the CSI driver deployment:
+  `CSIDriver.StorageCapacity=false` causes kube-scheduler to ignore storage
+  capacity for the driver. In addition, external-provisioner can be deployed so
+  that it does not publish capacity information (`--enable-capacity=false`).
+
+  Downgrading to a previous Kubernetes release may also disable the feature or
+  allow disabling it via a feature gate: In Kubernetes 1.19 and 1.20,
+  registration of the `CSIStorageCapacity` type was controlled by the feature
+  gate. In 1.21, the type will always be enabled in the v1beta1 API group. In
+  1.24, the type is always enabled in the v1 API unconditionally.
+
+  Depending on the combination of Kubernetes release and
   feature gate, the type will be disabled. However, any existing
   objects will still remain in the etcd database, they just won't be
   visible.
@@ -934,7 +940,7 @@ consumption, increased latency), specifically
 
 * **Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested?**
 
-Not yet, but will be done manually before transition to beta.
+This was done manually before transition to beta.
 
 * **Is the rollout accompanied by any deprecations and/or removals of features,
   APIs, fields of API types, flags, etc.?**
@@ -951,18 +957,16 @@ scheduling workloads onto nodes, but not while those run.
 That a CSI driver provides storage capacity information can seen in the
 following metric data that will be provided by external-provisioner instances:
 - total number of `CSIStorageCapacity` objects that the external-provisioner
-  is currently meant to manage for the driver
+  is currently meant to manage for the driver: `csistoragecapacities_desired_goal`
 - number of such objects that currently exist and can be kept because
-  they have a topology/storage class pair that is still valid
+  they have a topology/storage class pair that is still valid: `csistoragecapacities_desired_current`
 - number of such objects that currently exist and need to be deleted
-  because they have an outdated topology/storage class pair
-- work queue length for creating, updating or deleting objects
+  because they have an outdated topology/storage class pair: `csistoragecapacities_obsolete`
+- work queue length for creating, updating or deleting objects: `csistoragecapacity` work queue
 
 The CSI driver name will be used as label. When using distributed
 provisioning, the node name will be used as additional label.
 
-TODO: mention the exact metrics names once they are implemented.
-
 * **What are the SLIs (Service Level Indicators) an operator can use to
   determine the health of the service?**
 
@@ -1100,6 +1104,7 @@ to `CSIStorageCapacity` objects.
 - Kubernetes 1.19: alpha
 - Kubernetes 1.21: beta
 - Kubernetes 1.23: `CSIDriver.Spec.StorageCapacity` became mutable.
+- Kubernetes 1.24: GA
 
 ## Drawbacks
 
diff --git a/keps/sig-storage/1472-storage-capacity-tracking/kep.yaml b/keps/sig-storage/1472-storage-capacity-tracking/kep.yaml
index 7a075291ca2..71eb9c07384 100644
--- a/keps/sig-storage/1472-storage-capacity-tracking/kep.yaml
+++ b/keps/sig-storage/1472-storage-capacity-tracking/kep.yaml
@@ -17,21 +17,23 @@ approvers:
   - "@msau42"
 prr-approvers:
   - "@wojtek-t"
-stage: beta
+stage: stable
 see-also:
   - "https://docs.google.com/document/d/1WtX2lRJjZ03RBdzQIZY3IOvmoYiF5JxDX35-SsCIAfg"
-latest-milestone: "v1.21"
+latest-milestone: "v1.24"
 milestone:
   alpha: "v1.19"
   beta: "v1.21"
-  stable: "v1.23"
+  stable: "v1.24"
 feature-gates:
   - name: CSIStorageCapacity
     components:
       - kube-apiserver
       - kube-scheduler
-disable-supported: true
+disable-supported: false
 
-# The following PRR answers are required at beta release
-#metrics:
-#  - my_feature_metric
+metrics:
+  - csistoragecapacities_desired_goal
+  - csistoragecapacities_desired_current
+  - csistoragecapacities_obsolete
+  - csistoragecapacity work queue

From 9a07ab8da2a6057924b29990f6fd18ed4000e700 Mon Sep 17 00:00:00 2001
From: Patrick Ohly <patrick.ohly@intel.com>
Date: Mon, 28 Feb 2022 13:38:57 +0100
Subject: [PATCH 2/3] storage capacity tracking: review feedback

---
 .../sig-storage/1472-storage-capacity-tracking/README.md | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/keps/sig-storage/1472-storage-capacity-tracking/README.md b/keps/sig-storage/1472-storage-capacity-tracking/README.md
index f6dc2f67de6..39660b13d31 100644
--- a/keps/sig-storage/1472-storage-capacity-tracking/README.md
+++ b/keps/sig-storage/1472-storage-capacity-tracking/README.md
@@ -842,6 +842,11 @@ enhancement:
 ### Feature enablement and rollback
 
 * **How can this feature be enabled / disabled in a live cluster?**
+  - [X] Feature gate
+    - Feature gate name: CSIStorageCapacity
+    - Components depending on the feature gate:
+      - apiserver
+      - kube-scheduler
   - [X] CSIDriver.StorageCapacity field can be modified
     - Components depending on the field:
       - kube-scheduler
@@ -940,7 +945,9 @@ consumption, increased latency), specifically
 
 * **Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested?**
 
-This was done manually before transition to beta.
+This was done manually before transition to beta in a kubeadm-based cluster
+running on VMs. The experiment confirmed that rollback and re-enabling works
+as described above, with no unexpected behavior.
 
 * **Is the rollout accompanied by any deprecations and/or removals of features,
   APIs, fields of API types, flags, etc.?**

From 78ae07b40f8da2f7bd08bf8378ea9e116623f998 Mon Sep 17 00:00:00 2001
From: Patrick Ohly <patrick.ohly@intel.com>
Date: Tue, 1 Mar 2022 10:22:35 +0100
Subject: [PATCH 3/3] storage capacity: test examples + SLOs

---
 .../1472-storage-capacity-tracking/README.md           | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/keps/sig-storage/1472-storage-capacity-tracking/README.md b/keps/sig-storage/1472-storage-capacity-tracking/README.md
index 39660b13d31..46830910417 100644
--- a/keps/sig-storage/1472-storage-capacity-tracking/README.md
+++ b/keps/sig-storage/1472-storage-capacity-tracking/README.md
@@ -904,8 +904,10 @@ enhancement:
 * **Are there any tests for feature enablement/disablement?**
   The e2e framework does not currently support enabling and disabling feature
   gates. However, unit tests in each component dealing with managing data created
-  with and without the feature are necessary and will be added before
-  before the transition to beta.
+  with and without the feature are necessary and were added before
+  before the transition to beta, for example
+  [in the apiserver](https://github.com/kubernetes/kubernetes/blob/v1.21.0/pkg/apis/storage/validation/validation_test.go#L2091-L2131)
+  and the [volume binder](https://github.com/kubernetes/kubernetes/blob/v1.21.0/test/integration/volumescheduling/volume_binding_test.go#L706-L709).
 
 ### Rollout, Upgrade and Rollback Planning
 
@@ -992,7 +994,9 @@ calls will be recorded with their non-OK status code as value.
 The goal is to achieve the same provisioning rates with the feature
 enabled as those that currently can be achieved without it.
 
-This will need further discussion before going to GA.
+The SLOs depend on the CSI driver and how they are deployed. Therefore SLOs
+cannot be specified in more detail here. Cloud providers will have to determine
+what reasonable values are and document those.
 
 * **Are there any missing metrics that would be useful to have to improve
   observability if this feature?**