Start adding a KEP for csi volume resizing #780

gnufied · 2019-01-30T02:36:01Z

xref #556

childsb · 2019-01-30T16:51:56Z

/approve

keps/sig-storage/20190129-csi-volume-resizing.md

gnufied · 2019-01-31T04:44:05Z

@saad-ali addressed all the open comments. PTAL. thanks.

keps/sig-storage/20190129-csi-volume-resizing.md

msau42 · 2019-02-01T21:30:31Z

keps/sig-storage/20190129-csi-volume-resizing.md

+The design of CSI volume resizing is made of two parts.
+
+
+#### External resize controller


indent should be one level up

msau42 · 2019-02-01T21:30:45Z

keps/sig-storage/20190129-csi-volume-resizing.md

+
+#### External resize controller
+
+To support resizing of CSI volumes an external resize controller will monitor all PVCs.  If a PVC meets following critirea for resizing, it will be added to


msau42 · 2019-02-01T21:31:51Z

keps/sig-storage/20190129-csi-volume-resizing.md

+
+#### External resize controller
+
+To support resizing of CSI volumes an external resize controller will monitor all PVCs.  If a PVC meets following critirea for resizing, it will be added to


Does the internal resize controller need any changes to ignore CSI volumes?

no - this was resolved couple of releases back when another sig-storage member wanted to externally resize volumes.

msau42 · 2019-02-01T21:33:19Z

keps/sig-storage/20190129-csi-volume-resizing.md

+To support resizing of CSI volumes an external resize controller will monitor all PVCs.  If a PVC meets following critirea for resizing, it will be added to
+controller's workqueue:
+
+- The driver name disovered from PVC should match name of driver currently known(by querying driver info via CSI RPC call) to external resize controller.


How does it get the driver name? From StorageClass?

Let's clarify that.

keps/sig-storage/20190129-csi-volume-resizing.md

msau42 · 2019-02-01T22:36:28Z

lgtm please squash

Clarify dependency on online resizing.

gnufied · 2019-02-02T02:30:45Z

@msau42 done!

msau42

/lgtm

k8s-ci-robot · 2019-02-02T02:59:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: childsb, gnufied, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-storage/OWNERS~~ [childsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

saad-ali

Hmm looks like I forgot to hit submit on this.

You can address these in a follow up PR

saad-ali · 2019-02-02T01:24:12Z

keps/sig-storage/20190129-csi-volume-resizing.md

+
+[umbrella issues]: https://github.com/kubernetes/kubernetes/issues/62096
+
+## Implementation History


Please update the Implementation History to clarify what has been implemented.

saad-ali · 2019-02-02T01:26:15Z

keps/sig-storage/20190129-csi-volume-resizing.md

+
+### External resize controller
+
+To support resizing of CSI volumes an external resize controller will monitor all PVCs.  If a PVC meets following criteria for resizing, it will be added to


nit: monitor all changes to PVCs

saad-ali · 2019-02-02T01:26:34Z

keps/sig-storage/20190129-csi-volume-resizing.md

+To support resizing of CSI volumes an external resize controller will monitor all PVCs.  If a PVC meets following critirea for resizing, it will be added to
+controller's workqueue:
+
+- The driver name disovered from PVC should match name of driver currently known(by querying driver info via CSI RPC call) to external resize controller.


Let's clarify that.

saad-ali · 2019-02-02T01:27:30Z

keps/sig-storage/20190129-csi-volume-resizing.md

+- The driver name disovered from PVC should match name of driver currently known(by querying driver info via CSI RPC call) to external resize controller.
+- Once it notices a PVC has been updated and by comparing old and new PVC object, it determines more space has been requested by the user.
+
+Once PVC gets picked from workqueue, the controller will also compare requested PVC size with actual size of volume in `PersistentVolume`


What does this check do?

We also need to check PV size before actually doing controller resize because it is possible that, volume already has been resized and controller is just resyncing PVCs.

saad-ali · 2019-02-02T01:29:25Z

keps/sig-storage/20190129-csi-volume-resizing.md

+object. Once PVC passes all these checks, a CSI `ControllerExpandVolume` call will be made by the controller if CSI plugin implements `ControllerExpandVolume`
+RPC call.
+
+If `ControllerExpandVolume` call is successful and plugin implements `NodeExpandVolume`:


(i.e. the CSI driver has a EXPAND_VOLUME node capability)

yeah, I did not use use capabilities in this document on purpose because I thought "plugins implements NodeExpandVolume "is synonymous with "plugin has EXPAND_VOLUME node capability".

saad-ali · 2019-02-02T01:31:03Z

keps/sig-storage/20190129-csi-volume-resizing.md

+- if `ControllerExpandVolumeResponse` returns `true` in `node_expansion_required` then `FileSystemResizePending` condition will be added to PVC and `NodeExpandVolume` operation will be queued on kubelet. Also volume size reported by PV will be updated to new value.
+- if `ControllerExpandVolumeResponse` returns `false` in `node_expansion_required` then volume resize operation will be marked finished and both `pvc.Status.Capacity` and `pv.Spec.Capacity` will report updated value.
+
+If plugin does not implement `NodeExpandVolume` then volume resize operation will be marked as finished and both `pvc.Status.Capacity` and `pv.Spec.Capacity` will report updated value after successful completion of `ControllerExpandVolume` RPC call.


If ControllerExpandVolume call is successful and the plugin does not implement...

saad-ali · 2019-02-02T01:36:38Z

keps/sig-storage/20190129-csi-volume-resizing.md

+
+#### Offline volume resizing on kubelet:
+
+This is the default mode and in this mode `NodeExpandVolume` will only be called when volume is being mounted on the node. In other words, pod that was using the volume must be re-created for expansion on node to happen.


What causes kubelet to start this process of calling NodeExpandVolume? Based on above it seems like some component (mount operation?) watches the PVC and triggers this?

Yeah - offline resizing is done as part of MountVolume operation (in operation_executor.go). The actual checks we perform are outlined below. I am going to update this document to be more explicit.

saad-ali · 2019-02-02T01:38:52Z

keps/sig-storage/20190129-csi-volume-resizing.md

+RPC call.
+
+If `ControllerExpandVolume` call is successful and plugin implements `NodeExpandVolume`:
+- if `ControllerExpandVolumeResponse` returns `true` in `node_expansion_required` then `FileSystemResizePending` condition will be added to PVC and `NodeExpandVolume` operation will be queued on kubelet. Also volume size reported by PV will be updated to new value.


So kubelet will wait until the FileSystemResizePending condition is applied to PVC? If so, this smells fishy. My understanding was that conditions were informative for end user and not to be used to trigger behavior in other components?

No kubelet does not check for FileSystemResiznePending condition saad. From proposal:

When a pod that is using the PVC is started, kubelet will compare pvc.spec.resources.requests.storage and pvc.Status.Capacity. It also compares PVC's size with pv.Spec.Capacity and if it detects PV is reporting same size as pvc's spec but PVC's status is still reporting smaller value then it determines - a volume expansion is pending on the node. At this point if plugin implements NodeExpandVolume RPC call then, kubelet will call it

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 30, 2019

k8s-ci-robot requested review from childsb and saad-ali January 30, 2019 02:36

gnufied changed the title ~~{WIP} Start adding a KEP for csi volume resizing~~ Start adding a KEP for csi volume resizing Jan 30, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 30, 2019

gnufied mentioned this pull request Jan 30, 2019

Add resizing support to CSI volumes #556

Open

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 30, 2019

saad-ali reviewed Jan 30, 2019

View reviewed changes

msau42 reviewed Jan 31, 2019

View reviewed changes

keps/sig-storage/20190129-csi-volume-resizing.md Show resolved Hide resolved

keps/sig-storage/20190129-csi-volume-resizing.md Outdated Show resolved Hide resolved

keps/sig-storage/20190129-csi-volume-resizing.md Outdated Show resolved Hide resolved

leakingtapan mentioned this pull request Jan 31, 2019

Add Volume Resizing Support kubernetes-sigs/aws-ebs-csi-driver#146

Closed

msau42 reviewed Feb 1, 2019

View reviewed changes

Start adding a KEP for csi volume resizing

c4f4b1b

Clarify dependency on online resizing.

gnufied force-pushed the csi-volume-resize branch from fb99f34 to c4f4b1b Compare February 2, 2019 02:30

k8s-ci-robot assigned msau42 Feb 2, 2019

msau42 reviewed Feb 2, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2019

k8s-ci-robot merged commit 4ba2a6f into kubernetes:master Feb 2, 2019

saad-ali reviewed Feb 13, 2019

View reviewed changes

gnufied mentioned this pull request Feb 19, 2019

Add support for NodeExpand kubernetes-csi/external-resizer#10

Closed

gnufied mentioned this pull request Feb 19, 2019

Add support for CSI volume expansion in kubelet kubernetes/kubernetes#74264

Closed

justaugustus removed the sig/pm label Apr 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start adding a KEP for csi volume resizing #780

Start adding a KEP for csi volume resizing #780

gnufied commented Jan 30, 2019

childsb commented Jan 30, 2019

gnufied commented Jan 31, 2019

msau42 Feb 1, 2019

msau42 Feb 1, 2019

msau42 Feb 1, 2019

gnufied Feb 1, 2019

msau42 Feb 1, 2019

gnufied Feb 1, 2019

saad-ali Feb 2, 2019

msau42 commented Feb 1, 2019

gnufied commented Feb 2, 2019

msau42 left a comment

k8s-ci-robot commented Feb 2, 2019

saad-ali left a comment

saad-ali Feb 2, 2019

saad-ali Feb 2, 2019

saad-ali Feb 2, 2019

saad-ali Feb 2, 2019

gnufied Feb 13, 2019

saad-ali Feb 2, 2019

gnufied Feb 13, 2019

saad-ali Feb 2, 2019

saad-ali Feb 2, 2019

gnufied Feb 13, 2019

saad-ali Feb 2, 2019

gnufied Feb 13, 2019 •

edited

Loading

		The design of CSI volume resizing is made of two parts.


		#### External resize controller


		#### External resize controller

		To support resizing of CSI volumes an external resize controller will monitor all PVCs. If a PVC meets following critirea for resizing, it will be added to


		[umbrella issues]: https://github.com/kubernetes/kubernetes/issues/62096

		## Implementation History


		### External resize controller

		To support resizing of CSI volumes an external resize controller will monitor all PVCs. If a PVC meets following criteria for resizing, it will be added to


		#### Offline volume resizing on kubelet:

		This is the default mode and in this mode `NodeExpandVolume` will only be called when volume is being mounted on the node. In other words, pod that was using the volume must be re-created for expansion on node to happen.

Start adding a KEP for csi volume resizing #780

Start adding a KEP for csi volume resizing #780

Conversation

gnufied commented Jan 30, 2019

childsb commented Jan 30, 2019

gnufied commented Jan 31, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Feb 1, 2019

gnufied commented Feb 2, 2019

msau42 left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 2, 2019

saad-ali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied Feb 13, 2019 • edited Loading

Choose a reason for hiding this comment

gnufied Feb 13, 2019 •

edited

Loading