statefulset persistent volume claim resize #110522

areller · 2022-06-12T04:00:35Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Today when resizing a persistent volume claim that's created by a stateful set template, by modifying the size in the template, the users gets back a generic error, saying that they're only allowed to modify number of replicas or statefulset spec template.
This PR allows users to also modify the request size in a PVC template of a statefulset, and it modifies the statefulset controller to be able to reconcile request size differences (i.e. patches the PVC if the size in the template changes)

In addition,

Those kind of changes aren't affected by rollback (i.e. if you rollback a PVC resize, it won't actually resize back)
If the PVC patch fails during statefulset reconciliation (e.g. the user tries to decrease the PVC size and the storage driver doesn't support that), all other changes to the statefulset are blocked from being reconciled, until the user addresses the other (e.g. increase the PVC size back up in the statefulset template)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2022-06-12T04:00:36Z

@areller: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2022-06-12T04:00:43Z

@areller: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2022-06-12T04:00:43Z

Hi @areller. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2022-06-12T04:00:50Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: areller
To complete the pull request process, please assign liggitt after the PR has been reviewed.
You can assign the PR to them by writing /assign @liggitt in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2022-07-15T22:33:02Z

@areller: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jaypipes · 2022-09-13T15:00:57Z

pkg/controller/statefulset/stateful_set_control.go

+			resizing := false
+			if pvcActual.Spec.Resources.Requests.Storage().Equal(*pvc.Spec.Resources.Requests.Storage()) {
+				continue
+			} else {
+				resizing = true
+			}


unless I'm mistaken, you can just shorten this to:

curSize := pvcActual.Spec.Resources.Requests.Storage() newSize := pvc.Spec.Resources.Requests.Storage() if curSize.Equal(*newSize) { continue }

jaypipes · 2022-09-13T15:05:04Z

pkg/controller/statefulset/stateful_set_control.go

+			if err != nil {
+				err = fmt.Errorf("failed to resize PVC %s: %s", claimName, err)
+				errs = append(errs, err)
+				ssc.podControl.recordClaimEvent("resize", set, pod, pvcActual, err)
+				continue
+			}
+
+			if resizing {
+				ssc.podControl.recordClaimEvent("resize", set, pod, pvcActual, nil)
+			}


You can shorten this whole block to just this:

if err != nil { err = fmt.Errorf(("failed to resize PVC %s", claimName, err) errs - append(errs, err) } ssc.podControl.recordClaimEvent("resize", set, pod, pvc, err)

}

Agree, resizing var looks like no necessary here, at this point(line 637), we can be sure that the pvc has been resized

jaypipes · 2022-09-13T15:07:24Z

pkg/controller/statefulset/stateful_set_control.go

+				resizing = true
+			}
+
+			patch := fmt.Sprintf(`{"spec": {"resources": {"requests": {"storage": "%s"}}}}`, pvc.Spec.Resources.Requests.Storage().String())


If you take my suggested code above, you can do:

patch := fmt.Sprintf(`{"spec": {"resources": {"requests": {"storage": "%s"}}}}`, curSize)

areller · 2022-09-13T21:51:00Z

Thank you @jaypipes. these are good suggestions
I'm waiting for kubernetes/enhancements#3412 to be reviewed/approved
this PR is a draft, I'm not sure yet if I'll use it or open a new one for the full implementation, but I'll keep your suggestions in mind

yuchengwu · 2022-10-10T06:29:47Z

pkg/controller/statefulset/stateful_set_control.go

@@ -592,6 +598,51 @@ func (ssc *defaultStatefulSetControl) updateStatefulSet(
 	return &status, nil
 }

+func (ssc *defaultStatefulSetControl) resizePVCs(set *apps.StatefulSet, updateRevision *apps.ControllerRevision, pods []*v1.Pod) error {


If updateRevision is not used in this block, it's better not to pass in as a function argument

yuchengwu · 2022-10-10T06:33:39Z

pkg/controller/statefulset/stateful_set_control.go

+			}
+
+			patch := fmt.Sprintf(`{"spec": {"resources": {"requests": {"storage": "%s"}}}}`, pvc.Spec.Resources.Requests.Storage().String())
+			err = ssc.podControl.objectMgr.PatchClaim(set.Namespace, claimName, []byte(patch))


Why we need an extra PatchClaim method while there is already exists UpdateClaim

yuchengwu · 2022-10-10T06:44:44Z

pkg/controller/statefulset/stateful_set_control.go

+			if err != nil {
+				err = fmt.Errorf("failed to resize PVC %s: %s", claimName, err)
+				errs = append(errs, err)
+				ssc.podControl.recordClaimEvent("resize", set, pod, pvcActual, err)
+				continue
+			}
+
+			if resizing {
+				ssc.podControl.recordClaimEvent("resize", set, pod, pvcActual, nil)
+			}


Agree, resizing var looks like no necessary here, at this point(line 637), we can be sure that the pvc has been resized

yuchengwu · 2022-10-10T06:53:00Z

pkg/controller/statefulset/stateful_set_control.go

+	if err != nil {
+		return nil, err
+	}
+


if this err var is not used in other place, restrict its scope by rewriting like

if err := xxx(); err != nil { }

yuchengwu · 2022-10-10T07:06:17Z

pkg/controller/statefulset/stateful_pod_control.go

@@ -301,15 +308,22 @@ func (spc *StatefulPodControl) recordPodEvent(verb string, set *apps.StatefulSet
 // nil the generated event will have a reason of v1.EventTypeNormal. If err is not nil the generated event will have a
 // reason of v1.EventTypeWarning.
 func (spc *StatefulPodControl) recordClaimEvent(verb string, set *apps.StatefulSet, pod *v1.Pod, claim *v1.PersistentVolumeClaim, err error) {
+	spc.recordUnavailableClaimEvent(verb, set, pod, claim.Name, err)


Emm.. it's just werid to have an err var as a method argument, why we need wrap this func body as recordUnavailableClaimEvent

yuchengwu · 2022-10-10T07:10:56Z

pkg/controller/statefulset/stateful_set_control.go

@@ -329,6 +330,11 @@ func (ssc *defaultStatefulSetControl) updateStatefulSet(
 		// If the ordinal could not be parsed (ord < 0), ignore the Pod.
 	}

+	err = ssc.resizePVCs(set, updateRevision, replicas)
+	if err != nil {
+		return nil, err


not sure if ok to return nil status, does the return value been used by the caller ?

yuchengwu · 2022-10-10T10:06:46Z

Shall we respect the .spec.updateStrategy when resizing

dims · 2022-12-12T15:28:58Z

If you still need this PR then please rebase, if not, please close the PR

dims · 2022-12-12T15:47:20Z

This PR has the label work-in-progress, please revisit to see if you still need this, please close if not

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. labels Jun 12, 2022

k8s-ci-robot added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Jun 12, 2022

k8s-ci-robot requested review from dchen1107 and krmayankk June 12, 2022 04:00

k8s-ci-robot added sig/auth Categorizes an issue or PR as relevant to SIG Auth. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 12, 2022

areller added 4 commits June 12, 2022 04:59

resize pvc from sts

b4db06f

add feature flag + use in validation

8f26d56

start testing

9269f8f

more tests

8758086

areller force-pushed the sts-pvc-resize branch from c28009c to 8758086 Compare June 12, 2022 05:01

k8s-ci-robot removed the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Jun 12, 2022

areller mentioned this pull request Jun 12, 2022

KEP-0661: StatefulSet volume resize kubernetes/enhancements#2842

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2022

jaypipes reviewed Sep 13, 2022

View reviewed changes

yuchengwu reviewed Oct 10, 2022

View reviewed changes

MegaByte875 mentioned this pull request Oct 12, 2022

volume size reconcile support vesoft-inc/nebula-operator#158

Closed

areller closed this Dec 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statefulset persistent volume claim resize #110522

statefulset persistent volume claim resize #110522

areller commented Jun 12, 2022 •

edited

Loading

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jul 15, 2022

jaypipes Sep 13, 2022

jaypipes Sep 13, 2022

yuchengwu Oct 10, 2022

jaypipes Sep 13, 2022

areller commented Sep 13, 2022

yuchengwu Oct 10, 2022

yuchengwu Oct 10, 2022

yuchengwu Oct 10, 2022

yuchengwu Oct 10, 2022

yuchengwu Oct 10, 2022

yuchengwu Oct 10, 2022 •

edited

Loading

yuchengwu commented Oct 10, 2022

dims commented Dec 12, 2022

dims commented Dec 12, 2022

statefulset persistent volume claim resize #110522

statefulset persistent volume claim resize #110522

Conversation

areller commented Jun 12, 2022 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jun 12, 2022

k8s-ci-robot commented Jul 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

areller commented Sep 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuchengwu Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

yuchengwu commented Oct 10, 2022

dims commented Dec 12, 2022

dims commented Dec 12, 2022

areller commented Jun 12, 2022 •

edited

Loading

yuchengwu Oct 10, 2022 •

edited

Loading