Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

avoid concurrent processing of same PVC #6

Merged
merged 1 commit into from
Jul 29, 2019
Merged

avoid concurrent processing of same PVC #6

merged 1 commit into from
Jul 29, 2019

Conversation

mlmhl
Copy link
Contributor

@mlmhl mlmhl commented Jan 10, 2019

This PR try to handle double processing of PVCs correctly.

The resize controller will receive PVC Update events periodically due to SharedInformer's resync mechanism, when handling these events, we need to check PVC's ResourceVersion to avoid race condition between controller and kubelet.

xref: #4

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 10, 2019
@mlmhl
Copy link
Contributor Author

mlmhl commented Jan 10, 2019

/assign @gnufied

@gnufied
Copy link
Contributor

gnufied commented Feb 6, 2019

The PR looks okay but I am thinking if there is a better way of handling this. Lets not merge this for now until we consider other approaches. This PR is not blocking us from doing rest of the external resizer work, so we can merge this later as well.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 7, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mlmhl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 11, 2019
@mlmhl
Copy link
Contributor Author

mlmhl commented Jun 21, 2019

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 21, 2019
@gnufied
Copy link
Contributor

gnufied commented Jun 26, 2019

I have been thinking about this and I think rather than doing a previous version check etc, we should always default to doing a resourceVersion check while patching PVC and PV in this PR. This would solve majority of duplicate processing problems and I think it should be safe enough. @mlmhl what do you think? Can you update this PR to do that?

@mlmhl
Copy link
Contributor Author

mlmhl commented Jun 27, 2019

@gnufied Agreed. And the PR is already updated, PTAL :)

gnufied
gnufied reviewed Jun 27, 2019
View reviewed changes
pkg/controller/controller.go Outdated
// If pvc's ResourceVersion is equal to last processed ResourceVersion, it means this is a resync event,
// so we need to check PVC's version when patch its status to avoid double process on same PVC.
// More details see: https://github.com/kubernetes-csi/external-resizer/issues/4.
needVersionCheck := ctrl.lastProcessedPVCVersions.Get(key) == pvc.ResourceVersion
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant, lets not do all this stuff. Just add resource version to PVC and PV patch unconditionally. Let the patch operation fail and retry if we had stale object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, got it. Updated.

@mlmhl mlmhl force-pushed the consistency branch 3 times, most recently from 14f8899 to 97dc262 Compare July 1, 2019 03:10
@gnufied
Copy link
Contributor

gnufied commented Jul 29, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 29, 2019
@k8s-ci-robot k8s-ci-robot merged commit ee88e71 into kubernetes-csi:master Jul 29, 2019
@mlmhl mlmhl mentioned this pull request Jul 30, 2019
Closed
jsafrane pushed a commit to jsafrane/external-resizer that referenced this pull request Jan 8, 2020
@openshift-merge-robot
Merge pull request kubernetes-csi#6 from jsafrane/fix-windows
c80a2ba 
UPSTREAM: 60: Build Windows only for amd64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gnufied gnufied gnufied left review comments

@saad-ali saad-ali Awaiting requested review from saad-ali

Assignees

@gnufied gnufied

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mlmhl @gnufied @fejta-bot @k8s-ci-robot