Bugfix/azure wait for disk detach #248
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
prashanth26
added
kind/bug
prashanth26
force-pushed
the
bugfix/azure-wait-for-disk-detach
branch
3 times, most recently
from
364ab2c to
6aa7754
Compare
- Disks are now detached before deletion on Azure - Drain pod maximum grace period is aligned with drain timeout - Azure now longer poweroffs/shutdown VM before deletion
prashanth26
force-pushed
the
bugfix/azure-wait-for-disk-detach
branch
from
6aa7754 to
77072d2
Compare
|
@hardikdr /needs-review |
hardikdr
reviewed
Apr 1, 2019
pkg/controller/machine.go
Outdated
| @@ -470,7 +468,7 @@ func (c *controller) machineDelete(machine *v1alpha1.Machine, driver driver.Driv | |||
| c.targetCoreClient, | |||
| timeOutDuration, // TODO: Will need to configure timeout | |||
| nodeName, | |||
| -1, | |||
| int(timeOutDuration.Seconds()), | |||
There was a problem hiding this comment.
I thought to cap the graceful termination of any pod by the drain timeout should help in overall drain option going through in the drain timeout. But i dunno if it works as expected.
I am hoping that if a pod is set to a graceful termination of 2Hours, and our drain timeout is 5mins we max would only give 5mins?
There was a problem hiding this comment.
I have reverted this change here - 7510f9c#diff-d8287fe74b5273163c9f6b6c635ad912R475.
| // There are disks attached hence need to detach them | ||
| vm.StorageProfile.DataDisks = &[]compute.DataDisk{} | ||
|
|
||
| _, errChan := vmClient.CreateOrUpdate(d.AzureMachineClass.Spec.ResourceGroup, machineID, vm, cancel) |
There was a problem hiding this comment.
Can you please explain how does it work?
Where are we making explicit detach calls to VM? , or does the empty-disk array instructs API to delete all disk?
There was a problem hiding this comment.
Yes hardik. VM with an update call with empty data disks is how detachment of data disks is done in Azure. They lack a proper SDK call for it. Refer here - Azure/azure-sdk-for-go#1638 (comment)
prashanth26
added
the
reviewed/ok-to-test
gardener-robot-ci-1
added
needs/ok-to-test
prashanth26
force-pushed
the
bugfix/azure-wait-for-disk-detach
branch
4 times, most recently
from
adca4ce to
85feb71
Compare
prashanth26
added
reviewed/ok-to-test
gardener-robot-ci-1
removed
the
reviewed/ok-to-test
gardener-robot-ci-1
added
the
needs/ok-to-test
- Every pod termination now tries to be evicted for drain timeout period - If it fails to be evicted, it is deleted forcefully by setting the graceful period to 0s leading to a forceful deletion
- Drain is now invoked even in the case of forceful deletion (and) - After drain timeout duration of deletion call
prashanth26
force-pushed
the
bugfix/azure-wait-for-disk-detach
branch
from
85feb71 to
7510f9c
Compare
prashanth26
added
reviewed/ok-to-test
gardener-robot-ci-1
added
needs/ok-to-test
hardikdr
approved these changes
Apr 15, 2019
ghost
added
component/mcm
gardener-robot
added
the
area/ops-productivity
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR attempts to fix the Azure issue of VMs stuck in deletion state.
Now pods are tried to be evicted in the timeout period, and if it fails the pods are forcefully deleted by setting the grace period to 0. The data disks are then detached and the VM is deleted.
Which issue(s) this PR fixes:
Fixes #242
Special notes for your reviewer:
Release note: