Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cancel RollingUpgrades before deleting VMSS resources #15634

Closed

Conversation

harshavmb
Copy link
Contributor

@harshavmb harshavmb commented Feb 28, 2022

RollingUpgrades when setup on VM ScaleSets cause lot of problems during destroy. This gets worse when extensions are run on these scalesets.

At the moment, we use destroy provisioners, to cancel rolling upgrades & then destroy resources such as extensions & VMSS.

This tiny boolean flag cancel_rolling_upgrades_before_deletion when set to true will cancel rolling upgrades if they are running and then move forward to delete the VMSS. This way, we avoid writing destroy provisioners.

In addition to this, the logic in helpers.go file, future.Response().StatusCode == http.StatusConflict && strings.Contains(err.Error(), "There is no ongoing Rolling Upgrade to cancel.") is to handle a scenario on a newly provisioned VMSS resource where rolling upgrades are set but no extensions, auto-scaling are run.

The error parsing logic is to handle the error thrown by Azure API when no Rolling Upgrades are ongoing.
Get-Latest API URL ::
https://management.azure.com/subscriptions/{subId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachineScaleSets/{vmSSName}/rollingUpgrades/latest?api-version=2021-07-01

In other cases, we cancel, wait until the cancel is complete & move ahead to delete the resource associated with VMSS.

Have compiled & tested locally, it works as expected.

Hope this helps!

Harshavardhan Musanalli and others added 16 commits February 22, 2022 21:06
@harshavmb harshavmb changed the title Cancel RollingUpgrades before deleting VMSS resources when set Cancel RollingUpgrades before deleting VMSS resources Feb 28, 2022
@harshavmb
Copy link
Contributor Author

RollingUpgrades when setup on VM ScaleSets cause lot of problems during destroy. This gets worse when extensions are run on these scalesets.

At the moment, we use destroy provisioners, to cancel rolling upgrades & then destroy resources such as extensions & VMSS.

This tiny boolean flag cancel_rolling_upgrades_before_deletion when set to true will cancel rolling upgrades if they are running and then move forward to delete the VMSS. This way, we avoid writing destroy provisioners.

In addition to this, the logic in helpers.go file, future.Response().StatusCode == http.StatusConflict && strings.Contains(err.Error(), "There is no ongoing Rolling Upgrade to cancel.") is to handle a scenario on a newly provisioned VMSS resource where rolling upgrades are set but no extensions, auto-scaling are run.

The error parsing logic is to handle the error thrown by Azure API when no Rolling Upgrades are ongoing. Get-Latest API URL :: https://management.azure.com/subscriptions/{subId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachineScaleSets/{vmSSName}/rollingUpgrades/latest?api-version=2021-07-01

In other cases, we cancel, wait until the cancel is complete & move ahead to delete the resource associated with VMSS.

Have compiled & tested locally, it works as expected.

Hope this helps!

Finally apologies for these commits originated from my main fork as there is another PR waiting for approval. Next time, I create separate branches for each PR so that commits look clean.

@tombuildsstuff tombuildsstuff self-assigned this Mar 1, 2022
@harshavmb
Copy link
Contributor Author

Hi @tombuildsstuff @katbyte @jackofallops ,

Could you please provider some update on this PR? Waiting in the review queue for quite sometime.

@ecampeggio

This comment was marked as off-topic.

@evaroquaux
Copy link

Hello,

This PR will fix a known and real limitation when working with Azure VM scale sets in Terraform. Meanwhile, it is blocking the development of our Terraform-based automation.

Is there any chance to have it considered? It was submitted a long time ago and there has not been any movement for a while.

@katbyte
Copy link
Collaborator

katbyte commented Oct 25, 2022

closing in favour of #18973

@katbyte katbyte closed this Oct 25, 2022
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants