Unable to destroy VM protected items

[ranokarno]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Hi,
recently I am unable to delete VM protected items. I suspect this could be due to Azure backend change.
from Azure portal it mentioned VM is deleted in deferred delete but Terraform expect Not Found

Terraform (and AzureRM Provider) Version

Terraform v0.11.11
AzureRm 1.33.1

Affected Resource(s)

• azurerm_recovery_services_protected_vm

Terraform Configuration Files

resource "azurerm_recovery_services_protected_vm" "vm1" {
  resource_group_name = "${azurerm_resource_group.example.name}"
  recovery_vault_name = "${azurerm_recovery_services_vault.example.name}"
  source_vm_id        = "${azurerm_virtual_machine.example.id}"
  backup_policy_id    = "${azurerm_recovery_services_protection_policy_vm.example.id}"
}

Expected Behavior

azurerm_recovery_services_protected_vm.vm1: destroy completed.

Actual Behavior

azurerm_recovery_services_protected_vm.vm1: Still destroying... (ID: <vm_resource_id>, 30m0s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* azurerm_recovery_services_protected_vm.vm1 (destroy): 1 error(s) occurred:

* azurerm_recovery_services_protected_vm.vm1: Error waiting for the Recovery Service Protected VM "VM;iaasvmcontainerv2;vm_resource_group;test_vm1" to be false (Resource Group "recovery_vault_resource_group") to provision: timeout while waiting for state to become 'NotFound' (last state: 'Found', timeout: 30m0s)

Steps to Reproduce

1. terraform apply
2. terraform destroy

[Dev-Mivl]

Hi Rano, did you find a workaround for this solution? My plan wants to destroy these before applying the rest of the plan.

[marcpascual]

unfortunately as stated in MicrosoftDocs/azure-docs#39329, there are no plans to revert the change or make it a toggle, similar to AKV.

[ranokarno]

@Dev-Mivl ,
workaround is to manually remove Terraform State related with azurerm_protected_item resource address.
AzureRm provider may need to change the return code for this resource.

in my case, i will run terraform state rm <resource_address>
you can refer resource addressing for more info, or run terraform state list

[ivanov-aleksander]

The problem is that your Azure Backup Vault has configured Soft delete. As a resul the backup data is retained for 14 additional days, allowing the recovery of that backup item with no data loss. These additional 14 days retention of backup data in the "soft delete" state don’t incur any cost to the customer.

[katbyte]

@ranokarno, @ivanov-aleksander,

Currently there is no way to turn off soft delete, and despite the API "soft-deleting" the VM it returns an error. This has been raised with the Azure service team and they have talked about potential fixes.. however i don't have date for when they will make the them.

[ThoHah]

As there is a manual way (.1 deactivating soft state for the vault, 2.deleting backup data, 3. destroying the vault) this should be possible via API too.

[tombuildsstuff]

@ThoHah the API for disabling Soft Delete has only recently become available (in the last few weeks), so whilst this is likely now possible - unfortunately we've not gotten to it yet; there's an issue tracking support for this however :)

[dpbustos]

Is it possible to add an option to retain backup data on destroy of the azurerm_recovery_services_protected_vm resource? (This would be in the scenario where soft-delete has been disabled in the vault... Via the Portal, when you select Stop Backup you get the option to delete or retain backup data)

[jcanongfi]

This issue concern azurerm 1.xx and azurerm_recovery_services_protected_vm.
But the is still present on azurerm 2.xx and azurerm_backup_protected_vm.
I look at the code and the principe is in 2 steps :

• Sending the DELETE request on azure API.
• Loop waiting for state switching from "Found" to "NotFound".
  (File : terraform-provider-azurerm/azurerm/internal/services/recoveryservices/resource_arm_backup_protected_vm.go: 203)
  Because a lot of us have configured a soft delete policy, it could be interesting that the loop (step2) beeing optional.
  Is it possible to add an optional resource property (boolean type) for getting the "state loop" optional ?

[perlboy]

@jcanongfi Yes it's still there. Destroying a azurerm_backup_protected_vm resource with soft delete enabled results in endless loop (I guess for 14 days). What would be ideal is if azurerm_backup_protected_vm destroy could consider soft delete as "destroyed". I have no problems with soft delete existing but would prefer to avoid direct state manipulation to work around it hanging on destroy (even after manual deletion).

[jmloeffler]

It feels like here where we set up the target state and expect "NotFound" we should be able to pass multiple states in and allow for "NotFound" or "SoftDeleted" or whatever status they're returning. In my mind, if I tell terraform to destroy an item but it sits in soft-deleted state for 14 days, it's as good as gone (no expense, can be recreated).

It might require an overload of the state.WaitForState() method to allow multiple states to be terminal states, but it feels like a good extension since we're likely to encounter this kind of thing on a variety of resources. I'm no go expert but I could give it a try if the idea is suitable to others.

[pierreyvesv]

One point I have notice is while terraform apply returns "Still destroying", an e-mail is sent by Azure with the following content:

"A delete Backup data alert has been activated
You’re receiving this email because a delete Backup data alert has been activated for server1.
Your data for this Backup item has been deleted. This data will be temporarily available for 14 days, after which it will be permanently deleted."

so the backup is deleted, but the status seems not returned to terraform

[Pwd9000-ML]

It would be nice to have an option on the "azurerm_backup_protected_vm" when it is added to say: ignore_soft_delete true/false
So when a VM deployment is destroyed it just stops the backup and sets it to delete (with soft delete on) and continue the removal of the VM.

Or even having an option that says: in_state true/false
That would give the option to deploy the resource but not add it to the state in the first place.
So when the resource is removed it just removes the VM. I know this option would not stop the VM backup or delete it so the backup would fail. But at least that can be manually stopped and removed if an alert is triggered about failed backup.
Difficult situation. Because that is essentially the same outcome by removing the resource from the state manually. It doesn't actually stop a VM backup.

Something like "ignore_soft_delete true/false" would be ideal as it would be nice to when destroying a deployment that option allow the backup to be stopped and marked for deletion if "soft delete" was enabled on the RSV and allow a resource to be destroyed unhindered.

[pranavrajtyagi-ab]

Have we started looking into this yet? Currently removing state manually works but I don't think that should be the case as it hinders automation.

[lovelinuxalot]

The problem with the function that is deleting the backup is checking for the state of backup. Instead it should be looking first, if the backup has soft delete enabled. If soft delete is enabled, mark the backup for delete and ignore the change. Because if the backup is removed from state and somehow we dont want to delete it and we mark it manually as not to delete, then terraform deletes it anyway on the next run.

I would do it this way,

1. Check if the backup item has soft delete enabled
2. If not, delete it. Current code works with it
3. if soft delete enabled, mark the backup for delete.
4. return that the backup is scheduled for delete
5. Ignore the change on state file

Let me know if someone thinks this is good :)

[sjohnsonsf]

Any plans to pick this up? I have a rather large implementation and like soft to keep soft delete but have state reflect accurately when destroying with enabled. Agree those above that soft delete should = delete as it pertains to tfstate.

[src386]

Does someone knows a workaround ? Right now, I guess we have to:

• Terraform apply/destroy
• CTRL+C during execution
• terraform state rm 'azurerm_backup_protected_vm.item'

Does not seems clean at all, even impossible in CI environments.

[oWretch]

@ranokarno, @ivanov-aleksander,

Currently there is no way to turn off soft delete, and despite the API "soft-deleting" the VM it returns an error. This has been raised with the Azure service team and they have talked about potential fixes.. however i don't have date for when they will make the them.

@katbyte Have you heard anything more from the Azure service team on potential fixes?

[tenletters10]

Sadly the workaround I have had to introduce to get around this problem is to manage the backups outside of Terraform. Not something I wanted to do, but I need my pipeline to manage my deployments of infrastructure in Azure and this one off rm of the resource object is a pretty poor workaround for scaling.

Please add support for the soft delete API responses for the destroy of this resource type.

[mpjtaylor]

Update, seems our issue is around the Soft Delete feature on the vault being enabled which causes the error:

Error: waiting for the Azure Backup Protected VM to be false. context deadline exceeded

[sethbacon]

At a minimum, I think items like this with a long history of issues should be flagged in the documentation. I was just testing some terraform to add the backup protected vm item and discovered the issue. There aren't any notes in the documentation and I had to come hunting in the issues section of the github page. This is completely backwards from a customer service perspective. If there are known issues, they should really be reflected in the docs so there is at least an opportunity to make a decision on using terraform to manage an item or not BEFORE I spin something up that I now have to perform state surgery on. This is extremely frustrating; especially in the current cybersecurity environment where we are all running around making sure our backups are managed better, are immutable, are tested, and we have full control on recovery. PLEASE do a better job on this in future.

[myc2h6o]

Currently there is no way to turn off soft delete, and despite the API "soft-deleting" the VM it returns an error. This has been raised with the Azure service team and they have talked about potential fixes.. however i don't have date for when they will make the them.

The soft-deleting seems to be supported by service now and added to the provider by #5586. I've done a test with soft_delete_enabled set to false in azurerm_recovery_services_vault and the backup vm can be cleaned up properly.
Although it's not recommended in the document, it seems to be the only option to delete the backup item correctly.

I think it could cause more trouble to change the behavior of azurerm_recovery_services_protected_vm to consider it as successfully deleted when a backup vm is in soft-delete status because this would cause the deletion of azurerm_recovery_services_vault to fail unexpectedly.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.