azurerm_resource_group delete fails because of lag in resource deletion

[sebader]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

azurerm 3.0.2

Affected Resource(s)

• azurerm_resource_group

Expected Behaviour

deletion should work

Actual Behaviour

Related to
#16069
#16073

We have a couple of Private Endpoints in our resource group, all managed by Terraform. Terraform destroys the private endpoints but the NICs seem to take a bit longer to be deleted.

This should have helped but I guess the waiting period needs to be further extended

Error: deleting Resource Group "ace2e122e-stamp-swedencentral-rg": the Resource Group still contains Resources.
│ 
│ Terraform is configured to check for Resources within the Resource Group when deleting the Resource Group - and
│ raise an error if nested Resources still exist to avoid unintentionally deleting these Resources.
│ 
│ Terraform has detected that the following Resources still exist within the Resource Group:
│ 
│ * `/subscriptions/***/resourceGroups/ace2e122e-stamp-swedencentral-rg/providers/Microsoft.Network/networkInterfaces/ace2e122e-swedencen-acr-pe.nic.d11c52fb-0d23-420c-a325-e77f4fbe85bd`
│ * `/subscriptions/***/resourceGroups/ace2e122e-stamp-swedencentral-rg/providers/Microsoft.Network/networkInterfaces/ace2e122e-swedencen-evhns-pe.nic.eee7cd94-1af4-416e-99f5-a91a21ff31ad`
│ * `/subscriptions/***/resourceGroups/ace2e122e-stamp-swedencentral-rg/providers/Microsoft.Network/networkInterfaces/ace2e122e-swedencen-storage-blob-pe.nic.fbd60f8d-d419-4c1d-8947-d8cd7213d19f`
│ * `/subscriptions/***/resourceGroups/ace2e122e-stamp-swedencentral-rg/providers/Microsoft.Network/networkInterfaces/ace2e122e-swedencen-storage-table-pe.nic.0957bbb5-2dea-4c30-b5ae-f8d8e9e6abb1`
│ 
│ This feature is intended to avoid the unintentional destruction of nested Resources provisioned through some
│ other means (for example, an ARM Template Deployment) - as such you must either remove these Resources, or
│ disable this behaviour using the feature flag `prevent_deletion_if_contains_resources` within the `features`
│ block when configuring the Provider, for example:
│ 
│ provider "azurerm" {
│   features {
│     resource_group {
│       prevent_deletion_if_contains_resources = false
│     }
│   }
│ }

Steps to Reproduce

1. terraform apply

Important Factoids

References

@mbfrahry

[magodo]

@sebader Thank you for submitting this! Sorry that the issue is still happening with the retry logic. We've been reaching to the Azure team about this and will update here if we got any feedback. Before that, you can temporarily opt out the prevent_deletion_if_contains_resources feature in the provider block as stated in the error message, especially when you are sure you want to remove everything inside the resource group.

[LaurentLesle]

@magodo I have have been able to reproduce. I think there is a negative cache keeping the previous resource for at least 30sec. So when the resource group is getting destroyed, the api still respond saying the previous resource is in the resource group.

When using prevent_deletion_if_contains_resources = true why not adding a small wait / retry to let the cache to expire and then only fails if there is still some resources after the retry logic?

in my config:

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = true
    }
  }
}

Note: my goal is indeed to prevent deletion if the resource group still contains resources. However due to this cache on the API layer, the job fails. In the above example I would have expected the job to succeed.

error log:

module.example.module.diagnostic_event_hub_namespaces["event_hub_namespace1"].azurerm_eventhub_namespace.evh: Still destroying... [id=/subscriptions/xxx-71e9-4dae-92b3-...espaces/ormj-ehn-securityoperationlogs, 40s elapsed]
module.example.module.diagnostic_event_hub_namespaces["event_hub_namespace1"].azurerm_eventhub_namespace.evh: Destruction complete after 41s
module.example.module.resource_groups["rg1"].azurerm_resource_group.rg: Destroying... [id=/subscriptions/***/resourceGroups/ormj-rg-example-sqldb]

Error: deleting Resource Group "ormj-rg-example-sqldb": the Resource Group still contains Resources.

Terraform has detected that the following Resources still exist within the Resource Group:

* `/subscriptions/***/resourceGroups/ormj-rg-example-sqldb/providers/Microsoft.EventHub/namespaces/ormj-ehn-securityoperationlogs`

This feature is intended to avoid the unintentional destruction of nested Resources provisioned through some
other means (for example, an ARM Template Deployment) - as such you must either remove these Resources, or
disable this behaviour using the feature flag `prevent_deletion_if_contains_resources` within the `features`
block when configuring the Provider, for example:

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

[magodo]

@LaurentLesle We do have 10min to wait for the cache to be invalidated:

terraform-provider-azurerm/internal/services/resource/resource_group_resource.go

Line 134 in d70e6ea

err = pluginsdk.Retry(10*time.Minute, func() *pluginsdk.RetryError {

.

[LaurentLesle]

@magodo -> I have that issue on 2.99.0. I can see the retry has been included in 3.4.0. Will re-test with latest version