azurerm_private_endpoint fails intermittently with retriable error

[sebader]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

3.0.2

Affected Resource(s)

• azurerm_private_endpoint

Terraform Configuration Files

resource "azurerm_private_endpoint" "acr" {
  name                = "${local.prefix}-${local.location_short}-acr-pe"
  location            = azurerm_resource_group.stamp.location
  resource_group_name = azurerm_resource_group.stamp.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_dns_zone_group {
    name                 = "privatednsacr"
    private_dns_zone_ids = [azurerm_private_dns_zone.acr.id]
  }

  private_service_connection {
    name                           = "acr-privateserviceconnection"
    private_connection_resource_id = data.azurerm_container_registry.global.id
    is_manual_connection           = false
    subresource_names              = ["registry"]
  }

  tags = var.default_tags
}

Expected Behaviour

Terraform should automatically retry retryable errors and not fail.

Actual Behaviour

Sometimes during provisioning of a private endpoint, we have seen the following error. Looking into the Azure portal, the Private Endpoint indeed exists and is working. However, we cannot just run terraform apply again, since it does not exist in state. We need to manually delete the PE first (or could manually import it).

│ Error: waiting for creation of Private Endpoint "acintbf3e-westeurop-acr-pe" (Resource Group "acintbf3e-stamp-westeurope-rg"): Code="RetryableError" Message="A retryable error occurred." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/***/resourceGroups/ao-network/providers/Microsoft.Network/virtualNetworks/vnet-we/subnets/private-endpoints-snet used by resource /subscriptions/***/resourceGroups/acintbf3e-stamp-westeurope-rg/providers/Microsoft.Network/networkInterfaces/acintbf3e-westeurop-acr-pe.nic.8b3a01bb-91bd-4a50-b197-04119f1cb398 is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]

Steps to Reproduce

Not sure, it happens intermittently.

Important Factoids

We are provisioning multiple private endpoints in parallel into the same subnet, not sure if that is causing the issue.

[catalinpopa247]

Hi - is there any update on the topic? We're running into the same issue on our end. Is there any ETA when that retry function will be implemented?

[catalinpopa247]

Update: we managed to implement a workaround for this. It seems like the dependency mechanism which should be handled by Terraform automatically isn't working as expected.
Our issue occured when we were coding several actions at the subnet level. Our terraform file was doing a peering between VNETs and creating private endpoints.
Seems like Terraform doesn't map dependencies between these two actions, so we had to manually specify that in our TF. We made sure that first the privateEndpoints are deployed, afterwards doing the peering. This was achieved by manually specifying a depends_on component on the peering block to wait for the privateEndpoints.

So if you encounter this issue, look for actions specified in your TF code that might execute at the same time. That's the issue.

[sebader]

yeah thats how I'm also working around it for now:
I am creating about 7 different Private Endpoints and added a depends_on to each of them to depend on the previous one. So they all deploy in order. But of course this takes a lot longer and is a hassle to maintain

[rpearceukcdw]

Hi, is there any update/roadmap for this bug? I'm receiving the error using azurerm 3.10.0. I don't have the ability to run sequentially using dependsOn easily within our project, so have resorted to apply -parallelism=1 for now hurting performance.

[geekzter]

I found that adding this to azurerm_private_endpoint is needed, even when provisioning multiple in sequence using depends_on:

provisioner local-exec {
  command = "az resource wait --updated --ids ${self.subnet_id}"
}

This assumes you're sharing Azure CLI credentials.

[rpearceukcdw]

Thank you very much for this, I'll give it a try.

[github-actions]

This functionality has been released in v3.19.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.