Provider produced inconsistent result after apply

[Hoobie]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

We're not able to update the versions, atm.

Terraform v0.12.24

Providers:

• azurerm 2.28
• time 0.6.0
  ...

Affected Resource(s)

• azurerm ?

Terraform Configuration Files

We're forbidden to share it.

Debug Output

Panic Output

Expected Behaviour

Successful keyvault & Kubernetes cluster creation.

Actual Behaviour

�[0m�[1mmodule.key-vault.azurerm_key_vault.keyvault: Creation complete after 2m7s [id=/subscriptions/XXXXXXX-XXXXX-XXXX/resourceGroups/XXXXXXX-XXXXX-XXXX/providers/Microsoft.KeyVault/vaults/XXXXXXX-XXXXX-XXXX]�[0m�[0m
�[0m�[1mazurerm_key_vault_key.aks_disk_encryption_key: Creating...�[0m�[0m
�[1m�[31mError: �[0m�[0m�[1mProvider produced inconsistent result after apply�[0m

�[0mWhen applying changes to azurerm_key_vault_key.aks_disk_encryption_key,
provider "registry.terraform.io/-/azurerm" produced an unexpected new value
for was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.
�[0m�[0m
Terraform returned non-zero exit code - exiting

It happens to 10-20% of our deployments, so we have to restart.

Steps to Reproduce

terraform apply to create a Kubernetes cluster

Important Factoids

My initial suspicion was that we're using the time provider to build the keyvault_name:

keyvault_suffix = tostring(time_static.keyvault_time.unix)

May it be somehow related? In accordance with the docs: the timestamp is "stored in the Terraform state" and "This prevents perpetual differences", so it shouldn't cause the issue.

What is more, the key creation uses the vault id instead of its name:

resource "azurerm_key_vault_key" "aks_disk_encryption_key" {
  name         = local.cluster_name
  key_vault_id = module.key-vault.id

References

https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/static

[Hoobie]

The debug log is:

2021-01-19T08:38:34.918-0500 [DEBUG] plugin.terraform-provider-azurerm_v2.28.0_x5: [DEBUG] Unable to determine the Resource ID for the Key Vault at URL "https://XXXXX.vault.azure.net/" - removing from state!


2021/01/19 08:38:34 [DEBUG] azurerm_key_vault_key.aks_disk_encryption_key: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to azurerm_key_vault_key.aks_disk_encryption_key, provider "registry.terraform.io/-/azurerm" produced an unexpected new value for was present, but now absent.

[Hoobie]

Also, the issue persists in the version 2.40 and 2.43 of the provider.

[Hoobie]

Our workaround that seems to work is:

resource "time_sleep" "wait_30_seconds" {
  depends_on = [module.key-vault]

  create_duration = "30s"
}

resource "azurerm_key_vault_key" "encryption_key" {
  ...

  depends_on = [
    time_sleep.wait_30_seconds,
    ...
  ]
}

[dkirrane]

I hit this issue too with latest 2.44 version of the provider.
#10227

Don't think timeout will work in my case. KeyVault is created well before I try add a secret.

[tombuildsstuff]

hi @Hoobie

Thanks for opening this issue - apologies for the delayed response.

Taking a look through here this sounds like an issue in Terraform 0.12.x where module dependencies weren't inferred correctly - which has been fixed in a newer version of Terraform Core, I believe 0.14.x - would you be able to upgrade and see if that works for you?

Since this should be fixed by updating the version of Terraform Core in use, I'm going to close this issue for the moment - but if that doesn't work then please let us know, although it's worth noting that unfortunately we'll need a Terraform Configuration to diagnose this further.

Thanks!

[bbreckenridge]

I am hitting this problem on terraform 0.14.5. Going to try the timeout workaround.

[Hoobie]

@bbreckenridge I bumped the sleep to 60 seconds and we still hit this issue from time to time 😞 It just seems to be less frequent.

@tombuildsstuff we won't be able to upgrade anytime soon but it looks like it happens regardless of the Terraform version. I'm not allowed to send the configuration. What about other similar issues like: #10227 ?

[tombuildsstuff]

@Hoobie #10227 is a related issue due to recent caching changes - however in your case this appears to be the module dependency/ordering issue which whilst gives a similar error message for this resource is a slightly different issue (which has been fixed in TF Core, I believe 0.14 and won't be backported to 0.12). It's worth noting that Terraform Core 0.15 is coming out in the near future, so whilst Terraform 0.12 may continue to work for the moment, I'd strongly suggest staying up to date if possible.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!