Updating azurerm_key_vault_certificate fails when soft delete is enabled for Key Vault

[FreyMo]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.7
azurerm v2.48.0

Affected Resource(s)

• azurerm_key_vault_certificate (resource)

Terraform Configuration Files

terraform {
    required_providers {
        azurerm = {
            version = "2.48.0"
        }
        azuread = {
            version = "1.3.0"
        }
    }
}

provider "azurerm" {
  features {
    key_vault {
        purge_soft_delete_on_destroy = false
        recover_soft_deleted_key_vaults = true
    }
  }
}

locals {
    location = "westeurope"
}

output "hash" {
    description = "The MD5 file hash to prove that the file content has changed"
    value = filemd5("example.pfx")
}

data "azuread_client_config" "this" {}

resource "random_id" "this" {
  byte_length = 8
}

resource "azurerm_resource_group" "this" {
    name = "rg-${random_id.this.hex}"
    location = local.location
}

resource "azurerm_key_vault" "this" {
    name = "kv-${random_id.this.hex}"
    location = local.location
    resource_group_name = azurerm_resource_group.this.name
    tenant_id = data.azuread_client_config.this.tenant_id
    sku_name = "standard"
    purge_protection_enabled = true
}

resource "azurerm_key_vault_access_policy" "this" {
  key_vault_id = azurerm_key_vault.this.id
  tenant_id    = data.azuread_client_config.this.tenant_id
  object_id    = data.azuread_client_config.this.object_id

  certificate_permissions = [
    "create",
    "delete",
    "deleteissuers",
    "get",
    "getissuers",
    "import",
    "list",
    "listissuers",
    "managecontacts",
    "manageissuers",
    "setissuers",
    "update",
  ]

  secret_permissions = [
    "backup",
    "delete",
    "get",
    "list",
    "recover",
    "restore",
    "set",
  ]
}

resource "azurerm_key_vault_certificate" "this" {
    name = "my-certificate"
    key_vault_id = azurerm_key_vault.this.id

    depends_on = [
        azurerm_key_vault_access_policy.this
    ]

    certificate_policy {
        issuer_parameters {
          name = "Self"
        }

        key_properties {
            exportable = false
            key_size = 4096
            key_type = "RSA"
            reuse_key = false
        }

        secret_properties {
          content_type = "application/x-pkcs12"
        }
    }

    certificate {
      contents = filebase64("example.pfx")
      password = ""
    }
}

I have also created a sample repository that can be used to reproduce the issue.

Debug Output

Error: keyvault.BaseClient#ImportCertificate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="Conflict" Message="Certificate my-certificate is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged." InnerError={"code":"ObjectIsDeletedButRecoverable"}

  on main.tf line 81, in resource "azurerm_key_vault_certificate" "this":
  81: resource "azurerm_key_vault_certificate" "this" {

Expected Behaviour

Changing the certificate should upload a new version of the certificate. If the certificate is in a deleted state, it should be recovered so that a new version can be uploaded.

Actual Behaviour

The certificate gets left in a deleted state and does not get recovered. terraform apply fails because there is a conflict. azurerm_key_vault_secret is therefore unusable with a Key Vault with soft_delete_enabled.

Steps to Reproduce

1. terraform init
2. Provide a certificate named example.pfx
3. terraform apply
4. Exchange the certificate with a new example.pfx
5. terraform apply

Important Factoids

For secrets, this mechanism works fine. In fact, there is a workaround that recovers the secret before a new one gets created.

For certificates, the same workaround has been implemented but doesn't seem to work.

References

• key vault soft-delete causing failure when updating secrets or certificates  #5659 (Marked as closed in 2.41.0 but still not working for certificates)
• Support for KeyVault object versioning #6915 (Feature request instead of an issue but the same problem in the comments)

[jibinpb]

Used null_resource along with azurerm_key_vault_certificate as a temporary work around. The null_resource will update the certificate as new version via Az PowerShell and set to ignore changes in certificate resource so it won't mess up the state later.

[adamency]

@katbyte what is the status on this ? It's been reported for a year and a half that the keyvault certificate recover feature of the provider doesn't actually work which completely thwart the use of keyvault certificates with terraform if you don't have purge rights, which is the case for most users, while certificates are very much ubiquitous and at the center of online infrastructure.

@jibinpb Could you give some detail on your workaround solution ? I can't figure out how to keep the resource in the terraform declaration while allowing to destroy it and apply it again afterwards. Thanks in advance.

[jdelforno]

@adamency

Create the azurerm_key_vault_certificate as you would normally, ensure that:

  lifecycle {
    ignore_changes = [
      certificate
    ]
  }

Is on the resource.

Then export the certificate using

resource "local_file" "b64cert" {
  content_base64 = acme_certificate.primary.certificate_p12
  filename       = "${path.module}/yourCertificateName.pfx"
}

And then finally, use az cli to do the update against the keyvault;

resource "null_resource" "azCommandUpdateCertificate" {
  provisioner "local-exec" {
    command = "az keyvault certificate import --vault-name ${azurerm_key_vault.yourVault.name} --name ${azurerm_key_vault_certificate.yourCertificate.name} --file ${path.module}/yourCertificateName.pfx --password ${sensitive(azurerm_key_vault_secret.yourPassphrase .value)}"
  }
}

Do not omit the "sensitive" function or you'll output your pfx passphrase to logs.

[jdelforno]

@jibinpb how'd you go about omitting the --password parameter? I've got sensitive() on it but it looks like there's a bug, in that when it errors, it outputs the password to cli.

[mkocaks]

Thanks all!

[georgeflug]

I had to make one adjustment to @jdelforno 's workaround for it to work, otherwise it would try running the null_resource before the local_file existed:
Inside null_resource:

depends_on = [
  local_file.b64cert
]

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.