Unexpected modification of Certificate Content during azurerm_key_vault_certificate Update-In-Place

[slideroh]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.6.1

AzureRM Provider Version

3.75.0

Affected Resource(s)/Data Source(s)

azurerm_key_vault_certificate

Terraform Configuration Files

resource "azurerm_key_vault_certificate" "certs" {
  for_each = toset(var.certificates)

  key_vault_id = module.key_vault.id
  name = each.value
  certificate {
    contents = data.azurerm_key_vault_secret.cert[each.value].value
  }
  certificate_policy {
    key_properties {
      exportable = true
      key_size = data.azurerm_key_vault_certificate.cert[each.value].certificate_policy[0].key_properties[0].key_size
      key_type = data.azurerm_key_vault_certificate.cert[each.value].certificate_policy[0].key_properties[0].key_type
    }
    secret_properties {
      content_type = "application/x-pem-file"
    }
    lifetime_action {
      action {
        action_type = "EmailContacts"
      }
      trigger {
        lifetime_percentage = 80
      }
    }
  }
}

Debug Output/Panic Output

azurerm_key_vault_certificate.certs["test-cert"] will be updated in-place
  ~ resource "azurerm_key_vault_certificate" "cert" {
        id                              = "<>"
        name                            = "test-cert"
        tags                            = {}
        # (11 unchanged attributes hidden)

      ~ certificate_policy {
          ~ lifetime_action {
              ~ trigger {
                  ~ lifetime_percentage = 80 -> 85
                    # (1 unchanged attribute hidden)
                }
            }
        }
    }

Expected Behaviour

The update-in-place operation on the azurerm_key_vault_certificate resource for "test-cert" was expected to occur without modifying the actual content of the certificate.
Specifically, only the lifetime_action trigger (80->85) was to update as shown above.

Expected that changes would be limited to the lifetime_action trigger within the certificate policy.

Actual Behaviour

When performing an update-in-place on the azurerm_key_vault_certificate resource the unexpected behavior is observed.

Specifically, the content of the certificate undergoes changes that result in the removal of one certificate from the chain, and the private key is altered.

Consequently, after the update, the SSL certificate is rendered ineffective (without chain), leading to operational issues with the associated site.

Is the option to force recreate each time something change in this resource block?

Steps to Reproduce

1. az keyvault certificate download --name test-cert -f prechange-cert.pem
2. az keyvault secret download --name test-cert -f prechange-secret-cert.pem
3. terraform apply
4. az keyvault certificate download --name test-cert -f afterchange-cert.pem
5. az keyvault secret download --name test-cert -f afterchangesecret-cert.pem
6. Check content new cert, do md5sum, update cert on your site

Important Factoids

No response

References

No response

[slideroh]

This is the result of the md5sum:

• init-cert-secret - This is the secret of the certificate (with chain and private-key inside) - initial md5sum
• core-cert-secret - This secret is a copy of "init-cert-secret", done by data.azurerm_key_vault_secret.cert[each.value].value and is stored in another KeyVault. - the same md5sum
• core-cert-secret2 - The secret of the cert after update-in-place - different md5sum, different content, lack of 1 certificate.

^ Above is when you compare az keyvault secret download files, but the same is for az keyvault certificate. Only during creation the content is the same, after update-in-place the content is broken

[wuxu92]

@slideroh Thanks for filing this issue! However, I think this behavior is expected by pull request #20627 . This PR modifies the certificate_policy to allow updates and recreate a certificate when it is updated.

terraform-provider-azurerm/internal/services/keyvault/key_vault_certificate_resource.go

Lines 623 to 624 in f32f73c

	if d.HasChange("certificate_policy") {
	newCert, err := createCertificate(d, meta)

[slideroh]

Please keep in mind, that in azurerm_key_vault_certificate / data.azurerm_key_vault_certificate you will only see 1 certificate, however in azurerm_key_vault_secret with name of certificate (which is hidden) you can see chain + private_key.

After that "recreate a certificate" during change of certificate_policy I cannot see 2nd certificate (chain, in secret).

In that case, it creates the certificate incorrectly and skips the whole chain - as a result, after downloading the certificate secret - as I wrote there is only 1 certificate, instead of 2 (in my case)

Old one certs:

Common Name: test.cert
Issuer: Aero, Test Cert

and

Common Name: root.cert/intermediate.cert
Issuer: Root Aero, Test Cert

New one (after update-in-place):

Common Name: test.cert
Issuer: Aero, Test Cert

[slideroh]

@wuxu92
Script to generate test chain certs (obviously normal certificate is used on production).
The situation I am referring to has occurred several times and I wanted to check where the problem is. Hence the creation of this ticket.

# CA certificates
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 30 -out ca.pem -subj "/C=US/ST=New York/L=Manhattan/O=XYZ Corporation/OU=Intermediate-Root-CA/CN=RootCACorporateGateway"

# Core - Cert
openssl genrsa -out core.key 4096
openssl req -new -key core.key -out core.csr -subj "/C=US/ST=New York/L=Manhattan/O=XYZ Corporation/OU=Enduser/CN=CorporateGateway"
cat <<EOF > core.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
EOF
openssl x509 -req -in core.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out core.crt -days 30 -sha256 -extfile core.ext

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in core.key -out core.pkcs8.key
cat core.crt ca.pem core.pkcs8.key > core.pem

Result:

After update-in-place and download the cert (az keyvault secret download --name test-cert -f afterchangesecret-cert.pem)

Thanks

[wuxu92]

I apologize for not fully understanding what "it creates the certificate incorrectly and skips the whole chain" means. When recreating the certificate, a new version will be added and the provider will always retrieve the latest certificate. However, you can view all versions in the Portal as shown below:

As mentioned earlier, I agree that recreation should not be necessary when only updating the lifetime_action field. Unfortunately, the provider always creates a new version when certificate_policy is updated. I will attempt to update the logic to modify lifetime_action without recreation.

[slideroh]

@wuxu92

I will attempt to update the logic to modify lifetime_action without recreation.

Thank you for your understanding and for your help! I appreciate it very much

A new version will be added and the provider will always retrieve the latest certificate. However, you can view all versions in the Portal.

I'd like to emphasize that the issue we're facing is causing a significant problem for us. Specifically, our code relies on taking the last certificate, and with the current behavior, the last certificate is changed and missing the IntermediateCA, as illustrated in the examples provided earlier.

I believe it's a critical issue that needs attention and might generate outage, because of lifetime_action change, which is even not shown in Terraform Plan, that this will change the content (remove the CA)

Thank you again, if you need any help from our side to solve this issue (to test/generate more examples), please let us know too!

[jhitze]

We have run into a similar issue. We updated the certificate policy, and terraform plan spits out:

Terraform says that it is modifying the certificate, but it's really creating a new one. This caused an outage for us because we were not expecting the certificate contents to change.

Using

      source  = "hashicorp/azurerm"
      version = "~> 3.75.0"
 }

On terraform version 1.5.6

[slideroh]

Hi @wuxu92, are there any updates on this issue/bug?

[wuxu92]

@slideroh submitted a PR as above.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.