Empty merge result with empty input after destroy

[peterpramb]

Terraform Version

Terraform v0.13.2
+ provider registry.terraform.io/hashicorp/tls v2.2.0

Terraform Configuration Files

locals {
  tls_keylist1 = {
    "test1-1" = { algorithm = "ECDSA" }
  }
  tls_keylist2 = {
    "test2-1" = { algorithm = "ECDSA" }
  }
}

resource "tls_private_key" "tls_keys1" {
  for_each = local.tls_keylist1

  algorithm = each.value.algorithm
}

resource "tls_private_key" "tls_keys2" {
  for_each = local.tls_keylist2

  algorithm = each.value.algorithm
}

output "tls_keys" {
  value = merge(tls_private_key.tls_keys1, tls_private_key.tls_keys2)
}

Expected Behavior

merge will return all remaining objects, when one of the inputs to merge gets empty.

Actual Behavior

merge result is empty, when one of the inputs to merge gets empty.

Steps to Reproduce

1. terraform init
2. terraform apply
3. Empty one of the local maps:

locals {
  tls_keylist1 = {
#   "test1-1" = { algorithm = "ECDSA" }
  }
  tls_keylist2 = {
    "test2-1" = { algorithm = "ECDSA" }
  }
}

4. terraform apply
5. terraform apply

Additional Context

With the initial apply, everything works as expected:

Terraform will perform the following actions:

  # tls_private_key.tls_keys1["test1-1"] will be created
  + resource "tls_private_key" "tls_keys1" {
      + algorithm                  = "ECDSA"
      + ecdsa_curve                = "P224"
      + id                         = (known after apply)
      + private_key_pem            = (sensitive value)
      + public_key_fingerprint_md5 = (known after apply)
      + public_key_openssh         = (known after apply)
      + public_key_pem             = (known after apply)
      + rsa_bits                   = 2048
    }

  # tls_private_key.tls_keys2["test2-1"] will be created
  + resource "tls_private_key" "tls_keys2" {
      + algorithm                  = "ECDSA"
      + ecdsa_curve                = "P224"
      + id                         = (known after apply)
      + private_key_pem            = (sensitive value)
      + public_key_fingerprint_md5 = (known after apply)
      + public_key_openssh         = (known after apply)
      + public_key_pem             = (known after apply)
      + rsa_bits                   = 2048
    }

Plan: 2 to add, 0 to change, 0 to destroy.

tls_private_key.tls_keys1["test1-1"]: Creating...
tls_private_key.tls_keys2["test2-1"]: Creating...
tls_private_key.tls_keys2["test2-1"]: Creation complete after 0s [id=09bdf230615fda93e7acc5d2dce16b2963f156b2]
tls_private_key.tls_keys1["test1-1"]: Creation complete after 0s [id=e6c1bfd162e419c300f87313d528ebf917db8896]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

tls_keys = {
  "test1-1" = {
    "algorithm" = "ECDSA"
    "ecdsa_curve" = "P224"
    "id" = "e6c1bfd162e419c300f87313d528ebf917db8896"
    "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHDmoj0hG1d5vJ/LvfaaLSXig0pwxlYLCzdEqnb2gBwYFK4EEACGhPAM6\nAAQke9vNeguf8Rr1Qve0BmgkMnQDFjfoqBkCvWSL5Vv6mC9nxom7xeQF8OcucVIO\n0jLn1ND9eg8Ozg==\n-----END EC PRIVATE KEY-----\n"
    "public_key_fingerprint_md5" = ""
    "public_key_openssh" = ""
    "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEJHvbzXoLn/Ea9UL3tAZoJDJ0AxY36KgZ\nAr1ki+Vb+pgvZ8aJu8XkBfDnLnFSDtIy59TQ/XoPDs4=\n-----END PUBLIC KEY-----\n"
    "rsa_bits" = 2048
  }
  "test2-1" = {
    "algorithm" = "ECDSA"
    "ecdsa_curve" = "P224"
    "id" = "09bdf230615fda93e7acc5d2dce16b2963f156b2"
    "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHAzq+mlr0E1BxxUfgWR3JzyaDMOwZtcobPWY5CigBwYFK4EEACGhPAM6\nAAR7dyfyWGZGLyupIcivfhrg0FJ9xpErYcIy+rS0wknVy9MJNM++y8eSFjD9cDx9\nciTOpHEFFMvNpQ==\n-----END EC PRIVATE KEY-----\n"
    "public_key_fingerprint_md5" = ""
    "public_key_openssh" = ""
    "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEe3cn8lhmRi8rqSHIr34a4NBSfcaRK2HC\nMvq0tMJJ1cvTCTTPvsvHkhYw/XA8fXIkzqRxBRTLzaU=\n-----END PUBLIC KEY-----\n"
    "rsa_bits" = 2048
  }
}

Then, when one of the local maps is emptied, it is expected that the removed objects get destroyed and the remaining objects are not touched. Up to and including the destroy of the removed objects this works as expected, but afterwards the merge output is unexpectedly empty:

Terraform will perform the following actions:

  # tls_private_key.tls_keys1["test1-1"] will be destroyed
  - resource "tls_private_key" "tls_keys1" {
      - algorithm       = "ECDSA" -> null
      - ecdsa_curve     = "P224" -> null
      - id              = "e6c1bfd162e419c300f87313d528ebf917db8896" -> null
      - private_key_pem = (sensitive value)
      - public_key_pem  = <<~EOT
            -----BEGIN PUBLIC KEY-----
            ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEJHvbzXoLn/Ea9UL3tAZoJDJ0AxY36KgZ
            Ar1ki+Vb+pgvZ8aJu8XkBfDnLnFSDtIy59TQ/XoPDs4=
            -----END PUBLIC KEY-----
        EOT -> null
      - rsa_bits        = 2048 -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  ~ tls_keys = {
      - test1-1 = {
          - algorithm                  = "ECDSA"
          - ecdsa_curve                = "P224"
          - id                         = "e6c1bfd162e419c300f87313d528ebf917db8896"
          - private_key_pem            = <<~EOT
                -----BEGIN EC PRIVATE KEY-----
                MGgCAQEEHDmoj0hG1d5vJ/LvfaaLSXig0pwxlYLCzdEqnb2gBwYFK4EEACGhPAM6
                AAQke9vNeguf8Rr1Qve0BmgkMnQDFjfoqBkCvWSL5Vv6mC9nxom7xeQF8OcucVIO
                0jLn1ND9eg8Ozg==
                -----END EC PRIVATE KEY-----
            EOT
          - public_key_fingerprint_md5 = ""
          - public_key_openssh         = ""
          - public_key_pem             = <<~EOT
                -----BEGIN PUBLIC KEY-----
                ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEJHvbzXoLn/Ea9UL3tAZoJDJ0AxY36KgZ
                Ar1ki+Vb+pgvZ8aJu8XkBfDnLnFSDtIy59TQ/XoPDs4=
                -----END PUBLIC KEY-----
            EOT
          - rsa_bits                   = 2048
        } -> null
        test2-1 = {
            algorithm                  = "ECDSA"
            ecdsa_curve                = "P224"
            id                         = "09bdf230615fda93e7acc5d2dce16b2963f156b2"
            private_key_pem            = <<~EOT
                -----BEGIN EC PRIVATE KEY-----
                MGgCAQEEHAzq+mlr0E1BxxUfgWR3JzyaDMOwZtcobPWY5CigBwYFK4EEACGhPAM6
                AAR7dyfyWGZGLyupIcivfhrg0FJ9xpErYcIy+rS0wknVy9MJNM++y8eSFjD9cDx9
                ciTOpHEFFMvNpQ==
                -----END EC PRIVATE KEY-----
            EOT
            public_key_fingerprint_md5 = ""
            public_key_openssh         = ""
            public_key_pem             = <<~EOT
                -----BEGIN PUBLIC KEY-----
                ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEe3cn8lhmRi8rqSHIr34a4NBSfcaRK2HC
                Mvq0tMJJ1cvTCTTPvsvHkhYw/XA8fXIkzqRxBRTLzaU=
                -----END PUBLIC KEY-----
            EOT
            rsa_bits                   = 2048
        }
    }

tls_private_key.tls_keys1["test1-1"]: Destroying... [id=e6c1bfd162e419c300f87313d528ebf917db8896]
tls_private_key.tls_keys1["test1-1"]: Destruction complete after 0s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Please note the missing outputs. The state is now empty too:

{
  "version": 4,
  "terraform_version": "0.13.2",
  ...
  "outputs": {},
  ...
}

Just with the next apply the missing objects appear again:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + tls_keys = {
      + test2-1 = {
          + algorithm                  = "ECDSA"
          + ecdsa_curve                = "P224"
          + id                         = "09bdf230615fda93e7acc5d2dce16b2963f156b2"
          + private_key_pem            = <<~EOT
                -----BEGIN EC PRIVATE KEY-----
                MGgCAQEEHAzq+mlr0E1BxxUfgWR3JzyaDMOwZtcobPWY5CigBwYFK4EEACGhPAM6
                AAR7dyfyWGZGLyupIcivfhrg0FJ9xpErYcIy+rS0wknVy9MJNM++y8eSFjD9cDx9
                ciTOpHEFFMvNpQ==
                -----END EC PRIVATE KEY-----
            EOT
          + public_key_fingerprint_md5 = ""
          + public_key_openssh         = ""
          + public_key_pem             = <<~EOT
                -----BEGIN PUBLIC KEY-----
                ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEe3cn8lhmRi8rqSHIr34a4NBSfcaRK2HC
                Mvq0tMJJ1cvTCTTPvsvHkhYw/XA8fXIkzqRxBRTLzaU=
                -----END PUBLIC KEY-----
            EOT
          + rsa_bits                   = 2048
        }
    }

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

tls_keys = {
  "test2-1" = {
    "algorithm" = "ECDSA"
    "ecdsa_curve" = "P224"
    "id" = "09bdf230615fda93e7acc5d2dce16b2963f156b2"
    "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHAzq+mlr0E1BxxUfgWR3JzyaDMOwZtcobPWY5CigBwYFK4EEACGhPAM6\nAAR7dyfyWGZGLyupIcivfhrg0FJ9xpErYcIy+rS0wknVy9MJNM++y8eSFjD9cDx9\nciTOpHEFFMvNpQ==\n-----END EC PRIVATE KEY-----\n"
    "public_key_fingerprint_md5" = ""
    "public_key_openssh" = ""
    "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEe3cn8lhmRi8rqSHIr34a4NBSfcaRK2HC\nMvq0tMJJ1cvTCTTPvsvHkhYw/XA8fXIkzqRxBRTLzaU=\n-----END PUBLIC KEY-----\n"
    "rsa_bits" = 2048
  }
}

Additional info:

The provided code above is for simple reproduction. In the real code merge is used on a cloud resource, like shown:

resource "hcloud_ssh_key" "ssh_keys" {
  for_each = merge(
    { for name, data in tls_private_key.ssh_keys : name => data.public_key_openssh },
    { for name, data in data.local_file.ssh_public_keys : name => data.content }
  )

  name       = each.key
  public_key = each.value
}

There, when one of the merge inputs get empty due to destroyed objects, Terraform fails (after the actual destroy) with the following error:

Error: each.value cannot be used in this context

  on main.tf line 87, in resource "hcloud_ssh_key" "ssh_keys":
  87:   public_key = each.value

A reference to "each.value" has been used in a context in which it
unavailable, such as when the configuration no longer contains the value in
its "for_each" expression. Remove this reference to each.value in your
configuration to work around this error.

Interestingly, this error never occurs with create_before_destroy = true.

References

• https://discuss.hashicorp.com/t/empty-merge-after-partial-destroy/13815

[alisdair]

@peterpramb Thank you for this excellent reproduction case!

I was able to confirm this issue on Terraform 0.13.2. Happily, it seems to be fixed as of Terraform 0.13.3, I believe due to #26264. As a result, I'm marking this issue as closed, but if the bug in your real configuration still exists, please do comment here or open another issue so that we can investigate further.

[peterpramb]

Thanks, I can confirm that this is not reproducible anymore on 0.13.3 (both use cases).

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.