0.12upgrade cannot parse heredoc when final EOF is inline

[mildwonkey]

Terraform 0.11 allowed for this syntax (though the documentation specifically requires the terminating EOF to be on a new line):

locals {
   cert_options = <<EOF
--cert-file=/etc/ssl/etcd/server.crt \
  --peer-trusted-ca-file=/etc/ssl/etcd/ca.crt \
  --peer-client-cert-auth=trueEOF
}

The 0.12upgrade tool does not care for this:

Error: Syntax error in configuration file

  on main.tf line 1, in locals:
   1: locals {

Error while parsing: At 25:1: heredoc not terminated

terraform init also dislikes this syntax:

$ ./terraform init
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Unterminated template string

  on main.tf line 11, in locals:
   3: --cert-file=/etc/ssl/etcd/server.crt \
   4:   --peer-trusted-ca-file=/etc/ssl/etcd/ca.crt \
   5:   --peer-client-cert-auth=trueEOF
   6: }
   8: output "local" {
   9:   value = "${local.cert_options}"
  10: }

(capturing slack context)
It was suggested that this was allowed due to a bug in HCL that was fixed - 0.11 vendors an older version of HCL, 0.12 may need to be rolled back to the same version: hashicorp/hcl@13daa63

[apparentlymart]

I believe hashicorp/hcl#245 is the change that altered the behavior here. I'm hoping if we roll back as @mildwonkey suggested then the upgrade tool should be bug-compatible with 0.11's HCL parsing due to using the exact same parser.

[apparentlymart]

This was fixed as part of #20281.

[lanoxx]

I upgraded to Terrafrom 0.12 and now terraform reports that all my AWS instance's user_data hashsums have changed. As a result Terraform wants to recreate all AWS instances.

The hashsum change is caused by missing whitespace in 0.12 and originates from a different interpretation of the final EOF/EOT token.

I have the following user data in my terraform template:

    user_data = <<EOF
    #cloud-config
    repo_update: all
    packages:
       - python
    EOF

Please pay attention to the four spaces before the final EOF terminator and note that there is no terminating line break in the last line, since the EOF token immediately follows the whitespace.

The user data as it currently appears in the instance (created with Terraform 0.11) looks like this:

    #cloud-config
    repo_update: all
    packages:
      - python
    

Again, observe that there are four spaces of whitespace in the last line.

This results in the sha1 hashsum of fc84d80c42e4694ecd197507b6920e4c6faadc00. Note, if you try to reproduce the hashsum from the above snippet then you must pay attention to make sure it has no trailing line break. If you save the above snippet in a text file (e.g. user_data.txt), then it will have a trailing line break in the last line. To remove that line break use the head command to cut it off like: head -c 72 user_data.txt

Now, with Terraform 0.12, I get a different hashsum: d85ecd4480905ed693e2425dbf66f254d2f3770e, which results from the following user_data:

    #cloud-config
    repo_update: all
    packages:
      - python

In the last line the four spaces are missing.

Could someone from the Terraform team clarify, if it is possible to achieve the 0.11 behaviour in 0.12? Is it possible to get trailing whitespace in the result, if the final EOF/EOT token has whitespace in front of it. Should #20281 have restored that as well or am I seeing a different issue?

[ProTip]

@lanoxx I'm still seeing the behavior outlined in the issue as well on 0.12.0 and 0.12.1.

@apparentlymart I'm still seeing Unterminated template string; No closing marker was found for the string. with inline terminator while upgrading Rundeck. Is this issue perhaps still present in the latest Terraform? I've worked around for now by using %q instead of HEREDOC.

[lanoxx]

As a workaround, you can move the data into a templatefile(). Then you have better control on how the data is formatted.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.