Experimental "templatestring" function

[apparentlymart]

(This is motivated primarily by #30616, but since it's just an experiment for now merging this PR does not imply closing that issue.)

A long time ago #215 requested a way to parse and render a template from a separate file, and the Terraform of the day didn't have the necessary building blocks for that to be a function, and so I proposed the compromise that became the hashicorp/template provider and its template_file data source.

Later on it became clear that Terraform resource types that take filesystem paths as arguments are problematic because the path gets saved as part of the plan, and so it's hard to then apply the plan on a different computer. As part of a general pattern of replacing filename arguments with arguments that take contents of files, template_file changed from accepting filename to accepting template, with the intention that it would be used like this to get the original functionality of rendering a template from a file:

  template = file("${path.module}/template.txt")

This had the unintentional side-effect of making it possible to write arbitrary expressions that return strings that the provider could parse as templates, rather than only using the file function. People began to use this data source not for its intended purpose of rendering content from external files, but for other less-common situations like fetching a template from an S3 bucket and then rendering it.

Meanwhile, the template_file data source also grew to be a bit of an attractive nuisance, because those new to Terraform would search for ways to render templates in Terraform and would tend to turn up the template_file data source even though all they really needed was string template expressions, which are built in to the language. These new users would then get incredibly confused trying to write a string template that causes Terraform Core to generate a string template to parse to hashicorp/template which is then finally rendered. This was made extra confusing when folks inevitably tried to use it to render shell scripts from a template, since they then had three levels of nested interpolation to worry about, making it even more confusing than the typical two levels that arise when trying to render a bash script using a normal template.

Because of that experience, once it finally became possible to implement templatefile as a built-in function instead of as a provider with a data source, we switched it back to its original design of taking a filename directly and then reading the file from disk itself. That neatly dealt with the attractive nuisance because the function could no longer be accidentally misused by passing it an inline template. However, we then discovered the fact that some folks had used template_file for dynamic template generation/rendering instead of just for loading templates from disk as originally intended, and they were naturally displeased that the new solution did not support their use-case.

We've been at an impasse on this ever since, because our experience with template_file shows that a dynamic template rendering function is likely to be misunderstood by new authors as the way to render small templates (instead of using Terraform's built-in expression syntax) but a vocal minority of users had used dynamic template rendering in a load-bearing way and thus could not move away from the now-deprecated hashicorp/template provider.

---

This PR introduces a language experiment that hopes to break that impasse by making a compromise: the function imposes artificial extra constraints on its template argument to try to proactively dissuade people from using it in situations where an inline template would be more appropriate.

With this experiment enabled, it becomes valid to -- for example -- retrieve a template from S3 and dynamically render it:

data "aws_s3_object" "example" {
  bucket = "example"
  key    = "example"
}

output "example" {
  value = templatestring(data.aws_s3_object.example.body, {
    name = var.name
  })
}

However, unlike most other functions it isn't valid to set the first argument to anything other than a single reference to a string from elsewhere.

In particular, it will reject attempts to write an escaped template inline even if the resulting template string is valid:

  # not allowed
  value = templatestring("Hello, $${name}!", {
      name = var.name
  })

The goal of that restriction is to intentionally divert a new author from this mistake toward the better alternative of just writing the template expression directly:

  value = "Hello, ${var.name}"

This restriction does not actually block more questionable use of the function when it's intentional, because it's always possible to factor out the template literal into a local value and then refer to it:

locals {
  ill_advised_inline_template = "Hello, $${name}!"
}

output "example" {
  value = templatestring(local.ill_advised_inline_template, {
    name = var.name
  })
}

The goal here is not to stop people from doing this sort of thing if they are particularly determined to do it for any reason, and only to reduce the chances of people doing it accidentally because they haven't yet learned enough about Terraform to know what other options they have.

---

Since this is just an experiment it does not yet have any documentation. My intention with proposing this is to include it in at least one alpha release and ask those who requested dynamic template rendering to try it and see whether it supports their use-cases, as a replacement for the deprecated hashicorp/template provider.

If successful, I would expect then to write documentation for this function which emphasizes that this is a rather esoteric function that most authors won't need, and to link directly to the string templates documentation and to the templatefile function as more likely candidates for common use-cases.

My hope then would be that if someone finds out about this function through some means other than our documentation and tries to use it for an unintended purpose, they'll encounter an error message advising them against that usage and will hopefully then refer to our documentation and learn about another feature that better suits their goals.

This should then, hopefully 🤞 serve as a suitable compromise that allows the power users with unusual needs to finally migrate away from hashicorp/template but without recreating the learning hazard that the template_file data source became.

---

I had previously proposed #28700 as an attempt to narrowly solve a related use-case of allowing modules to take parsed-but-not-evaluated templates as arguments.

While meeting that use-case with strings containing template syntax is not ideal due to the need for the module user to write an inline escaped template, the situations where a template needs to be rendered somewhere other than where it's defined are also relatively rare and so I don't think it's justified to implement something as complicated as that earlier prototype for such a rare need. Therefore if we were to move forward with this I would expect to close that prototype PR and document how to solve that use-case as part of the templatestring function documentation, while also including a caution to module authors about the usabililty hazards of doing so.

[github-actions]

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.