Explicit provider installation source configuration

[apparentlymart]

Until now, the set of places where Terraform's automatic provider installer (part of terraform init) has looked for available providers was generated based on a set of rules inside Terraform CLI, not directly customizable by the end-user.

The goal of this PR is to allow the user to optionally override those implied defaults in the CLI config, to allow for explicit control of source locations in special situations such as when Terraform is running in automation.

For example, the following configuration achieves the same effect as setting -plugin-dir=/tmp/example on the terraform init command line, where Terraform will only search in the given directory:

provider_installation {
  filesystem_mirror {
    path = "/tmp/example"
  }
}

The following configuration disables all of the local directory search locations and forces all providers to be installed solely from their origin provider registries:

provider_installation {
  direct {}
}

If the user knows that in-house providers are not actually available at an origin provider registry and can be installed only from a local directory, they might say something like this:

provider_installation {
  filesystem_mirror {
    include = ["tf.example.com/awesomecorp/*"]
  }
  direct {
    exclude = ["tf.example.com/awesomecorp/*"]
  }
}

This also includes some syntax for specifying network-based mirrors (on HTTP servers), but the actual implementation of that will follow in a later PR.

---

This configuration structure of having a sequence of blocks of different types where the total order is significant is not something HCL 1's decoder can represent, so unlike the rest of the cliconfig package this part is implemented directly using HCL 1's AST API where we can have more control over how we interpret the physica configuration structure.

That has the nice side-effect of allowing us to replicate some of the strictness we'd normally expect from the HCL 2 decoder, which I did here mainly to ease the later migration of the CLI config to HCL 2 at least for this new part of the CLI configuration. (The rest remains rather lax, as before.)

---

This includes an initial draft of documentation for this new feature of the CLI configuration. This is mainly intended as a placeholder for now, because there are other documentation updates pending for the new provider namespacing and installation scheme and we'll likely want to revise these docs to better complement the broader documentation once it's written.

---

While working on this I noticed that the TF_CLI_CONFIG_FILE environment variable we implemented a while back to allow the user to temporarily override the CLI config search location was not disabling the CLI configuration directory, and so it was ending up making a strange blend of overridden and standard options.

There's a commit in this set that fixes that bug, so that TF_CLI_CONFIG_FILE completely disables all of the normal CLI configuration file search locations. That makes TF_CLI_CONFIG_FILE more useful as a way to isolate wrapper script behavior from ambient CLI configuration, but it is a breaking change we'll need to note in the changelog once we merge this.

[codecov]

Codecov Report

Merging #24728 into master will decrease coverage by 0.03%.
The diff coverage is 45.93%.

Impacted Files	Coverage Δ
internal/getproviders/http_mirror_source.go	0.00% <0.00%> (ø)
provider_source.go	22.44% <4.54%> (-39.32%)	⬇️
main.go	38.62% <9.09%> (-2.16%)	⬇️
command/cliconfig/cliconfig.go	49.39% <41.66%> (+1.39%)	⬆️
command/cliconfig/provider_installation.go	69.23% <69.23%> (ø)
dag/marshal.go	52.27% <0.00%> (-1.14%)	⬇️

[mildwonkey]

Does this need a log level? "[INFO]" (or whatever you deem appropriate) at the start of the string?

[apparentlymart]

It seems like some other log lines in this area don't tend to have it, but I suppose I don't really know why that is, so I will add levels to all the new ones I've added around here.

[mildwonkey]

I started to write a comment that I'd like to see more detail here for future readers (ie, me) but found it in the function instead so 🎉 thank you!

[mildwonkey]

👏 SUCH GOOD DOCUMENTATION thank you

[mildwonkey]

I think this comment can be updated now.

[mildwonkey]

tiny nitpick (and totally ok if you don't agree): it's more specific and accurate to say this features allows installing/using terraform providers in these situations

[apparentlymart]

Fair! I think I was assuming that Terraform is useless if you can't install providers, but no need to be vague about it.

[mildwonkey]

This looks great! I left a couple of small comments, nothing that needs to block you from merging this.

I have two larger questions about this new feature that are out of scope for this PR, but I want to write them down somewhere before I forget:

1. Do you think we should check for (relatively) obvious conflicts in the provider_installation block? Something like the example below, where the include and exclude attributes are the same:

  network_mirror {
    url     = "https://tf-Mirror.example.com/"
    include = ["registry.terraform.io/*/*"]
    exclude = ["registry.Terraform.io/*/*"]
  }

I can imagine that there are too many possible permutations to make checking this for the user onerous - at the very least we can wait and see what kind of edge cases and issues pop up.

2. We've talked about the provider developer experience and a few different ideas for simplifying the process. What do you think about adding a specific attribute (exclusive to filesystem_mirrors) to let provider developers point directly to a binary, forgoing the need to create a directory structure?

I'm making this up as I go and have zero attachment to the wording or syntax, but something that conveys my intent:

  filesystem_mirror {
    include = ["tf.example.com/awesomecorp/*"]

    # this is my awkward attempt to only use attributes instead of blocks
    exact = ["{provider_source}:{/path/to/binary}"]

    # for example:  
    exact = ["example.com/myorg/happcloud":"$HOME/go/bin"]
  }

We don't have to discuss these questions here!

[apparentlymart]

@mildwonkey and I discussed the follow-on questions above in an offline discussion today. I just wanted to capture the high-level conclusions here for posterity:

1. We will not have any explicit checks for overlapping/conflicting declarations at first because the wildcard support makes that tricky to do robustly and we want to see what sorts of problems tend to arise in practice before spending the time to research and implement such checks.

2. We will be having a follow-up discussion with provider developers about a few different options for how to approximate the old provider development workflow while giving Terraform the metadata it needs to understand how to use a locally-built provider. A CLI configuration option for this is one of the ideas on the table, but it's not clear that this would be the most ergonomic answer because it could make it easy to accidentally leave temporary development configuration in your main CLI config and then be confused later when terraform init installs something other than what it would do by default. We'll review a few different options and see which one seems best to implement for the 0.13 release.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.