-
Notifications
You must be signed in to change notification settings - Fork 9.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explicit provider installation source configuration #24728
Conversation
This new CLI config block type allows explicitly specifying where Terraform should look to find provider plugins for installation. This is not used anywhere as of this commit, but in a future commit we'll change package main to treat the presence of a block of this type as a request to disable the default set of provider sources and use these explicitly- specified ones instead.
This is a placeholder for later implementation of a mirror source that talks to a particular remote HTTP server and expects it to implement the provider mirror protocol.
If the CLI configuration contains a provider_installation block then we'll use the source configuration it describes instead of the implied one we'd build otherwise.
Codecov Report
|
fc4dc7a
to
98fe60a
Compare
command/cliconfig/cliconfig.go
Outdated
} | ||
} else { | ||
log.Printf("Not reading CLI config directory because config location is overridden by environment variable") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this need a log level? "[INFO]"
(or whatever you deem appropriate) at the start of the string?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like some other log lines in this area don't tend to have it, but I suppose I don't really know why that is, so I will add levels to all the new ones I've added around here.
@@ -136,6 +153,13 @@ func loadConfigFile(path string) (*Config, tfdiags.Diagnostics) { | |||
return result, diags | |||
} | |||
|
|||
// Deal with the provider_installation block, which is not handled using | |||
// DecodeObject because its structure is not compatible with the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I started to write a comment that I'd like to see more detail here for future readers (ie, me) but found it in the function instead so 🎉 thank you!
// decoder would be in terms of exactly what configuration shape it is | ||
// expecting. | ||
// | ||
// Note that this function wants the top-level file object which might or |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👏 SUCH GOOD DOCUMENTATION thank you
main.go
Outdated
@@ -168,7 +169,22 @@ func wrappedMain() int { | |||
// direct from a registry. In future there should be a mechanism to | |||
// configure providers sources from the CLI config, which will then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this comment can be updated now.
may not be able to access an origin registry due to firewall restrictions | ||
within your organization or your locality. | ||
|
||
To allow using Terraform in these situations, there are some alternative |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tiny nitpick (and totally ok if you don't agree): it's more specific and accurate to say this features allows installing/using terraform providers in these situations
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair! I think I was assuming that Terraform is useless if you can't install providers, but no need to be vague about it.
This looks great! I left a couple of small comments, nothing that needs to block you from merging this. I have two larger questions about this new feature that are out of scope for this PR, but I want to write them down somewhere before I forget:
I can imagine that there are too many possible permutations to make checking this for the user onerous - at the very least we can wait and see what kind of edge cases and issues pop up.
I'm making this up as I go and have zero attachment to the wording or syntax, but something that conveys my intent:
We don't have to discuss these questions here! |
When we originally introduced this environment variable it was intended to solve for the use-case where a particular invocation of Terraform needs a different CLI configuration than usual, such as if Terraform is being run as part of an automated test suite or other sort of automated situation with different needs than normal use. However, we accidentally had it only override the original singleton CLI config file, while leaving the CLI configuration directory still enabled. Now we'll take the CLI configuration out of the equation too, so that only the single specified configuration file and any other environment-sourced settings will be included.
An earlier commit added a redundant stub for a new network mirror source that was already previously stubbed as HTTPMirrorSource. This commit removes the unnecessary extra stub and changes the CLI config handling to use it instead. Along the way this also switches to using a full base URL rather than just a hostname for the mirror, because using the usual "Terraform-native service discovery" protocol here doesn't isn't as useful as in the places we normally use it (the mirror mechanism is already serving as an indirection over the registry protocol) and using a direct base URL will make it easier to deploy an HTTP mirror under a path prefix on an existing static file server.
…ation In the first pass of implementing this it was strict about what arguments are allowed inside source blocks, but that was counter to our usual design principles for CLI config where we tend to ignore unrecognized things to allow for some limited kinds of future expansion without breaking compatibility with older versions of Terraform that will be sharing the same CLI configuration files with newer versions. However, I'd removed the tracking of that prior to the initial commit. I missed some leftover parts when doing that removal, so this cleans up the rest of it.
Unfortunately in the user model the noun "source" is already used for the argument in the required_providers block to specify which provider to use, so it's confusing to use the same noun to also refer to the method used to obtain that provider. In the hope of mitigating that confusion, here we use the noun "method", as in "installation method", to talk about the decision between getting a provider directly from its origin registry or getting it from some mirror. This is distinct from the provider's "source", which is the location where a provider _originates_ (prior to mirroring). This noun is also not super awesome, but better than overloading an existing term in the same feature.
The CLI config can be written in both native HCL and HCL JSON syntaxes, so the provider_installation block must be expressible using JSON too. Our previous checks to approximate HCL 2-level strictness were too strict for HCL JSON where things are more ambiguous even in HCL 2, so this includes some additional relaxations if we detect that we're decoding an AST produced from a JSON file. This is still subject to the quirky ways HCL 1 handles JSON though, so the JSON value must be structured in a way that doesn't trigger HCL's heuristics that try to guess what is a block and what is an attribute. (This is the issue that HCL 2 fixes by always decoding using a schema; there's more context on this in: https://log.martinatkins.me/2019/04/25/hcl-json/ )
This exercises the ability to customize the installation methods used by the provider plugin installer, in this case forcing the use of a custom local directory with a result essentially the same as what happens when you pass -plugin-dir to "terraform init".
Previously we were incorrectly using the Include configuration for both the include and exclude list, making the include portion totally ineffective.
…onfig This is an initial draft of documentation for this new feature of the CLI configuration. This is mainly intended as a placeholder for now, because there are other documentation updates pending for the new provider namespacing and installation scheme and we'll likely want to revise these docs to better complement the broader documentation once it's written.
These were being used in an earlier iteration of the provider installation configuration but it was all collapsed down into a single ProviderInstallationMethod type later, making these redundant.
da7ae0f
to
622abf7
Compare
@mildwonkey and I discussed the follow-on questions above in an offline discussion today. I just wanted to capture the high-level conclusions here for posterity:
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
Until now, the set of places where Terraform's automatic provider installer (part of
terraform init
) has looked for available providers was generated based on a set of rules inside Terraform CLI, not directly customizable by the end-user.The goal of this PR is to allow the user to optionally override those implied defaults in the CLI config, to allow for explicit control of source locations in special situations such as when Terraform is running in automation.
For example, the following configuration achieves the same effect as setting
-plugin-dir=/tmp/example
on theterraform init
command line, where Terraform will only search in the given directory:The following configuration disables all of the local directory search locations and forces all providers to be installed solely from their origin provider registries:
If the user knows that in-house providers are not actually available at an origin provider registry and can be installed only from a local directory, they might say something like this:
This also includes some syntax for specifying network-based mirrors (on HTTP servers), but the actual implementation of that will follow in a later PR.
This configuration structure of having a sequence of blocks of different types where the total order is significant is not something HCL 1's decoder can represent, so unlike the rest of the
cliconfig
package this part is implemented directly using HCL 1's AST API where we can have more control over how we interpret the physica configuration structure.That has the nice side-effect of allowing us to replicate some of the strictness we'd normally expect from the HCL 2 decoder, which I did here mainly to ease the later migration of the CLI config to HCL 2 at least for this new part of the CLI configuration. (The rest remains rather lax, as before.)
This includes an initial draft of documentation for this new feature of the CLI configuration. This is mainly intended as a placeholder for now, because there are other documentation updates pending for the new provider namespacing and installation scheme and we'll likely want to revise these docs to better complement the broader documentation once it's written.
While working on this I noticed that the
TF_CLI_CONFIG_FILE
environment variable we implemented a while back to allow the user to temporarily override the CLI config search location was not disabling the CLI configuration directory, and so it was ending up making a strange blend of overridden and standard options.There's a commit in this set that fixes that bug, so that
TF_CLI_CONFIG_FILE
completely disables all of the normal CLI configuration file search locations. That makesTF_CLI_CONFIG_FILE
more useful as a way to isolate wrapper script behavior from ambient CLI configuration, but it is a breaking change we'll need to note in the changelog once we merge this.