diff --git a/docs/README.md b/docs/README.md index 360cb41eafbc..d70ae2d0fde9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -39,6 +39,11 @@ to [the main Terraform CLI documentation](https://www.terraform.io/docs/cli/inde * [Terraform Core RPC API](../internal/rpcapi/README.md): an integration point for external software that needs to integrate Terraform Core functionality. +* [Upgrading Terraform's Dependencies](./dependency-upgrades.md): guidance on + some special details that arise when we upgrade Go Module dependencies, due + to this codebase containing Terraform CLI, Terraform Core, and the various + remote state backends which all have some overlapping dependencies. + * [How Terraform Uses Unicode](./unicode.md): an overview of the various features of Terraform that rely on Unicode and how to change those features to adopt new versions of Unicode. diff --git a/docs/dependency-upgrades.md b/docs/dependency-upgrades.md new file mode 100644 index 000000000000..226f5eefeb1b --- /dev/null +++ b/docs/dependency-upgrades.md @@ -0,0 +1,153 @@ +# Upgrading Terraform's Library Dependencies + +This codebase depends on a variety of external Go modules. At the time of +writing, a Terraform CLI build includes the union of all dependencies required +by Terraform CLI and Core itself and each of the remote state backends. + +Because the remote state backends have quite a different set of needs than +Terraform itself -- special client libraries, in particular -- we've declared +them as being separate Go Modules with their own `go.mod` and `go.sum` files, +even though we don't intend to ever publish them separately. The goal is just +to help track which dependencies are used by each of these components, and to +more easily determine which of the components are affected by a particular +dependency upgrade so that we can make sure to do the appropriate testing. + +## Dependency Synchronization + +Because all of these components ultimately link into the same executable, there +can be only one version of each distinct module and thus all of the modules +must agree on which version to use. + +The makefile target `syncdeps` runs a script which synchronizes all of the +modules to declare compatible dependencies, selecting the newest version of +each external module selected across all of the internal modules: + +```shell +make syncdeps +``` + +After running this, use `git status` to see what's been changed. If you've +changed the dependencies of any of the modules then that should typically +cause an update to the root module, because that one imports all of the others. + +## Upgrading a Dependency + +To select a newer version of one of the dependencies, use `go get` in the +root to specify which version to use: + +```shell +go get example.com/foo/bar@v1.0.0 +``` + +Or, if you just want to move to the latest stable release, you can use the +`latest` pseudo-version: + +```shell +go get example.com/foo/bar@latest +``` + +Then run `make syncdeps` to update any of the child modules that also use +this dependency. The remote state backends use only a subset of the packages +in Terraform CLI/Core, so not all dependency updates will affect the remote +state backends, and an update might affect only a subset of the backends. + +When you open the pull request for your change, our code owners rules will +automatically request review from the team that maintains any affected remote +state backend. The affected teams can judge whether the update seems likely +to affect their backend and run their acceptance tests if so, before approving +the pull request. As usual, these PRs should also be reviewed by at least +one member of the Terraform Core team since they are ultimately responsible +for the complete set of dependencies used in Terraform CLI releases. + +**Note:** Currently our code owners rules are simplistic and will request +review for _any_ change under a remote state backend module directory, but +in practice an update that only changes a backend's `go.sum` cannot affect +the runtime behavior of the backend, and so those review requests are not +strictly required. You should therefore remove the review requests for +any backend whose only diff is the `go.sum` file once you've opened the +pull request. + +## Dependabot Updates + +When Dependabot automatically opens a pull request to upgrade a dependency, +unfortunately it isn't smart enough to automatically synchronize the change +across the modules and so the code consistency checks for the change will +typically fail. + +To apply the proposed change, you'll need to check out the branch that +Dependabot created on your development system, run `make syncdeps`, add +all of the files that get modified, and then amend dependabot's commit using +`git commit --amend`. + +After you've done this, use `git push --force` to replace dependabot's original +commit with your new commit, and then wait for GitHub to re-run the PR +checks. The code consistency checks should now pass. + +We've configured Dependabot to monitor only the root `go.mod` file for potential +upgrades, because that one aggregates the dependencies for all other child +modules. Therefore there should never be a dependabot upgrade targeting a +module in a subdirectory. If one _does_ get created somehow, you should close +it and perform the same upgrade at the root of the repository instead, using +the instructions in [Upgrading a Dependency](#upgrading-a-dependency) above. + +## Dependencies with Special Requirements + +Most of our dependencies can be treated generically, but a few have some +special constraints due to how Terraform uses them: + +* HCL, cty, and their indirect dependencies `golang.org/x/text` and + `github.com/apparentlymart/go-textseg` all include logic based on Unicode + specifications, and so should be updated with care to make sure that + Terraform's support for Unicode follows a consistent Unicode version + throughout. + + Additionally, each time we adopt a new minor release of Go, we may need to + upgrade some or all of these dependencies to match the Unicode version used + by the Go standard library. + + For more information, refer to [How Terraform Uses Unicode](unicode.md). + + (This concern does not apply if the new version we're upgrading to is built + for the same version of Unicode that Terraform was already using.) + +* `github.com/hashicorp/go-getter` represents a significant part of Terraform + CLI's remote module installer, and is the final interpreter of Terraform's + module source address syntax. Because the module source address syntax is + protected by the Terraform v1.x Compatibility Promises, for each upgrade + we must make sure that: + + - The upgrade doesn't expand the source address syntax in a way that is + undesirable from a Terraform product standpoint or in a way that we would + not feel comfortable supporting indefinitely under the compatibility + promises. + - The upgrade doesn't break any already-supported source address forms + that would therefore cause the next Terraform version to break the + v1.x compatibility promises. + + Terraform's use of `go-getter` is all encapulated in `internal/getmodules` + and is set up to try to minimize the possibility that a go-getter upgrade + would immediately introduce new functionality, but that encapsulation cannot + prevent adoption of changes made to pre-existing functionality that + Terraform already exposes. + +* `github.com/hashicorp/go-tfe` -- the client library for the Terraform Cloud + API -- includes various types corresponding to Terraform Cloud API + requests and responses. The internal package `internal/cloud` contains mock + implementations of some of those types, which may need to be updated when + the client library is upgraded. + + These upgrades should typically be done only in conjunction with a project + that will somehow use the new features through the Cloud integration, so + that the team working on that project can perform any needed updates to + the mocks as part of their work. + +* `go.opentelemetry.io/otel` and the other supporting OpenTelemetry modules + should typically be upgraded together in lockstep, because some of the + modules define interfaces that other modules implement, and strange behavior + can emerge if one is upgraded without the other. + + The main modules affected by this rule are the ones under the + `go.opentelemetry.io/otel` prefix. The "contrib" packages can be trickier + to upgrade because they tend to have dependencies that overlap with ours + and so might affect non-telemetry-related behavior, and so it's acceptable + for those to lag slightly behind to reduce risk in routine upgrades.