[NET-5619] ci: Single source of truth for Go version in CI and Dockerfile

[zalimeni]

Description

To simplify Go version management and enable simpler conflict-less (CE + Ent) upgrades in the future, move all Go version management for CI, Docker builds, and related tooling to a single source of truth.

To do this, introduce a new .go-version file and reusable workflow to read it that all jobs using setup-go can depend on. Having a separate reusable job allows multiple dependent jobs to use the version after it's determined once. It also enables us to centralize control over the version in our own CI config rather than depending on the behavior of setup-go or other imported actions (see more below).

Make all reusable workflows require a Go version as input s.t. we can ensure it is always provided.

Rationale

We have two needs for our CI when it comes to Go versions:

• Using the most current version of Go that we support to take advantage of security patches and bugfixes. This should apply to our produced binaries and Docker containers at minimum; it should also apply to most tests to keep in sync w/ binaries and local dev.
• Proving backwards compatibility w/ our indicated minimum-required version; this primarily impacts consumers of our submodules and mitigates the risk of backwards-incompatible changes to repository code (e.g. using a new std lib function). This is accomplished by version-specific tests such as this for our published submodules.

When we upgrade to Go 1.21, we'll have the new toolchain directive available in addition to the go directive. More on how these work here; the salient point is that Go now attempts to install the correct toolchain for us when we update the go or toolchain directive and the local version of Go does not satisfy those versions. It also means that the go directive provides an enforced minimum version, not the suggested one.

However, this presents some rough edges in CI:

• setup-go does not yet support toolchain detection (see also Tar errors on cache restore after toolchain installation actions/setup-go#424)
• We don't want to install Go twice in every job due to a mismatch between the version installed by setup-go and the one required locally; this sort of mistake would be easy to overlook
• In addition to setup-go, we rely on hashicorp/action-go-build to produce binaries, which has a required input for the Go version that is not managed by setup-go; managing this separately from setup-go runs the risk of error when changing Go versions

Given these constraints, and the clarity provided by setting essential versions explicitly to avoid confusion or side-effect mistakes, this change proposes enforcing a consistently installed Go version throughout our workflows, still allowing for explicit overrides where we need a different version. It's easy to reverse if we want to go back to individual go-version-file declarations.

Futures

My hope is that we can move to calling go version to retrieve the toolchain version provided in Go 1.21+, as well as introduce a versioned go.work in the near future (which will bring several benefits, including removing submodule replaces, explicitly setting Go version via a single toolchain directive). The current change allows us to control the version from a single source of truth in CI, which is beneficial regardless of Go-native toolchain management changes in the future.

Testing & Reproduction steps

• CI continues to pass.
• Overrides for backwards compatibility tests still result in the desired version: Go 1.19 api and sdk tests
• Checked for missed updates (explicit Go versions are backwards compatibility tests noted above):

rg -t yaml 'go-version-file:' -B1
test-integrations.yml
101-          # Do not explicitly set Go version here, as it should depend on what Nomad declares.
102:          go-version-file: 'go.mod'
--
184-          # Do not explicitly set Go version here, as it should depend on what Vault declares.
185:          go-version-file: 'go.mod'

rg -t yaml 'go-version: ' | grep -v 'needs.get-go-version' | grep -v 'inputs.go-version'
go-tests.yml:      go-version: "1.19"
go-tests.yml:      go-version: "1.20"
go-tests.yml:      go-version: "1.19"
go-tests.yml:      go-version: "1.20"
reusable-get-go-version.yml:      go-version: ${{ steps.get-go-version.outputs.go-version }}

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

[zalimeni]

Just to note, backports will probably end up being semi-manual for this change; I've labeled the PR to reflect the intended backports for now.

[wilkermichael]

This is a great quality of life change @zalimeni! This will make things much easier going forward. Thanks for the well thought out breakdown of the work and future considerations as well!

[wilkermichael]

👨‍🍳

[curtbushko]

Awesome! It is these sorts of 'little' changes that just chip away at making all things easier.

Thanks so much for taking this on!

[hc-github-team-consul-core]

@zalimeni, a backport is missing for this PR [20058] for versions [1.15,1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

[hc-github-team-consul-core]

@zalimeni, a backport is missing for this PR [20058] for versions [1.15,1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

[hc-github-team-consul-core]

@zalimeni, a backport is missing for this PR [20058] for versions [1.15,1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>