dynamic circleci config for streamlining test execution

[raulk]

This PR introduces dynamic CircleCI config generation to speed up our CI. Concretely, it generates scoped integration and unit test jobs, all of which run in parallel.

An integration test job is generated for every file in itest. We use the go test <file> invocation form, which has the limitation that itests cannot reference other symbols in the same package (they can of course reference symbols from other packages, like itests/kit), but that's a fair tradeoff because itests are supposed to be self-contained anyway.

Unit test jobs are generated as per configuration. I tried to respect the old grouping of suites. All other top-level directories that are NOT grouped under a suite get run under the rest suite.

I explored using CircleCI dynamic configuration, which enables you to generate the CircleCI config on their CI, and then re-submit it for execution. I found this introduced a head-of-line latency of around 1min, which I didn't like. So I removed it in favour of a make target.

Therefore, developers are responsible for regenerating the CircleCI config on their workstations, using make circleci. They should do this every time they add a new itest, or a new top-level package with unit tests.

---

⚠️ TODO

• In the future, we may want to add a check that runs the make target on CI, and fails the workflow if the diff is non-empty, similar to what we do with go mod tidy.

---

To parallelise things further, all you need to do is break up itest files containing many/slow tests into more itest files. This is for integration tests. For unit tests, you can create new groups.

---

I've removed the dependency between the build jobs and the test-short job. Didn't really make much sense.

[magik6k]

This is great, just 1 comment

[magik6k]

Therefore, developers are responsible for regenerating the CircleCI config on their workstations, using make circleci. They should do this every time they add a new itest, or a new top-level package with unit tests.

The nice thing is that make gen runs go generate ./..., and the gen-check CI step will fail when one won't do that

[BigLep]

@raulk : this is awesome.

In the future, we may want to add a check that runs the make target on CI, and fails the workflow if the diff is non-empty, similar to what we do with go mod tidy.

If I'm understanding this right, I think we need to do this now. I don't think we should check anything into the codebase the relies on "best intentions" to get right.

[raulk]

@BigLep agree with your reasoning. Based on what @magik6k said above, this is already taken into account by running go generate ./..., and then checking for diffs and failing the CI task if diffs are present. I just pushed a commit to check if it works as intended, because I suspect that go generate ./... might ignore hidden directories (starting with a dot, like the .circleci dir is).

[raulk]

Confirmed, it doesn't work as expected. I'll modify the gen-check CI job to specifically go generate the hidden directory too, as it's not picked up by the ./... expression.

[raulk]

Bingo! 0c31676 (#6561) did the trick.

@BigLep @magik6k the TODOs and review comments are all addressed.

[raulk]

@magik6k work here is done.