Consider releasing the our test folder as a separate go module

[vincepri]

There is a couple of PRs that introduce a dependency on docker (#4499 and #4413) which would become a dependency in the main Cluster API go.mod.

While these are good improvements, it'd be great to completely separate everything under testing into a different go module and release the module separately from the main module.

These are the changes that would need to happen:

• Run go mod init under test/ folder.
• Remove the go module from test/infrastructure/docker.
• Move the test/helpers into util/test folder (this is primarily envtest, shouldn't need to be pushed separately).
• Update the release guide to make sure every time a new tag is cut, one is also cut for test. For example, if we're releasing v0.4.20, we should also tag test/v0.4.20.
• Update the v1alpha3-to-v1alpha4 provider implementers guide to make sure that folks import the testing framework from the new module.

/kind cleanup
/kind api-change
/priority important-soon

cc @stmcginnis @CecileRobertMichon @elmiko @fabriziopandini @yastij @sedefsavas

If this change is accepted, we should mark it as a potential release blocker.

[fabriziopandini]

I'm +100 to move envtest env into util/test (and finally drop the test/helpers folder)

I'm also ok with the go mod reorg; this has the nice side effect that kind will be removed from top level CAPI dependencies

WRT to introduce docker as a dependency, this is an interesting change, and hopefully it will help us in having better error messages in case of problems (vs shelling out). However, In my mind the primary goal of this effort still remains to make CAPD unit testable, so we can eventually reshuffle priorities to introduce the interface required for decoupling docker call first, and change how do we make docker calls only after this. This would allow us some more time before taking this decision.

[yastij]

+1 from me

[elmiko]

+1, preventing some of those top level dependencies seems really worthwhile.

[stmcginnis]

Would it make sense to separate the docker engine interaction from the rest of the test code? And keep CAPD separate too?

The changes being proposed for not exec'ing docker calls impacts test infra and CAPD. It might be nice to keep these independent. Then if someone wants to use CAPD outside of our testing, things can be limited to just the parts needed. Vice versa, for a user of the test infrastructure running against a different provider, the unused CAPD bits can be kept separate.

If we would want to keep CAPD separate, then it would make sense to move the new docker interfacing code to its own module so it can be picked up as a dependency separately by test infra and CAPD.