Ensure we test beta releases of CAPI with some providers before release

[CecileRobertMichon]

Recently CAPI v1.4.0 was released. Two different regressions were found while testing with CAPZ (#8427 and #8462) for dual stack clusters and MachinePools, respectively. Unfortunately, the bugs were not found early enough to block the release, and an emergency v1.4.1 patch had to be cut to address the issues.

As discussed in this week's office hours, this issue tracks potential improvements we could make to ensure this doesn't happen again in the future.

Here are some ideas from the discussion:

Improvements to CAPI new release integration testing with infra providers

Preview Give feedback

1. Encourage providers to test the release beta early by adding a release task for the release team to open issues to test the beta in some of the major provider repos as soon as the beta release is available
2. Add continuous integration tests with some of the infra providers using the nightly build of CAPI

[CecileRobertMichon]

/cc @killianmuldoon @fabriziopandini @ykakarap

[CecileRobertMichon]

cc @kubernetes-sigs/cluster-api-release-team

[fabriziopandini]

/triage accepted

Happy to revive the work we did in the past to publish nightly images/manifests for providers to use

[joekr]

What do you all think about me adding the "Encourage providers to test the release beta early by adding a release task for the release team to open issues" to the Comm's team task list, or possibly the Release Lead's task list, as part of the Beta.0 release? We can add it to the timeline as well.

[sbueringer]

Happy to revive the work we did in the past to publish nightly images/manifests for providers to use

I think the only gap is that we don't run the nightly job on release branches (I think we still do on main). Before we invest in extending this though, it would be good to know if providers would use nightly release from release branches. I think we dropped it a while back because nobody was using those releases.

Somewhat related:

Document how to consume nightly Cluster API releases with clusterctl and with the test framework
#8418

[sbueringer]

Add continuous integration tests with some of the infra providers using the nightly build of CAPI

I assume this is something which would be done as part of the provider CI (vs core CAPI)? Just like we are currently testing upstream Kubernetes latest in CAPI & providers?

[CecileRobertMichon]

I assume this is something which would be done as part of the provider CI (vs core CAPI)? Just like we are currently testing upstream Kubernetes latest in CAPI & providers?

yes exactly

The only thing is this may be difficult in practice if there are breaking changes in CAPI, for example the test framework, because then we wouldn't be able to test latest CAPI without updating the provider.

[fabriziopandini]

Should we try to convert this discussion into actionable items so we can get ready for when the beta release is cut?
Might be it could help to approach this with a practical approach, by cutting an alpha tag and trying to set up the first version of those tests in providers?

WRT to possible blockers, like breaking change in CAPI, I think we should design for that otherwise we will miss this signal when major changes in CAPI happen, e.g. when there is a contract change or a new API version. This makes things a little bit more complex, because it could imply testing in a branch + rebase, but this can also be a follow-up iteration given that for this release AFAIK we are not planning breaking changes

[CecileRobertMichon]

Should we try to convert this discussion into actionable items so we can get ready for when the beta release is cut?

This issue already has two concrete tasks, one of them is already complete (#8552), let me know what else you think we should add.

The second one covers the continuous integration tests using a nightly build that we've been describing

[fabriziopandini]

/kind feature
/priority important-soon

[sbueringer]

I think one way to address the second part: "Add continuous integration tests with some of the infra providers using the nightly build of CAPI" would be to implement e.g. the Kubernetes conformance test for some providers (that are then using CAPI "main" images). But I think this has to be maintained on the infra provider side.

We could surface the results by having a core CAPI release-informing dashboard in testgrid to which we add these jobs.

[fabriziopandini]

We recently talked about this in office hours again, and a few providers (softly) committed to use nighly build for testing.
Considering this, and the fact that now a few providers use beta/rc releases published by CAPI, I agree we can close this
and eventually reconsider in future the idea of a CAPI release-informing dashboard

/close

[chrischdi]

/close

[k8s-ci-robot]

@chrischdi: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.