Framework: Support starting controller version in clusterctl upgrade tests

[lentzi90]

User Story

As a provider developer I would like to select the controller version to start from for clusterctl upgrade tests since the provider API version can differ from the CAPI contract.

Detailed Description

Currently the clusterctl upgrade test always starts from the latest version supporting the selected contract:

cluster-api/test/e2e/clusterctl_upgrade.go

Line 265 in 6c65434

InfrastructureProviders: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, input.E2EConfig.InfrastructureProviders()...),

Providers such as CAPO can have different API versions even for the same contract version. For example, CAPO v0.5 and v0.6 are both compatible with CAPI v1beta1, but have different CAPO API versions (v1alpha4 and v1alpha5). This means that there is no way to select v0.5 as starting point if v0.6 is also in the e2e config.

It is possible to work around this limitation by crafting specific e2e configs for each situation, but it is hardly convenient.

Anything else you would like to add:

For reference, CAPO compatibility matrix: https://github.com/kubernetes-sigs/cluster-api-provider-openstack#compatibility-with-cluster-api-and-kubernetes-versions

/kind feature

[sbueringer]

/triage accepted

Sound reasonable to me.

Looking at:

cluster-api/test/e2e/clusterctl_upgrade.go

Lines 262 to 267 in e032bec

	CoreProvider: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, config.ClusterAPIProviderName)[0],
	BootstrapProviders: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, config.KubeadmBootstrapProviderName),
	ControlPlaneProviders: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, config.KubeadmControlPlaneProviderName),
	InfrastructureProviders: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, input.E2EConfig.InfrastructureProviders()...),
	IPAMProviders: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, input.E2EConfig.IPAMProviders()...),
	RuntimeExtensionProviders: input.E2EConfig.GetProviderLatestVersionsByContract(initContract, input.E2EConfig.RuntimeExtensionProviders()...),

I think the easiest way is to just add additional fields to ClusterctlUpgradeSpecInput. I would suggest:

	InitWithCoreProvider              string
	InitWithBootstrapProviders        []string
	InitWithControlPlaneProviders     []string
	InitWithInfrastructureProviders   []string
	InitWithIPAMProviders             []string
	InitWithRuntimeExtensionProviders []string

If one of the fields is set we would use its value directly instead of calculating it via the contract. If I see correctly the format would be something like aws:v0.5.0.

[fabriziopandini]

I will comment on the issue as well, I'm only a bit concerned about the spreading of versions in the config file, but as soon as we sort this out, I'm +1 for changes making the E2E test framework easier to use for providers