[e2e] Replace conformance test with ClusterUpgradeConformance spec

[sedefsavas]

CAPI e2e tests are taking 30 min longer after adding ClusterUpgradeConformanceSpec #2935, as it runs conformance tests.

There is a redundancy now, we are running conformance tests in 2 different prow jobs.

There is one issue with replacing conformance test suite with ClusterUpgradeConformanceSpec, we run conformance tests on the latest unstable Kubernetes version as well. So the version needs to be dynamically set for ClusterUpgradeConformanceSpec template.

/priority important-soon
/triage accepted
/milestone v1.2.0

[sedefsavas]

cc @pydctw

[pydctw]

CAPI e2e tests are taking 30 min longer after adding ClusterUpgradeConformanceSpec #2935, as it runs conformance tests.

I also noticed it.

My plan is

• Skip conformance test for ClusterUpgradeConformanceSpec in CAPI e2e test. We will only test cluster upgrade functionality here.
• Looks like we can replace currently disabled KCPUpgradeSpecs with ClusterUpgradeConformanceSpec. Asking clarifying questions to CAPI team for control plane count programmability.
• Will work on replacing the existing conformance suite with ClusterUpgradeConformanceSpec.

[sedefsavas]

Sounds good. If you can replace conformance suite with the upgrade spec, then you won't need to test it in CAPI e2e tests anymore, so only second and third point is needed.

[pydctw]

/assign

[pydctw]

• Skip conformance test for ClusterUpgradeConformanceSpec in CAPI e2e test. We will only test cluster upgrade functionality here.
• Looks like we can replace currently disabled KCPUpgradeSpecs with ClusterUpgradeConformanceSpec. Asking clarifying questions to CAPI team for control plane count programmability.

Submitted a PR to address the first two points.

[pydctw]

• Will work on replacing the existing conformance suite with ClusterUpgradeConformanceSpec.

Questions for replacing the existing conformance suite.

• Are there gaps in the existing conformance test that ClusterUpgradeConformanceSpec will address? Trying to understand the why.
• If I understand it correctly, we are running two conformance tests, one with stable k8s version, the other against k8s main branch. I am not sure how to do it in ClusterUpgradeConformance test. The test uses KUBERNETES_VERSION_UPGRADE_TO in e2e_conf.yaml. I may be missing something simple - is there a way to program it dynamically?

[sedefsavas]

We can move forward with the rest of the changes as cluster-api removed KCPUpgrade spec.
Updating the cluster api version in this PR:
#3058

[sedefsavas]

@pydctw

• Are there gaps in the existing conformance test that ClusterUpgradeConformanceSpec will address? Trying to understand the why.

One extra signal we are getting by using ClusterUpgradeConformanceSpec is upgraded cluster is passing the conformance, not just the standalone cluster with that particular version.
Also, we won't need to maintain conformance suit if we move to ClusterUpgradeConformanceSpec and this will make tests more standardized. e2e tests are already complex so calling cluster-api specs wherever possible, reduces our burden to maintain.

If I understand it correctly, we are running two conformance tests, one with stable k8s version, the other against k8s main branch. I am not sure how to do it in ClusterUpgradeConformance test. The test uses KUBERNETES_VERSION_UPGRADE_TO in e2e_conf.yaml. I may be missing something simple - is there a way to program it dynamically?

I think this might work:
Use flavor = "conformance-ci-artifacts and set the version variable like this:
shared.SetEnvVar("KUBERNETES_VERSION_UPGRADE_TO", v1.23.2)`

[sedefsavas]

Also we have a disabled test:
ginkgo.PDescribe("[Serial] Upgrade to main branch Kubernetes", func()

We should probably delete this, looks redundant.

[sedefsavas]

@pydctw
We don't need worker machines during KCP upgrade tests, it is supported to use 0 worker machines:
kubernetes-sigs/cluster-api#5926

[pydctw]

@pydctw We don't need worker machines during KCP upgrade tests, it is supported to use 0 worker machines: kubernetes-sigs/cluster-api#5926

Yes, when I submitted the PR, having 0 worker machines were not supported in CAPI yet.
When I come back to this issue, will refactor the code. Thanks for the info.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.