Test flaking with "machine.cluster.x-k8s.io/certificates-expiry" does not exist

[killianmuldoon]

Which jobs are flaking?

e.g. https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main

Which tests are flaking?

e.g. capi-e2e.[It] When testing ClusterClass rollouts [ClusterClass] Should successfully rollout the managed topology upon changes to the ClusterClass

Since when has it been flaking?

Since merging #8711

Testgrid link

https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main

Reason for failure (if possible)

Probably related to changes in #8711 where the order of putting this annotation in place was changed.

Failed run here: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-main/1661640513482133504

Anything else we need to know?

No response

Label(s) to be applied

/kind flake

[killianmuldoon]

/triage accepted

[sbueringer]

Just curious. Can you add a link to the specific job run please?

[killianmuldoon]

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-main/1661640513482133504

[killianmuldoon]

I'm hopefully going to do a deep dive on this later on - hopefully not to tough to fix seeing as it's just been introduced Offchance it's SSA 🤔 - which would be more complex, but I'm betting it's a timing issue.

[sbueringer]

Thx for the link!

[killianmuldoon]

/assign @adilGhaffarDev

[k8s-ci-robot]

@killianmuldoon: GitHub didn't allow me to assign the following users: adilGhaffarDev.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @adilGhaffarDev

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[killianmuldoon]

/reopen

Until we see some evidence this fixes the flake.

[k8s-ci-robot]

@killianmuldoon: Reopened this issue.

In response to this:

/reopen

Until we see some evidence this fixes the flake.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[killianmuldoon]

Fix in #8758 didn't work - looking at this again I think the fail in

cluster-api/test/e2e/clusterclass_rollout.go

Line 815 in c340c13

Fail(fmt.Sprintf("key %q does not exist in map %s", key, m))

needs to be more tolerant instead of directly failing if this annotation isn't available yet.

@sbueringer - WDYT?

[killianmuldoon]

Change in #8761 seemed to fix this for main, but the solution is to completely ignore the error. I'd like to revisit this to understand how we can handle the error inside the Eventually call instead of not handling it.

/assign

[sbueringer]

@killianmuldoon Can we please rollback the change in #8761? Let's instead pass in Gomega into without and then use g.Fail

[killianmuldoon]

@killianmuldoon Can we please rollback the change in #8761? Let's instead pass in Gomega into without and then use g.Fail

Yeah - something like that is the plan. Let's wait until we have some functioning CI first.

[killianmuldoon]

This test is now green.

/close

[k8s-ci-robot]

@killianmuldoon: Closing this issue.

In response to this:

This test is now green.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[killianmuldoon]

/reopen

This issue has re-emerged in the last few days - https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#7b3ebfaf894e9e2e1fe0

[k8s-ci-robot]

@killianmuldoon: Reopened this issue.

In response to this:

/reopen

This issue has re-emerged in the last few days - https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#7b3ebfaf894e9e2e1fe0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[killianmuldoon]

/close

The fix in #8897 has solved this for now.

[k8s-ci-robot]

@killianmuldoon: Closing this issue.

In response to this:

/close

The fix in #8897 has solved this for now.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.