A deleted Machine can block cluster initialization indefinitely (flake)

[gab-satchi]

What steps did you take and what happened:

• We found a cluster that was stuck initializing the first control plane.
• The kubeadmConfig for the machine was in a loop with "A control plane is already being initialized, requeing until control plane is ready"
• Looking at the configMap holding the init lock for that cluster, we found a machine that no longer exists.

The theory is the lock failed to release here and the machine + kubeadm config was removed before the next reconcile which left the outdated configMap lock behind.

What did you expect to happen:

• the lock will eventually get released if the machine owning it no longer exists

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version: v0.3.23
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[vincepri]

/assign @killianmuldoon
Let's investigate if the lock removal is not re-entrant

/milestone v1.1

[randomvariable]

For a hint on reproducability, create X number of workload clusters simultaneously above the controller concurrency limit. The issue was reported when CAPV was being used. Would be interesting to see if it can be reproduced using CAPD.

[killianmuldoon]

For a hint on reproducability, create X number of workload clusters simultaneously above the controller concurrency limit. The issue was reported when CAPV was being used. Would be interesting to see if it can be reproduced using CAPD.

I can't seem to reproduce on CAPD as described - I've got the concurrency limits for clusters, machines and kubeadmbootstrap set to 1 (and a couple of combinations of them set and unset) and I'm able to initialize 10 cluster control planes simultaneously (though it's heating the room nicely 😄 )

[killianmuldoon]

I haven't been able to reproduce the flake - but there is no unlocking mechanism for the ControlPlaneInitMutex if a machine is deleted - I've been able to reproduce that locally and put a fix into the locking mechanism to check if the exiting lock is valid in #5824

@gab-satchi do you have some way to reproduce this flake?

[killianmuldoon]

#5824 may be the source of this issue. This issue should be reopened if the test continues to be flaky.

[killianmuldoon]

#5855 Ports this to release 1.0.X
#5856 Ports this to release 0.4.X
#5860 Ports this to release 0.3.X