KCP does not reconcileEtcdMembers for deleted machines (release-0.3 branch)

[fabriziopandini]

What steps did you take and what happened:

• Created a 3 CP cluster using CAPI from release-0.3 branch
• Deleted a machine (using KCP remediation)
• KCP blocked when trying to restore the 3rd CP machine and reporting

failed to pass etcd health check: there are 2 healthy etcd pods, but 3 etcd members

What did you expect to happen:
KCP to re-create the 3rd CP machine

Anything else you would like to add:
Most probably this is a regression introduced by #3806 and more specifically by the nested if introduced in this change
https://github.com/kubernetes-sigs/cluster-api/blob/release-0.3/controlplane/kubeadm/controllers/controller.go#L502-L516

After a machine is deleted, EtcdIsHealthy returns an error due to

cluster-api/controlplane/kubeadm/internal/workload_cluster_etcd.go

Lines 129 to 131 in 7ed9b74

	if expectedMembers != len(knownMemberIDSet) {
	return response, errors.Errorf("there are %d healthy etcd pods, but %d etcd members", expectedMembers, len(knownMemberIDSet))
	}

as a consequence we are entering in

cluster-api/controlplane/kubeadm/controllers/controller.go

Line 503 in 7ed9b74

errList = append(errList, errors.Wrap(err, "failed to pass etcd health check"))

and not calling ReconcileEtcdMembers

Environment:

/kind bug

[fabriziopandini]

/milestone v0.3.11
/priority critical

[k8s-ci-robot]

@fabriziopandini: The label(s) priority/critical cannot be applied, because the repository doesn't have them

In response to this:

/milestone v0.3.11
/priority critical

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabriziopandini]

/priority critical-urgent

[fabriziopandini]

/assign
/lifecycle active

[fabriziopandini]

#3900 is merged
/close

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

#3900 is merged
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.