CAPA control plane machine health checks

[alex-dabija]

Story

-As a cluster admin, I want the control plane nodes to be recreated if basic machine health checks fail in order to improve the stability of CAPA clusters.

Towards epic.

Background

CAPA clusters with control plane machines in invalid states fail to recover because there are no health checks configured.

CAPI supports machine health checks for control plane nodes and they need to be configured for CAPA clusters.

Links

• Configure a MachineHealthCheck;
• CAPI healthchecking limitations.

[bavarianbidi]

my 2 cents:

contrary to my initial concerns about having MachineHealthChecks for KCP the current implementation in CAPI is strived to not break an existing etcdCluster (result of func canSafelyRemoveEtcdMember from kcp remediation).

The only "critical" path i still see is the fact that if an etcd cluster is "broken" and a human tries to fix the etcd cluster by doing some manual steps, the MachineHealthCheck could kick in and rotate a machine you're still relying on.

Kudos to Erkan for creating this great list of Scenarios:

Example Scenarios (tested in Add MachineHealthCheck for all nodes in Openstack):
Setup: 1 Control Plane + 3 Infra
Case: The control plane node is unhealthy
Behavior: No remediation because of etcd quorum loss.

Setup: 1 Control Plane + 3 Infra
Case: One infra node is unhealthy
Behavior: Infra node will be remediated

Setup: 1 Control Plane + 3 Infra
Case: 2 infra nodes are unhealthy
Behavior: No remediation because of maxUnhealthy limit (50% > 40%)

Setup: 3 Control Plane + 3 Infra
Case: One control plane node is unhealthy
Behavior: Control plane node will be remediated

Setup: 3 Control Plane + 3 Infra
Case: One control plane, one infra node are unhealthy
Behavior: Both will be remediated (33% > 40%)

Setup: 3 Control Plane + 3 Infra
Case: Two infra nodes are unhealthy
Behavior: Both will be remediated (33% > 40%)

Setup: 3 Control Plane + 3 Infra
Case: Two control plane nodes are unhealthy
Behavior: No remediation because of etcd quorum loss.

Open issues for other providers:

• MachineHealthCheck for CAPVCD #25845
• MachineHealthCheck for CAPV #2022

[erkanerol]

The only "critical" path i still see is the fact that if an etcd cluster is "broken" and a human tries to fix the etcd cluster by doing some manual steps, the MachineHealthCheck could kick in and rotate a machine you're still relying on.

To solve this issue, we were disabling/enabling machine health checks while fixing the cluster manually.

# to disable MachineHealthCheck temporarily
function annotate_machines_to_disable_remediation(){
    local namespace="$1"
    local cluster="$2"
    kubectl -n "$namespace" annotate machine -l cluster.x-k8s.io/cluster-name="$cluster" "cluster.x-k8s.io/skip-remediation"=""
}

function deannotate_machines_to_enable_remediation(){
    local namespace="$1"
    local cluster="$2"
    kubectl -n "$namespace" annotate machine -l cluster.x-k8s.io/cluster-name="$cluster" "cluster.x-k8s.io/skip-remediation"-
}

See https://intranet.giantswarm.io/docs/support-and-ops/ops-recipes/remove-errors-from-capi-capo-crs/

---

The biggest issue we had in CAPO was that there was a bug in CAPO controller that adds transient errors to infra CRs, which are propagated to CAPI CRs too. We had to delete them manually to fix clusters because of the maxUnhealthy limit. See https://github.com/giantswarm/giantswarm/issues/22443

[fiunchinho]

@calvix @erkanerol @bavarianbidi do you think having two different MachineHealthChecks (one for the control plane and another one for the workers) is needed / better? Any issues re-using the same MachineHealthCheck for both the CP and the workers?

[erkanerol]

It depends on whether you want to define different unhealthyConditions for CP nodes. Note that maxUnhealthy will be applied for all nodes if you use a single MHC.

For CAPVCD, I added for only worker nodes. I want to observe the experience for a while.

[bavarianbidi]

As control plane nodes are in multiple cases a bit "different" i would propose doing two different MachineHealthChecks.

[calvix]

So in the end, this is only applied for the control plane nodes as stated in the issue name as the support for machine-pool indeed does not exist, and I added a filter to avoid killing the bastion node.

[alex-dabija]

We'll update the following MCs:

• golem;
• goat;
• grizzly.

[calvix]

done