Recovering from node failures

[aedades-cw]

Hello,

My team and I were simulating a node failure by cordoning a node and force killing the Kafka broker pod on it, but Kubernetes was unable to schedule the pod on other nodes because of the PVC.

The KafkaNodePool is configured to use a storage class that relies on local persistent volumes that have a node affinity. Snippet of kubectl describe pv:

Status:            Bound
Reclaim Policy:    Retain
Node Affinity:     
  Required Terms:  
    Term 0:        topology.topolvm.io/node in [{node ID}]

• We tried to create a KafkaRebalance resource as described in https://strimzi.io/docs/operators/latest/deploying#proc-scaling-down-node-pools-str to remove the broker from the cluster and assign partition replicas, but KCC wasn’t able to because the Kafka cluster was not in a healthy state.

Workaround:

• We deleted the pod & PVC by applying an annotation to the pod as described in https://strimzi.io/docs/operators/0.22.1/full/using#proc-manual-delete-pod-pvc-kafka-str and the pod and PVC were recreated on a different node. Partitions were replicated to the broker on the new node.

In the event of an actual node failure, we would not want to delete the PVC until the cluster were recovered with all partitions replicated so that we can recover data from the disk if necessary. We weren’t able to reschedule the broker pod on another node without first deleting the PVC - what is the recommended action in this situation?

*I saw a similar thread (though the brokers there used ephemeral storage)

[scholzj]

You have to delete the PVC. However, the PVC just provides a link between the Pod and the PV. So, deleting the PVC does not mean deleting your PV / your data. If you configure your storage class to retain the PVs when PVCs are deleted, you will keep the PV and the data and you can use them if needed.

[aedades-cw]

Thank you for confirming! Thinking more about different failure modes in general, does Cruise Control have any current ability/use to help recover in certain failure scenarios (whether broker, node, etc...) or is it purely for just standard administrative cluster balancing?

[scholzj]

I don't think Cruise Control has anyhting to do with thi.