The operator does not watch clickhouse Keeper StatefulSets

[mandreasik]

Hi,

I'm using clickhouse-operator version 0.24.0 and I've encountered the following issue:

The operator does not watch clickhouse Keeper StatefulSets.

I accidentally deleted one of the clickhouse Keeper StatefulSets, and the operator did not take any actions. This left me with a broken cluster as the pod vanished along with the StatefulSet.

I tried to force operator to take action by modifying sth in the clickHouseKeeperInstallation settings section, but after that all pods were killed, operator started only first one clickhouse-keeper and hanged forever.

On the other hand, clickhouse instances have this feature. If you delete a StatefulSet, it immediately gets restored in the cluster, as one would expect from the operator.

How to reproduce:

1. Use the example cluster configuration: https://github.com/Altinity/clickhouse-operator/blob/master/docs/chk-examples/02-extended-3-nodes.yaml
2. Delete one of the StatefulSets.
3. Check the CHK resource status:

NAME       CLUSTERS   HOSTS   STATUS      HOSTS-COMPLETED   AGE
extended   1          3       Completed                     4m57s

4. Check the pods:

kubectl get pods -l clickhouse-keeper.altinity.com/chk=extended
NAME                          READY   STATUS    RESTARTS   AGE
chk-extended-cluster1-0-0-0   1/1     Running   0          5m20s
chk-extended-cluster1-0-1-0   1/1     Running   0          5m20s

Congratulations, you have a degraded cluster ;/