Support automatic and graceful config changes

[benbromhead]

Upon a user modifying a configmap, we need to detect this change and gracefully propagate it throughout the cluster that references that configmap.

[smiklosovic]

Depends whether the changes to be propagated require the restart of a node or not.

If they do and we need to restart a cluster in a rolling fashion, the rolling restart would have to be coordinated from the operator itself. We might be receptive on the changes of ConfigMap in operator and if a change is detected (another problem how to know something has actually changed), we would restart in rolling fashion. There is already restart operation in place and we just need to code a logic in operator for that.

If it is enough to change yaml's and this would be propagated internaly by Cassandra itself (is that ever the case?) I am not sure how to tackle this one because we have ConcatenatedYaml resolver and we are not actually having any cassandra.yaml file as such ...

[smiklosovic]

relates to #107 , e.g. for image changes, we just update and delete so it is restarted with new one.

On the other hand, these changes are done on statefulset itself, we are not patching our CRDs. If patching should be possible on our CRDs, we would have to detect this change on reconciliation and change / patch underlying statefulset accordingly.

I would focus only on use cases where changes to raw statefulset will not change what we want, e.g. Cassandra configuration itself. Whatever is achievable by Kubernetes itself should be delegated to it or at most we should write some layer on top of that so operator orchestrates it nicely.