ClusterOperator unable to connect to zk and broker when installing/renewing your own CA

[FrankWang1108]

Describe the bug
This happens when we try to install our own CA when the strimzi is running and the strimzi ca is in use. It could also happen when updating the CA.

It happens after uploading new CA and certs into the *-cluster-ca-certs.
Following the renew own CA document, we have both ca.crt and ca-dates.crt in the cluster-ca-certs, and because the keystore and old ca-dates.crt can trust each other, there should still have connection. This is what is expected and the connection between zk and broker are stable because of this.

However the ClusterOperator only uses ca.crt to check for connection between zk and broker. Therefore after uoloading the new CA and certs. The ClusterOperator loses connection to zk and broker, it tries to connect to it multiple times until eventually it will perform a rolling update for zk and broker.
Once the zk updated, it will pickup the new keystore, then it will be able to connect to ClusterOperator again.

To Reproduce
Steps to reproduce the behavior:

1. bring up a cluster
2. change the cluster-ca and cluster-ca-cert to your own ca
3. check the log of ClusterOperator to see it trying to connect to zk and broker.
4. See error

Expected behavior
I would expect when the ClusterOperator trying to connect to zk and broker, it will use all .crt files in the cluster-ca-cert to check for connection, instead of just using ca.crt.
I would expect it to stay connected and trigger a rolling update because it sees new certs updates, not because it was unable to connect.

Environment (please complete the following information):

• Strimzi version: 0.25.1 (but I think the bug is still there for newer versions)
• Installation method: YAML files

[scholzj]

I'm not sure this is expected to work. Renewal is something else then replacing one CA with completely different CA. I know @tombentley is working on some improvements for CA handling, maybe he has some ideas how to do it smoothly.

[FrankWang1108]

hmm so what you are saying is: For replacing CA (not just updating certs), it's expected to have a period of time that ClusterOperator is unable to connect.

[xhl1988]

The renewal has the same issue. It's the same as we need ca-2018-09-27T17-32-00Z.crt in the renew own CA process. So the cluster operator needs all *.crt in its truststore.

[xhl1988]

@scholzj @tombentley Is there any update on this?