Clickhouse v18.14.17 did not refresh configration in zookeeper when it changed

[dqminh]

Hi,

We are seeing cases in our cluster where clickhouse on many nodes ( but not all ) didnt refresh their configuration in zookeeper when it changed. Also number of watches in the new version differs much from our old version (1.1.54342). With old version we have around 25K watches per server, but with new version we have only around 1K

select version()

┌─version()─┐
│ 1.1.54388 │
└───────────┘

cc @bobrik @vavrusa @bocharov @alexey-milovidov

[dqminh]

We are still seeing this issue on v18.14.17.

An example of the timeline:

1. Node get SessionExpired
2. Some time after, to test this, i manually update the config in zookeeper, and check generated config in /etc/clickhouse-server/config-preprocessed.xml. Change is not propagated. This is tried a few times without any results.
3. Manually system reload config on the node
4. Repeat 2. This is always successful now

Looking at ConfigReloader and ZookeeperNodeCache, i dont see how watches for existing configs can be re-established after SessionExpired event. @alexey-milovidov @ztlpn do you have any suggestions ?

[ztlpn]

Will look into this. The intention is that SessionExpired event should clear the znode cache and all needed znodes would be reloaded.

[ztlpn]

Fixed since 19.1.6.