Scalable cluster state propagation (zen discovery)

[ghost]

In the current implementation of zen discovery the cluster state broadcasting by the master does not scale too well as every change in the cluster state results in the cluster state being sent to every single node by the master.

This can lead to quite some traffic, in an cluster with 500 nodes and 15000 shards, even when the cluster state by itself has only 1MB, the master node will sent a total of 500MB per cluster state update.

So it should be possible to only send a diff of the changes to the nodes instead of the full cluster state. In case the nodes recognize that they missed an update, they could query the master node to resend the full cluster state to them.

Another possibility would be a tiered distribution of the state, where the master node does not directly send the updates to every single node, but through other nodes which forward the state to the other nodes in the cluster. A gossipping protocol could here be used.

If the clusterstate does not need to be distributed in realtime, one could also throttle the propagation of the state, so that when changes come in rapid succession, the changes can be merged before the state has been sent out to every node.

Any ideas?

[clintongormley]

Duplicate of #6295