[META][2.15] Cluster State Publication using Remote State

[soosinha]

Please describe the end goal of this project

For remote store enabled clusters, the cluster state is being published to a remote repository. This can be used to publish the cluster state to all nodes in the cluster. Instead of sending the cluster state object over the transport layer, the cluster manager node can just send the term and version of the cluster state over the transport layer which can then be used to download the corresponding cluster state by all the nodes directly from the remote store.

This meta issue includes the tasks required for experiemental release in 2.15.

Supporting References

RFC: #13257

Issues

• [Remote Cluster State] Add new transport call to send cluster state term and version #13730
• [Remote Cluster State] Create interfaces for remote cluster state #13791
• Implementation of RemoteWritableEntity for objects to uploaded to remote store #13834
• Upload blob from input stream #13836
• Create serde utility for Writable classes #14095
• Add POJO classes required for cluster state publication from remote #14006
• [Remote Cluster State] Download cluster state on follower nodes #13682
• [Remote Cluster State] Upload ephemeral cluster state objects(cluster blocks, nodes, customs) to remote #13728
• [Remote Cluster State] Download the cluster state objects asynchronously and in parallel #13729

Related component

Cluster Manager

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@soosinha Thanks for creating this issue

[lukas-vlcek]

Is #13576 already part of this effort? I just want to make sure it is linked if it is related (and possible listed in the Issues list).

[shiv0408]

@lukas-vlcek Yes, that PR is required for this. Created a new issue #13691 as a task for this.
@soosinha can you please help in adding this issue to the task list of this Meta issue.