Remote state publication (#7364)

* Remote state publication

Signed-off-by: Sooraj Sinha <[email protected]>

* Address comments

Signed-off-by: Sooraj Sinha <[email protected]>

* Simplify the remote publication description

Signed-off-by: Sooraj Sinha <[email protected]>

* Update remote-cluster-state.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update _tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Sooraj Sinha <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

_tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md

@@ -54,12 +54,43 @@ In addition to the mandatory static settings, you can configure the following dy
 
 Setting | Default | Description
 :--- | :--- | :---
-`cluster.remote_store.state.index_metadata.upload_timeout` | 20s | The amount of time to wait for index metadata upload to complete. Note that index metadata for separate indexes is uploaded in parallel.
-`cluster.remote_store.state.global_metadata.upload_timeout` | 20s | The amount of time to wait for global metadata upload to complete. Global metadata contains globally applicable metadata, such as templates, cluster settings, data stream metadata, and repository metadata.
-`cluster.remote_store.state.metadata_manifest.upload_timeout` | 20s | The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files. 
+`cluster.remote_store.state.index_metadata.upload_timeout` | 20s | Deprecated. Use `cluster.remote_store.state.global_metadata.upload_timeout` instead.
+`cluster.remote_store.state.global_metadata.upload_timeout` | 20s | The amount of time to wait for the cluster state upload to complete.
+`cluster.remote_store.state.metadata_manifest.upload_timeout` | 20s | The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files.
+`cluster.remote_store.state.cleanup_interval` | 300s | The interval at which the asynchronous remote state clean-up task runs. This task deletes any old remote state files. 
 
 
 ## Limitations
 
 The remote cluster state functionality has the following limitations:
 - Unsafe bootstrap scripts cannot be run when the remote cluster state is enabled. When a majority of cluster-manager nodes are lost and the cluster goes down, the user needs to replace any remaining cluster manager nodes and reseed the nodes in order to bootstrap a new cluster.
+
+## Remote cluster state publication
+
+
+The cluster manager node processes updates to the cluster state. It then publishes the updated cluster state through the local transport layer to all of the follower nodes. With the `remote_store.publication` feature enabled, the cluster state is backed up to the remote store during every state update. The follower nodes can then fetch the state from the remote store directly, which reduces the overhead on the cluster manager node for publication. 
+
+To enable the feature flag for the `remote_store.publication` feature, follow the steps in the [experimental feature flag documentation]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/).
+
+Enabling the setting does not change the publication flow, and follower nodes will not send acknowledgements back to the cluster manager node
+until they download the updated cluster state from the remote store.
+
+You must enable the remote cluster state feature in order for remote publication to work. To modify the remote publication behavior, the following routing table repository settings can be used, which contain the shard allocation details for each index in the remote cluster state:
+
+```yml
+# Remote routing table repository settings
+node.attr.remote_store.routing_table.repository: my-remote-routing-table-repo
+node.attr.remote_store.repository.my-remote-routing-table-repo.type: s3
+node.attr.remote_store.repository.my-remote-routing-table-repo.settings.bucket: <Bucket Name 3>
+node.attr.remote_store.repository.my-remote-routing-table-repo.settings.region: <Bucket region>
+```
+
+You do not have to use different remote store repositories for state and routing because both state and routing can use the same repository settings.
+
+To configure remote publication, use the following cluster settings.
+
+Setting | Default | Description
+:--- | :--- | :---
+`cluster.remote_store.state.read_timeout` | 20s | The amount of time to wait for remote state download to complete on the follower node.
+`cluster.remote_store.routing_table.path_type` | HASHED_PREFIX | The path type to be used for creating an index routing path in the blob store. Valid values are `FIXED`, `HASHED_PREFIX`, and `HASHED_INFIX`.
+`cluster.remote_store.routing_table.path_hash_algo` | FNV_1A_BASE64 | The algorithm to be used for constructing the prefix or infix of the blob store path. This setting is applied if `cluster.remote_store.routing_table.path_type` is `hashed_prefix` or `hashed_infix`. Valid algorithm values are `FNV_1A_BASE64` and `FNV_1A_COMPOSITE_1`.