[DOCS] uni-directional CCR disaster recovery

docs/reference/ccr/index.asciidoc

@@ -308,6 +308,7 @@ and use {ccr}:
 * <<ccr-upgrading>>
 
 include::getting-started.asciidoc[]
+include::uni-directional-disaster-recovery.asciidoc[]
 include::managing.asciidoc[]
 include::auto-follow.asciidoc[]
 include::upgrading.asciidoc[]

docs/reference/ccr/uni-directional-disaster-recovery.asciidoc

@@ -0,0 +1,195 @@
+[role="xpack"]
+[[ccr-disaster-recovery-uni-directional-tutorial]]
+=== Tutorial: Disaster recovery based on uni-directional {ccr}
+++++
+<titleabbrev>Uni-directional disaster recovery</titleabbrev>
+++++
+
+////
+[source,console]
+----
+PUT kibana_sample_data_ecommerce
+----
+// TESTSETUP 
+
+[source,console]
+----
+DELETE kibana_sample_data_ecommerce
+----
+// TEARDOWN
+////
+
+
+Learn how to failover and failback between two clusters based on
+uni-directional {ccr}. You can also visit <<ccr-disaster-recovery-bi-directional-tutorial,ccr bi-directional tutorial>> to set up replicating data streams that automatically failover and failback without human
+intervention.
+
+* Setting up uni-directional {ccr} replicated from `clusterA`
+to `clusterB`.
+* Failover. If `clusterA` goes offline, `clusterB` needs to "promote" follower
+indices to regular indices to allow write operations. All ingestion will need to
+be redirected to `clusterB`, this is controlled by the clients (Logstash, Beats,
+Agents, etc). 
+* Failback. When `clusterA` is back online, it assumes the role of a follower
+and replicates the leader indices from `clusterB`.
+
+image::images/ccr-uni-directional-disaster-recovery.png[Uni-directional cross cluster replication failover and failback]
+
+NOTE: {ccr-cap} provides functionality to replicate user-generated indices only.
+{ccr-cap} isn't designed for replicating system-generated indices or snapshot
+settings, and can't replicate {ilm-init} or {slm-init} policies across clusters.
+Learn more in {ccr} <<ccr-limitations,limitations>>
+
+==== Prerequisites
+Before completing this tutorial,
+<<ccr-getting-started-tutorial,set up cross-cluster replication>> to connect two 
+clusters and configure a follower index. 
+
+In this tutorial, `kibana_sample_data_ecommerce` is replicated from `clusterA` to `clusterB`.
+
+[source,console]
+----
+### On clusterB ###
+PUT _cluster/settings
+{
+  "persistent": {
+    "cluster": {
+      "remote": {
+        "clusterA": {
+          "mode": "proxy",
+          "skip_unavailable": "true",
+          "server_name": "clustera.es.australia-southeast1.gcp.elastic-cloud.com",
+          "proxy_socket_connections": "18",
+          "proxy_address": "clustera.es.australia-southeast1.gcp.elastic-cloud.com:9400"
+        }
+      }
+    }
+  }
+}
+----
+// TEST[setup:host]
+// TEST[s/"server_name": "clustera.es.australia-southeast1.gcp.elastic-cloud.com",//]
+// TEST[s/"proxy_socket_connections": 18,//]
+// TEST[s/clustera.es.australia-southeast1.gcp.elastic-cloud.com:9400/\${transport_host}/]
+// TEST[s/clusterA/remote_cluster/]
+
++
+[source,console]
+----
+### On clusterB ###
+PUT /kibana_sample_data_ecommerce_copy/_ccr/follow?wait_for_active_shards=1
+{
+  "remote_cluster": "clusterA",
+  "leader_index": "kibana_sample_data_ecommerce"
+}
+----
+// TEST[continued]
+// TEST[s/clusterA/remote_cluster/]
+
+IMPORTANT: Writes (such as ingestion or updates) should occur only on the leader
+index. Follower indices are read-only and will reject any writes.
+
+
+==== Failover when `clusterA` is down
+
+. Promote the follower indices in `clusterB` into regular indices so 
+that it will be accepting writes. This can be achieved by:
+* Pauses indexing following for the follower index.
+* Closes the follower index.
+* Unfollows the leader index.
+* Opens the follower index (which at this point is a regular index).
+
++
+[source,console]
+----
+### On clusterB ###
+POST /kibana_sample_data_ecommerce_copy/_ccr/pause_follow
+POST /kibana_sample_data_ecommerce_copy/_close           
+POST /kibana_sample_data_ecommerce_copy/_ccr/unfollow    
+POST /kibana_sample_data_ecommerce_copy/_open
+----
+// TEST[continued]
+
+. On the Client (logstash, beats, agent) side, manually re-enable ingestion of
+`kibana_sample_data_ecommerce` and redirect traffic to the `clusterB`. You should
+also redirect all search traffic to the `clusterB` cluster during
+this time. We can simulate this by ingesting documents into this index. You should
+notice this index is now writable.
++
+[source,console]
+----
+### On clusterB ###
+POST kibana_sample_data_ecommerce_copy/_doc/
+{
+  "user": "kimchy"
+}
+----
+// TEST[continued]
+
+==== Failback when `clusterA` comes back
+
+When `clusterA` comes back, `clusterB` becomes the new leader and `clusterA` becomes the follower. 
+
+. Set up remote cluster `clusterB` on `clusterA`.
++
+[source,console]
+----
+### On clusterA ###
+PUT _cluster/settings
+{
+  "persistent": {
+    "cluster": {
+      "remote": {
+        "clusterB": {
+          "mode": "proxy",
+          "skip_unavailable": "true",
+          "server_name": "clusterb.es.australia-southeast1.gcp.elastic-cloud.com",
+          "proxy_socket_connections": "18",
+          "proxy_address": "clusterb.es.australia-southeast1.gcp.elastic-cloud.com:9400"
+        }
+      }
+    }
+  }
+}
+----
+// TEST[setup:host]
+// TEST[s/"server_name": "clusterb.es.australia-southeast1.gcp.elastic-cloud.com",//]
+// TEST[s/"proxy_socket_connections": 18,//]
+// TEST[s/clusterb.es.australia-southeast1.gcp.elastic-cloud.com:9400/\${transport_host}/]
+// TEST[s/clusterB/remote_cluster/]
+
+. Existing data need to be discarded before you can turn any index into a
+follower. Ensure the most up-to-date data is available on `clusterB` prior to
+deleting any indices on `clusterA`.  
++
+[source,console]
+----
+### On clusterA ###
+DELETE kibana_sample_data_ecommerce
+----
+// TEST[skip:need dual cluster setup]
+
+
+. Create a follower index on `clusterA`, now following the leader index in
+`clusterB`.  
++
+[source,console]
+----
+### On clusterA ###
+PUT /kibana_sample_data_ecommerce/_ccr/follow?wait_for_active_shards=1
+{ 
+  "remote_cluster": "clusterB", 
+  "leader_index": "kibana_sample_data_ecommerce_copy" 
+}
+----
+// TEST[continued]
+// TEST[s/clusterB/remote_cluster/]
+
+. You should now see updated documents from this index on the follower cluster.
++
+[source,console]
+----
+### On clusterA ###
+GET kibana_sample_data_ecommerce/_search?q=kimchy
+----
+// TEST[continued]