Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] uni-directional CCR disaster recovery #87099

Closed
wants to merge 15 commits into from
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/reference/ccr/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ and use {ccr}:
* <<ccr-upgrading>>

include::getting-started.asciidoc[]
include::disaster-recovery.asciidoc[]
Leaf-Lin marked this conversation as resolved.
Show resolved Hide resolved
include::managing.asciidoc[]
include::auto-follow.asciidoc[]
include::upgrading.asciidoc[]
154 changes: 154 additions & 0 deletions docs/reference/ccr/uni-directional-disaster-recovery.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
[role="xpack"]
[[ccr-disaster-recovery-uni-directional-tutorial]]
=== Tutorial: Disaster recovery based on uni-directional {ccr}
++++
<titleabbrev>Uni-directional disaster recovery</titleabbrev>
++++


Learn how to failover and failback between two clusters based on
uni-directional {ccr}. You can also visit <<ccr-disaster-recovery-bi-directional-tutorial,ccr bi-directional tutorial>> to set up replicating data streams that automatically failover and failback without human
intervention.

* Setting up uni-directional {ccr} replicated from `clusterA`
to `clusterB`.
* Failover. If `clusterA` goes offline, `clusterB` needs to "promote" follower
indices to regular indices to allow write operations. All ingestion will need to
be redirected to `clusterB`, this is controlled by the clients (Logstash, Beats,
Agents, etc).
* Failback. When `clusterA` is back online, it assumes the role of a follower
and replicates the leader indices from `clusterB`.

image::images/ccr-uni-directional-disaster-recovery.png[Uni-directional cross cluster replication failover and failback]

NOTE: {ccr-cap} provides functionality to replicate user-generated indices only.
{ccr-cap} isn't designed for replicating system-generated indices or snapshot
settings, and can't replicate {ilm-init} or {slm-init} policies across clusters.
Learn more in {ccr} <<ccr-limitations,limitations>>

==== Prerequisites
Before completing this tutorial,
<<ccr-getting-started-tutorial,set up cross-cluster replication>> to connect two
clusters and configure a follower index.

In this tutorial, `kibana_sample_data_ecommerce` is replicated from `clusterA` to `clusterB`.

[source,console]
----
### On clusterB ###
PUT _cluster/settings
{
"persistent": {
"cluster": {
"remote": {
"clusterA": {
"mode": "proxy",
"skip_unavailable": "true",
"server_name": "clustera.es.australia-southeast1.gcp.elastic-cloud.com",
"proxy_socket_connections": "18",
"proxy_address": "clustera.es.australia-southeast1.gcp.elastic-cloud.com:9400"
}
}
}
}
}
PUT /kibana_sample_data_ecommerce/_ccr/follow
{
"remote_cluster": "clusterA",
"leader_index": "kibana_sample_data_ecommerce"
}
----
IMPORTANT: Writes (such as ingestion or updates) should occur only on the leader
index. Follower indices are read-only and will reject any writes.


==== Failover when `clusterA` is down

. Promote the follower indices in `clusterB` into regular indices so
that it will be accepting writes. This can be achieved by:
* Pauses indexing following for the follower index.
* Closes the follower index.
* Unfollows the leader index.
* Opens the follower index (which at this point is a regular index).

+
[source,console]
----
### On clusterB ###
POST /kibana_sample_data_ecommerce/_ccr/pause_follow
POST /kibana_sample_data_ecommerce/_close
POST /kibana_sample_data_ecommerce/_ccr/unfollow
POST /kibana_sample_data_ecommerce/_open
----

. On the Client (logstash, beats, agent) side, manually re-enable ingestion of
`kibana_sample_data_ecommerce` and redirect traffic to the `clusterB`. You should
also redirect all search traffic to the `clusterB` cluster during
this time. We can simulate this by ingesting documents into this index. You should
notice this index is now writable.
+
[source,console]
----
### On clusterB ###
POST kibana_sample_data_ecommerce/_doc/
{
"user": "kimchy"
}
----

==== Failback when `clusterA` comes back

When `clusterA` comes back, `clusterB` becomes the new leader and `clusterA` becomes the follower.

. Set up remote cluster `clusterB` on `clusterA`.
+
[source,console]
----
### On clusterA ###
PUT _cluster/settings
{
"persistent": {
"cluster": {
"remote": {
"clusterB": {
"mode": "proxy",
"skip_unavailable": "true",
"server_name": "clusterb.es.australia-southeast1.gcp.elastic-cloud.com",
"proxy_socket_connections": "18",
"proxy_address": "clusterb.es.australia-southeast1.gcp.elastic-cloud.com:9400"
}
}
}
}
}
----
. Existing data need to be discarded before you can turn any index into a
follower. Ensure the most up-to-date data is available on `clusterB` prior to
deleting any indices on `clusterA`.
+
[source,console]
----
### On clusterA ###
DELETE kibana_sample_data_ecommerce
----

. Create a follower index on `clusterA`, now following the leader index in
`clusterB`.
+
[source,console]
----
### On clusterA ###
PUT /kibana_sample_data_ecommerce/_ccr/follow
{
"remote_cluster": "clusterB",
"leader_index": "kibana_sample_data_ecommerce"
}
----

. You should now see updated documents from this index on the follower cluster.
+
[source,console]
----
### On clusterA ###
GET kibana_sample_data_ecommerce/_search?q=kimchy
----