Round up shard allocation / recovery / relocation concepts

docs/reference/cat/recovery.asciidoc

@@ -39,7 +39,7 @@ The cat recovery API returns information about shard recoveries, both
 ongoing and completed. It is a more compact view of the JSON
 <<indices-recovery,index recovery>> API.
 
-include::{es-ref-dir}/indices/recovery.asciidoc[tag=shard-recovery-desc]
+include::{es-ref-dir}/modules/shard-recovery-desc.asciidoc[]
 
 
 [[cat-recovery-path-params]]

docs/reference/high-availability/cluster-design.asciidoc

@@ -246,7 +246,8 @@ accumulate into a noticeable performance penalty. An unreliable network may
 have frequent network partitions. {es} will automatically recover from a
 network partition as quickly as it can but your cluster may be partly
 unavailable during a partition and will need to spend time and resources to
-resynchronize any missing data and rebalance itself once the partition heals.
+<<shard-recovery,resynchronize any missing data>> and <<shard-relocation,rebalance>> 
+itself once the partition heals.
 Recovering from a failure may involve copying a large amount of data between
 nodes so the recovery time is often determined by the available bandwidth.
 

docs/reference/indices/recovery.asciidoc

@@ -35,21 +35,7 @@ index, or alias.
 Use the index recovery API to get information about ongoing and completed shard
 recoveries.
 
-// tag::shard-recovery-desc[]
-Shard recovery is the process of initializing a shard copy, such as restoring a
-primary shard from a snapshot or syncing a replica shard from a primary shard.
-When a shard recovery completes, the recovered shard is available for search
-and indexing.
-
-Recovery automatically occurs during the following processes:
-
-* Node startup. This type of recovery is called a local store recovery.
-* Primary shard replication.
-* Relocation of a shard to a different node in the same cluster.
-* <<snapshots-restore-snapshot,Snapshot restore>> operation.
-* <<indices-clone-index,Clone>>, <<indices-shrink-index,shrink>>, or
-<<indices-split-index,split>> operation.
-// end::shard-recovery-desc[]
+include::{es-ref-dir}/modules/shard-recovery-desc.asciidoc[]
 
 The index recovery API reports information about completed recoveries only for
 shard copies that currently exist in the cluster. It only reports the last

docs/reference/intro.asciidoc

@@ -197,11 +197,7 @@ redundancy, which both protects against hardware failures and increases
 query capacity as nodes are added to a cluster. As the cluster grows (or shrinks),
 {es} automatically migrates shards to rebalance the cluster.
 
-There are two types of shards: primaries and replicas. Each document in an index
-belongs to one primary shard. A replica shard is a copy of a primary shard.
-Replicas provide redundant copies of your data to protect against hardware
-failure and increase capacity to serve read requests
-like searching or retrieving a document.
+include::{es-ref-dir}/primaries-replicas-desc.asciidoc[]
 
 The number of primary shards in an index is fixed at the time that an index is
 created, but the number of replica shards can be changed at any time, without

docs/reference/modules/cluster.asciidoc

@@ -1,9 +1,7 @@
 [[modules-cluster]]
 === Cluster-level shard allocation and routing settings
 
-_Shard allocation_ is the process of allocating shards to nodes. This can
-happen during initial recovery, replica allocation, rebalancing, or
-when nodes are added or removed.
+include::{es-ref-dir}/modules/shard-allocation-desc.asciidoc[]
 
 One of the main roles of the master is to decide which shards to allocate to
 which nodes, and when to move shards between nodes in order to rebalance the

docs/reference/modules/shard-allocation-desc.asciidoc

@@ -0,0 +1,3 @@
+Shard allocation is the process of assigning shards to nodes. This can
+happen during initial recovery, replica allocation, rebalancing, or
+when nodes are added or removed.

docs/reference/modules/shard-ops.asciidoc

@@ -0,0 +1,69 @@
+[[shard-allocation-relocation-recovery]]
+=== Shard allocation, relocation, and recovery
+
+Each <<documents-indices,index>> in Elasticsearch is divided into one or more <<scalability,shards>>, each of which acts as a copy of the index and can be replicated across multiple <<modules-node,nodes>> to protect against hardware failures.
+
+include::{es-ref-dir}/primaries-replicas-desc.asciidoc[]
+
+Over the course of normal operation, Elasticsearch allocates shards to nodes, relocates shards across nodes to balance the cluster, and recovers shards to initialize new copies. In this topic, you'll learn how these operations work and how you can control them.
+
+TIP: To learn about optimizing the number and size of shards in your cluster, refer to <<size-your-shards,Size your shards>>. To learn about how read and write operations are replicated across shards, refer to <<docs-replication,Reading and writing documents>>.
+
+[[shard-allocation]]
+==== Shard allocation
+
+include::{es-ref-dir}/modules/shard-allocation-desc.asciidoc[]
+
+By default, the primary and replica shards for an index can be allocated to any node in the cluster, and may be relocated to rebalance the cluster. 
+
+===== Adjust shard allocation settings
+
+You can control how shards are allocated using the following settings:
+
+- <<modules-cluster,Cluster-level shard allocation settings>>: Use these settings to control how shards are allocated and balanced across the entire cluster. For example, you might want to allocate nodes availability zones, or prevent certain nodes from being used so you can perform maintenance.
+
+- <<index-modules-allocation,Index-level shard allocation settings>>: Use these settings to control how the shards for a specific index are allocated. For example, you might want to allocate an index to a node in a specific data tier, or to an node with specific attributes.
+
+===== Monitor shard allocation
+
+If a shard is unassigned, it means that the shard is not allocated to any node in the cluster. This can happen if there are not enough nodes in the cluster to allocate the shard, or if the shard can't be allocated to any node that satisfies the shard allocation filtering rules.
+
+You can use the following APIs to monitor shard recovery:
+
+- <<cluster-allocation-explain,Cluster allocation explain>>
+- <<cat-allocation,cat allocation>>
+
+[[shard-recovery]]
+==== Shard recovery
+
+include::{es-ref-dir}/modules/shard-recovery-desc.asciidoc[]
+
+===== Adjust shard recovery settings
+
+To control how shards are recovered, for example the number of recoveries that can happen concurrently, the resources that can be used by recovery operations, and which indices should be prioritized for recovery, you can adjust the following settings: 
+
+- <<recovery,Index recovery settings>>
+- <<modules-cluster,Cluster-level shard allocation settings>>
+- <<index-modules-allocation,Index-level shard allocation settings>>, including <<delayed-allocation,delayed allocation>> and <<recovery-prioritization,index recovery prioritization>> 
+
+Shard recovery operations also respect general shard allocation settings. 
+
+===== Monitor shard recovery
+
+You can use the following APIs to monitor shard allocation:
+
+ - View a list of in-progress and completed recoveries using the <<cat-recovery,cat recovery API>>
+ - View detailed information about a specific recovery using the <<indices-recovery,index recovery API>>
+
+[[shard-relocation]]
+==== Shard relocation
+
+Shard relocation is the process of moving shards from one node to another. This can happen when a node joins or leaves the cluster, or when the cluster is rebalancing.
+
+When a shard is relocated, it is created as a new replica shard on the target node. When the shard is fully allocated and recovered, the old shard is deleted. If the shard being relocated is a primary shard, then the new shard is marked as primary before the old shard is deleted.
+
+===== Adjust shard relocation settings
+
+You can control how and when shards are relocated. For example, you can adjust the rebalancing settings that control when shards are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger shard relocation. These settings are part of the <<modules-cluster,cluster-level shard allocation settings>>.
+
+Shard relocation operations also respect shard allocation and recovery settings. 

docs/reference/modules/shard-recovery-desc.asciidoc

@@ -0,0 +1,13 @@
+Shard recovery is the process of initializing a shard copy, such as restoring a
+primary shard from a snapshot or syncing a replica shard from a primary shard.
+When a shard recovery completes, the recovered shard is available for search
+and indexing.
+
+Recovery automatically occurs during the following processes:
+
+* Node startup or failure. This type of recovery is called a local store recovery.
+* Replication of primary shards to replica shards.
+* Relocation of a shard to a different node in the same cluster.
+* A <<snapshots-restore-snapshot,snapshot restore>> operation.
+* A <<indices-clone-index,clone>>, <<indices-shrink-index,shrink>>, or
+<<indices-split-index,split>> operation.

docs/reference/primaries-replicas-desc.asciidoc

@@ -0,0 +1,5 @@
+There are two types of shards: primaries and replicas. Each document in an index
+belongs to one primary shard. A replica shard is a copy of a primary shard.
+Replicas provide redundant copies of your data to protect against hardware
+failure and increase capacity to serve read requests
+like searching or retrieving a document.

docs/reference/setup.asciidoc

@@ -33,7 +33,6 @@ include::setup/configuration.asciidoc[]
 
 include::setup/important-settings.asciidoc[]
 
-
 include::setup/secure-settings.asciidoc[]
 
 include::settings/audit-settings.asciidoc[]
@@ -82,6 +81,8 @@ include::modules/indices/search-settings.asciidoc[]
 
 include::settings/security-settings.asciidoc[]
 
+include::modules/shard-ops.asciidoc[]
+
 include::modules/indices/request_cache.asciidoc[]
 
 include::settings/snapshot-settings.asciidoc[]
@@ -93,7 +94,9 @@ include::modules/threadpool.asciidoc[]
 include::settings/notification-settings.asciidoc[]
 
 include::setup/advanced-configuration.asciidoc[]
+
 include::setup/sysconfig.asciidoc[]
+
 include::setup/bootstrap-checks.asciidoc[]
 
 include::setup/bootstrap-checks-xes.asciidoc[]