Add indexing pressure documentation

docs/reference/cluster/nodes-stats.asciidoc

@@ -58,6 +58,10 @@ using metrics.
   `http`::
       HTTP connection information.
 
+  `indexing_pressure`::
+        Indexing pressure statistics about current and total indexing load and
+        indexing rejections.
+
   `indices`::
       Indices stats about size, document count, indexing and deletion times,
       search times, field cache size, merges and flushes.
@@ -2099,6 +2103,62 @@ Number of failed operations for the processor.
 =======
 ======
 
+[[cluster-nodes-stats-api-response-body-indexing-pressure]]
+`indexing_pressure`::
+(object)
+Contains <<index-modules-indexing-pressure,indexing pressure>> statistics for the node.
++
+.Properties of `indexing_pressure`
+[%collapsible%open]
+======
+`total`::
+(object)
+Contains statistics for cumulative indexing load since the node started.
++
+.Properties of `<total>`
+[%collapsible%open]
+=======
+`coordinating_and_primary_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating or primary stage.
+
+`replica_bytes`::
+(integer)
+Bytes consumed by indexing requests in the replica stage.
+
+`all_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating, primary, or replica stage.
+
+`coordinating_and_primary_memory_limit_rejections`::
+(integer)
+Rejections of indexing requests in the coordinating or primary stage.
+
+`replica_memory_limit_rejections`::
+(integer)
+Rejections of indexing requests in the replica stage.
+=======
+`current`::
+(object)
+Contains statistics for current indexing load.
++
+.Properties of `<current>`
+[%collapsible%open]
+=======
+`coordinating_and_primary_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating or primary stage.
+
+`replica_bytes`::
+(integer)
+Bytes consumed by indexing requests in the replica stage.
+
+`all_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating, primary, or replica stage.
+=======
+======
+
 [[cluster-nodes-stats-api-response-body-adaptive-selection]]
 `adaptive_selection`::
 (object)

docs/reference/docs/data-replication.asciidoc

@@ -20,12 +20,15 @@ responsible for replicating the operation to the other copies.
 This purpose of this section is to give a high level overview of the Elasticsearch replication model and discuss the implications
 it has for various interactions between write and read operations.
 
-[float]
+[float,id="basic-write-model"]
 ==== Basic write model
 
 Every indexing operation in Elasticsearch is first resolved to a replication group using <<index-routing,routing>>,
-typically based on the document ID. Once the replication group has been determined,
-the operation is forwarded internally to the current _primary shard_ of the group. The primary shard is responsible
+typically based on the document ID. Once the replication group has been determined, the operation is forwarded
+internally to the current _primary shard_ of the group. This stage of indexing is referred to as the coordinating
+stage.
+
+The next stage of indexing is the primary stage which is performed on the primary shard. The primary shard is responsible
 for validating the operation and forwarding it to the other replicas. Since replicas can be offline, the primary
 is not required to replicate to all replicas. Instead, Elasticsearch maintains a list of shard copies that should
 receive the operation. This list is called the _in-sync copies_ and is maintained by the master node. As the name implies,
@@ -42,6 +45,14 @@ The primary shard follows this basic flow:
 . Once all replicas have successfully performed the operation and responded to the primary, the primary acknowledges the successful
    completion of the request to the client.
 
+Each in-sync replica copy performs the indexing operation locally so that it has a copy. This stage of indexing is the replication
+stage.
+
+These indexing stages (coordinating, primary, and replication) are sequential. However, each upstream stage is inclusive of the
+downstream stages. The coordinating stage is not complete until each primary stage (which might be spread out across different
+primary shards) has completed. Each primary stage will not complete until the in-sync replicas have completed replication and
+responded to the replication requests.
+
 [float]
 ===== Failure handling
 

docs/reference/index-modules/indexing-pressure.asciidoc

@@ -0,0 +1,60 @@
+[[index-modules-indexing-pressure]]
+== Indexing Pressure
+
+Indexing documents into Elasticsearch introduces system load in the form of
+memory and CPU load. Each indexing operation includes coordinating, primary, and
+replica stages. These stages can be performed across multiple nodes in the
+cluster. If too much indexing work is introduced into the system, the cluster
+can become saturated. This can adversely impact other components of the system
+such as searches, cluster coordination, background processing, etc.
+
+Indexing pressure is primarily generated by external operations such as indexing
+requests or internally by mechanisms such recoveries and cross-cluster
+replication.
+
+Elasticsearch internally monitors indexing load. When the load exceeds
+certain limits, new indexing work will be rejected.
+
+[float]
+=== Indexing Stages
+
+External indexing operations go through three stages: coordinating, primary, and
+replication. This write model is explain <<basic-write-model,here>>.
+
+[float]
+=== Memory Limits
+
+Elasticsearch exposes a node setting `indexing_pressure.memory.limit` which
+restricts the number of bytes for outstanding indexing requests. Be default,
+this setting is configured to be 10% of the heap.
+
+At the beginning of each <<indexing stage,here>>, Elasticsearch accounts for the
+bytes consumed by an indexing request. This accounting is only released at the
+end of the indexing stage. This means that upstream stages will account for the
+request overheard until all downstream stages are complete. For example, the
+coordinating request will remain accounted for until primary and replication
+stages are complete. The primary request will remain accounted for until the
+replication stage is complete.
+
+A node will start rejecting new indexing work at the coordinating or primary
+stage when the number of outstanding coordinating, primary, and replica indexing
+bytes are greater than the configured limit.
+
+A node will start rejecting new indexing work at the replication stage when the
+number of outstanding replication indexing bytes are greater than 1.5x the
+configured limit. This design means that as indexing pressure builds on nodes,
+they will naturally stop accepting coordinating and primary work in favor of
+outstanding replication work.
+
+The default limit `indexing_pressure.memory.limit` (10%) is generously sized and
+should only be modified after careful consideration. Only indexing requests
+contribute to this limit meaning that there is additional indexing overhead
+(buffers, listeners, etc) which also require heap space. Finally, other
+components of Elasticsearch also require memory. Configuring this limit to be
+too high can starve other system components of operating memory.
+
+[float]
+=== Monitoring
+
+Indexing pressure metrics are exposed by the
+<<cluster-nodes-stats-api-response-body-indexing-pressure,Node Stats API>>.