Add indexing pressure documentation

docs/reference/cluster/nodes-stats.asciidoc

@@ -58,6 +58,9 @@ using metrics.
   `http`::
       HTTP connection information.
 
+  `indexing_pressure`::
+        Statistics about the node's indexing load and related rejections.
+
   `indices`::
       Indices stats about size, document count, indexing and deletion times,
       search times, field cache size, merges and flushes.
@@ -2099,6 +2102,82 @@ Number of failed operations for the processor.
 =======
 ======
 
+[[cluster-nodes-stats-api-response-body-indexing-pressure]]
+`indexing_pressure`::
+(object)
+Contains <<index-modules-indexing-pressure,indexing pressure>> statistics for the node.
++
+.Properties of `indexing_pressure`
+[%collapsible%open]
+======
+`total`::
+(object)
+Contains statistics for the cumulative indexing load since the node started.
++
+.Properties of `<total>`
+[%collapsible%open]
+=======
+`combined_coordinating_and_primary_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating or primary stage. This
+value is not the sum of coordinating_bytes and primary_bytes as a node can reuse
+the coordinating bytes if the primary is executed locally.
+
+`coordinating_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating stage.
+
+`primary_bytes`::
+(integer)
+Bytes consumed by indexing requests in the primary stage.
+
+`all_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating, primary, or replica stage.
+
+`coordinating_rejections`::
+(integer)
+Number of indexing requests rejected in the coordinating stage.
+
+`primary_rejections`::
+(integer)
+Number of indexing requests rejected in the primary stage.
+
+`replica_rejections`::
+(integer)
+Number of indexing requests rejected in the replica stage.
+=======
+`current`::
+(object)
+Contains statistics for current indexing load.
++
+.Properties of `<current>`
+[%collapsible%open]
+=======
+`combined_coordinating_and_primary_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating or primary stage. This
+value is not the sum of coordinating_bytes and primary_bytes as a node can reuse
+the coordinating bytes if the primary is executed locally.
+
+`coordinating_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating stage.
+
+`primary_bytes`::
+(integer)
+Bytes consumed by indexing requests in the primary stage.
+
+`replica_bytes`::
+(integer)
+Bytes consumed by indexing requests in the replica stage.
+
+`all_bytes`::
+(integer)
+Bytes consumed by indexing requests in the coordinating, primary, or replica stage.
+=======
+======
+
 [[cluster-nodes-stats-api-response-body-adaptive-selection]]
 `adaptive_selection`::
 (object)

docs/reference/docs/data-replication.asciidoc

@@ -20,12 +20,15 @@ responsible for replicating the operation to the other copies.
 This purpose of this section is to give a high level overview of the Elasticsearch replication model and discuss the implications
 it has for various interactions between write and read operations.
 
-[float]
+[discrete]
+[[basic-write-model]]
 ==== Basic write model
 
 Every indexing operation in Elasticsearch is first resolved to a replication group using <<index-routing,routing>>,
-typically based on the document ID. Once the replication group has been determined,
-the operation is forwarded internally to the current _primary shard_ of the group. The primary shard is responsible
+typically based on the document ID. Once the replication group has been determined, the operation is forwarded
+internally to the current _primary shard_ of the group. This stage of indexing is referred to as the _coordinating stage_.
+
+The next stage of indexing is the _primary stage_, performed on the primary shard. The primary shard is responsible
 for validating the operation and forwarding it to the other replicas. Since replicas can be offline, the primary
 is not required to replicate to all replicas. Instead, Elasticsearch maintains a list of shard copies that should
 receive the operation. This list is called the _in-sync copies_ and is maintained by the master node. As the name implies,
@@ -42,6 +45,14 @@ The primary shard follows this basic flow:
 . Once all replicas have successfully performed the operation and responded to the primary, the primary acknowledges the successful
    completion of the request to the client.
 
+Each in-sync replica copy performs the indexing operation locally so that it has a copy. This stage of indexing is the
+_replica stage_.
+
+These indexing stages (coordinating, primary, and replica) are sequential. To enable internal retries, the lifetime of each stage
+encompasses the lifetime of each subsequent stage. For example, the coordinating stage is not complete until each primary
+stage, which may be spread out across different primary shards, has completed. Each primary stage will not complete until the
+in-sync replicas have finished indexing the docs locally and responded to the replica requests.
+
 [float]
 ===== Failure handling
 

docs/reference/index-modules.asciidoc

@@ -281,6 +281,10 @@ Other index settings are available in index modules:
 
     Control over the retention of a history of operations in the index.
 
+<<index-modules-indexing-pressure,Indexing pressure>>::
+
+    Configure indexing back pressure limits.
+
 [float]
 [[x-pack-index-settings]]
 === [xpack]#{xpack} index settings#
@@ -311,3 +315,5 @@ include::index-modules/translog.asciidoc[]
 include::index-modules/history-retention.asciidoc[]
 
 include::index-modules/index-sorting.asciidoc[]
+
+include::index-modules/indexing-pressure.asciidoc[]

docs/reference/index-modules/indexing-pressure.asciidoc

@@ -0,0 +1,75 @@
+[[index-modules-indexing-pressure]]
+== Indexing pressure
+
+Indexing documents into Elasticsearch introduces system load in the form of
+memory and CPU load. Each indexing operation includes coordinating, primary, and
+replica stages. These stages can be performed across multiple nodes in the
+cluster. If too much indexing work is introduced into the system, the cluster
+can become saturated. This can adversely impact other components of the system
+such as searches, cluster coordination, background processing, etc.
+
+Indexing pressure is primarily generated by external operations such as indexing
+requests or internally by mechanisms such recoveries and cross-cluster
+replication.
+
+Elasticsearch internally monitors indexing load. When the load exceeds
+certain limits, new indexing work will be rejected.
+
+[discrete]
+[[indexing-stages]]
+=== Indexing stages
+
+External indexing operations go through three stages: coordinating, primary, and
+replica. See <<basic-write-model,here>>.
+
+[discrete]
+[[memory-limits]]
+=== Memory limits
+
+The `indexing_pressure.memory.limit` node setting restricts the number of bytes
+available for outstanding indexing requests. This setting defaults to 10% of
+the heap.
+
+At the beginning of each indexing stage, {es} accounts for the
+bytes consumed by an indexing request. This accounting is only released at the
+end of the indexing stage. This means that upstream stages will account for the
+request overheard until all downstream stages are complete. For example, the
+coordinating request will remain accounted for until primary and replica
+stages are complete. The primary request will remain accounted for until each
+in-sync replica has responded to enable replica retries if necessary.
+
+A node will start rejecting new indexing work at the coordinating or primary
+stage when the number of outstanding coordinating, primary, and replica indexing
+bytes exceeds the configured limit.
+
+A node will start rejecting new indexing work at the replica stage when the
+number of outstanding replica indexing bytes exceeds 1.5x the configured limit.
+This design means that as indexing pressure builds on nodes, they will naturally
+stop accepting coordinating and primary work in favor of outstanding replica
+work.
+
+The default limit `indexing_pressure.memory.limit` (10%) is generously sized and
+should only be modified after careful consideration. Only indexing requests
+contribute to this limit meaning that there is additional indexing overhead
+(buffers, listeners, etc) which also require heap space. Finally, other
+components of Elasticsearch also require memory. Configuring this limit to be
+too high can starve other system components of operating memory.
+
+[discrete]
+[[indexing-pressure-monitoring]]
+=== Monitoring
+
+You can use the
+<<cluster-nodes-stats-api-response-body-indexing-pressure,node stats API>> to
+retrieve indexing pressure metrics.
+
+[discrete]
+[[indexing-pressure-settings]]
+=== Indexing pressure settings
+
+`indexing_pressure`::
+
+  A configurable limit for the number of outstanding bytes consumed by indexing
+  requests. New coordinating and primary operations will start being rejected
+  once this limit is hit. New replica operations will start being rejected when
+  replica operations consumed 1.5X this limit. Defaults to `10%` of the heap.