Improve pending indexing metrics and back pressure

[Tim-Brooks]

Currently indexing back pressure is limited to the size of the write queue. This does not effectively reflect the amount of outstanding indexing work for a node. We would like to add new mechanisms which better reflect the amount of outstanding work.

Target 7.9

Indexing metrics and back pressure

In 7.9 we are adding metrics about the number of indexing request bytes outstanding at each point in the indexing process (coordinating, primary, and replication). These metrics will be exposed in the node stats API. Additionally, we will introduce a new setting indexing_pressure.memory.limit which allows a maximum number of bytes to be outstanding. This setting will be 10% of the heap by default. Once 10% of a node's heap is consumed by outstanding indexing bytes, we will start rejecting new coordinating and primary requests.

Additionally, since a failed replication operation can fail a replica, we will assign 1.5X limit for the number of replication bytes. Additionally, only replication bytes can trigger this limit. So if replication bytes increase to high levels, the node will stop accepting new coordinating and primary operations until the replication work load has dropped.

• Add metrics about outstanding indexing request bytes at the coordinating, primary, and replica levels (bulk shard, bulk operations, resync, etc) (Add memory tracking to queued write operations #57573)
• Use metrics to reject new operations when too many bytes are outstanding (Implement rejections in WriteMemoryLimits #58885)
• Expose metrics to node stats API (Adding indexing pressure stats to node stats API #59247)
• Add tests at the REST layer (Adding indexing pressure stats to node stats API #59247)
• Increase write queue size (Increase default write queue size #59464)
• Update release docs with write queue size change (Increase default write queue size #59464)
• Consider separating coordinating and primary bytes at the metrics level (Separate coordinating and primary bytes in stats #59487)
• Bring Increase default write queue size #59464 to 8.0 without release docs (Increase default write queue size #59559)
• Add documentation (Add indexing pressure documentation #59456)
• Add unit level tests to ensure metrics and rejection logic is correct (Fix indexing pressure replica rejections logic #60150)

7.9 Node stats API with human readable enabled

      "indexing_pressure": {
        "memory": {
          "current": {
            "combined_coordinating_and_primary": "0b",
            "combined_coordinating_and_primary_in_bytes": 0,
            "coordinating": "0b",
            "coordinating_in_bytes": 0,
            "primary": "0b",
            "primary_in_bytes": 0,
            "replica": "0b",
            "replica_in_bytes": 0,
            "all": "0b",
            "all_in_bytes": 0
          },
          "total": {
            "combined_coordinating_and_primary": "8.1kb",
            "combined_coordinating_and_primary_in_bytes": 8325,
            "coordinating": "8.1kb",
            "coordinating_in_bytes": 8325,
            "primary": "10.4kb",
            "primary_in_bytes": 10725,
            "replica": "0b",
            "replica_in_bytes": 0,
            "all": "8.1kb",
            "all_in_bytes": 8325,
            "coordinating_rejections": 0,
            "primary_rejections": 0,
            "replica_rejections": 0
          }
        }
      }

Replication Retries

In order to mitigate the potential of transient disruptions failing a replica, we will enable replication retries at the primary level. When an operation fails because of connection error, circuit breaking, rejected, etc we the primary will retry until the new timeout setting is exhausted (indices.replication.retry_timeout).

• Enable replication retries (Retry failed replication due to transient errors #55633)
• Add documentation.

Target 7.10

• Evaluate mechanisms for back presssure related to the CPU cost of indexing

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/CRUD)

[AnthonyFoiani-at]

Hi! I'm curious if the related CPU-based enhancement ever landed in 7.10?

Target 7.10: Evaluate mechanisms for back presssure related to the CPU cost of indexing

[howardhuanghua]

Hi @tbrooks8, could I know that why do we set indexing_pressure.memory.limit as static? Could users change it dynamically for different scenarios?