elasticsearch_thread_pool_rejected_count no longer returns values (other than 0)

[shawnclarke7]

We are using elasticsearch-exporter 1.3.0 with elasticsearch 7.17.6
As of elasticsearch 7.9, elasticsearch_thread_pool_rejected_count is no longer a valid metric which we used as a leading indicator of load on the elasticsearch cluster. Elastic Support suggested that a possible replacement would be to use indexing_pressure.memory.current.combined_coordinating_and_primary_in_bytes and indexing_pressure.memory.current.all_in_bytes as alternatives however these values do not seem to be exposed.
These values are available per node in the elasticsearch node stats API

Here is a quote from Elastic Support:

Starting in Elasticsearch 7.9, we’ve increased Elasticsearch ingestion throughout by removing the fixed ingestion queue of 200 documents and replacing it with a new flexible memory-based queue. Through our internal testing, we’ve been able to drastically improve ingestion throughput in Elasticsearch with this new ingestion subsystem, especially when there are large numbers of clients writing data to Elasticsearch.

At each stage of the indexing process (coordinating, primary, and replication), the new ingestion subsystem accounts for the number of bytes associated with an operation and rejects requests if a byte-based limit is exceeded (10% of heap memory by default). This feature is enabled by default in Elasticsearch 7.9, and detailed indexing information is exposed via the node stats API (see the ‘indexing_pressure’ sections).

Since this change "" will basically never return a value other than 0 therefore we are looking for a different metric to act as a leading indicator to load on the system outside of using the built-in rules in Kibana. Elastic Support offered this suggestion:

According to our ES Devs, both are important and you could set your desired threshold when monitoring all_in_bytes.

• indexing_pressure.memory.current.combined_coordinating_and_primary_in_bytes:
  reports the number of bytes of heap needed to hold all the documents in the coordinating and primary stages

• indexing_pressure.memory.current.all_in_bytes:
  reports bytes from all stages of indexing (coordinating, primary, and replica)

Additionally, indexing_pressure.memory.limit_in_bytes would be helpful to have too.

If these values are already supported, please let me know in which version of elasticsearch-exporter and what they are called. If they are not currently exposed, could you please add them?

[shawnclarke7]

@sysadmind Hey Joe, should I assume that by adding the feature request flag that confirms that the above metrics are not currently exposed?

[sysadmind]

That is correct. We don't have any metrics for indexing_pressure today.