exporter failed to get nodes data

[lechen26]

Hi

i'm using the elastic-exporter to fetch data from our elastic.co clusters.
it used to work properly but suddenly i see a lot of errors on the exporter log:
level=warn ts=2020-09-01T16:07:17.58499204Z caller=nodes.go:1865 msg="failed to fetch and decode node stats" err="HTTP Request failed with code 400"

when looking on the exporter metrics as http://localhost:9114 i dont see any metrics of nodes (such as elasticsearch_indices_indexing_index_total).

what can be the reason for this annoying error?

the exporter command on the deployment looks like the following

- command:
    - /bin/elasticsearch_exporter
    - --es.uri=$(ES_HOST)
    - --es.all
    - --es.cluster_settings
    - --es.shards

both staging and production clusters are 7.7.1.
staging cluster works as usual and we see all metrics, while prod not.

this 400 error can be related to overhead on the ES side?
how can i investigate it further?

10x

[Freyert]

Given the log output it would be coming from this function.

https://github.com/justwatchcom/elasticsearch_exporter/blob/bbc32de9b976774d2003a65d7fad7d4f5dfb423b/collector/nodes.go#L1801-L1837

Which could be either of these two paths:

https://github.com/justwatchcom/elasticsearch_exporter/blob/bbc32de9b976774d2003a65d7fad7d4f5dfb423b/collector/nodes.go#L1806-L1810

Potential Related ElasticSearch Tickets

• IllegalArgumentException from node stats elastic/elasticsearch#56739