Improve robustness of monitoring APIs

[imotov]

We observed some cases (#50241 for example) where a data node responding slowly can cause accumulation of ResponseContexts for indices:monitor/recovery[n], indices:monitor/stats[n], cluster:monitor/stats[n] and cluster:monitor/xpack/ml/job/stats/get[n] which correspond to _xpack/usage and _nodes/stats calls.

We would like to improve robustness of stats and usage call in case of a slowly responding data nodes by

1. introducing timeout on stats and usage APIs and/or
2. making stats and usage APIs tasks cancellable and cancel them if the REST client disconnects

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Monitoring)

[imotov]

Just an additional data point. It looks like the metricbeat's elaticsearch module has 10s timeout. So if we don't deliver stats in 10 seconds, it doesn't care anymore.

[DaveCTurner]

Option 2 means that clients can safely implement their own end-to-end timeouts, which means we don't need option 1; conversely if we do option 1 then that still requires clients to implement their own timeouts to protect against requests getting lost on the way to Elasticsearch. I therefore prefer option 2 alone.

Relates #60188 which is pretty much the same issue but for the transport-like client that the internal monitoring system uses, which wouldn't be solved by cancelling tasks on a client disconnection since the internal client doesn't disconnect like that.

Also relates #51992.

[dhwanilpatel]

Hello Elastic team,

Is team working on this or expecting any community contribution?
If team is already working, Can you please provide any timeline for it? It will be very helpful. Thanks!

[DaveCTurner]

Copying here the info that @Bukhtawar intends to work on cancellation of the various stats APIs, see #66992 (comment).

@Bukhtawar you might be interested in #67413 which implements the right sort of behaviour on a different API.

[DaveCTurner]

As of 7.13 the following APIs can be cancelled by a client-side timeout:

• GET _cluster/state
• GET _cluster/stats
• GET _stats
• GET _segments
• GET _cat/segments
• GET _recovery
• GET _cat/recovery

I spoke with the ML folks and we concluded that cluster:monitor/xpack/ml/job/stats/get shouldn't cause an excess of junk on the coordinating node since the per-node responses are pretty small; if we saw them building up it was probably due to something else clogging up the MANAGEMENT threads.

I also took a quick look at GET _xpack/usage and didn't see anything that would cause problems: this is a TransportMasterNodeAction so it doesn't accumulate responses from multiple nodes, and also the response appears to be fairly small.

I believe that's all the APIs that need to be addressed for this issue so I am closing this.