[BUG] MonitorRunners - SearchRequest missing cancelAfterTimeInterval parameter

[petardz]

What is the bug?
In all monitor runners we don't set cancelAfterTimeInterval parameter on SearchRequest object. Consequence of this is that global cluster setting search.cancel_after_time_interval will override it, which if set too low, could impact monitor run success rate.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Set cluster setting search.cancel_after_time_interval to something very low, like 1s
2. Create monitor on index with some number of documents/shards
3. See Cancellation timeout error (TaskCancelledExceptioncancelled exception)

[rishabhmaurya]

Are you proposing its resolution to be individual plugin's responsibility to set the search.cancel_after_time_interval with max(cluster setting search.cancel_after_time_interval, plugin's default minimum value say 5s) ?

[petardz]

@rishabhmaurya If I understood correctly, cluster setting search.cancel_after_time_interval will be used only if same param in SearchRequest is null?

If that's the case, then plugin can override this setting by setting a cancelAfterTimeInterval param in every SearchRequest before calling _search.

[rishabhmaurya]

I was thinking about the case when search.cancel_after_time_interval is set at cluster with some higher value and the plugin overrides it to a lower value. Thus, max(cluster setting search.cancel_after_time_interval, plugin's default minimum value say 5s) should be used while overriding before calling _search from plugin.
I agree with you that plugins, from where search traffic pattern as a constant function number of index/alerts/other plugins settings, overriding this timeout makes sense. Feel free to raise a PR. Thanks!

[petardz]

I was thinking about the case when search.cancel_after_time_interval is set at cluster with some higher value and the plugin overrides it to a lower value. Thus, max(cluster setting search.cancel_after_time_interval, plugin's default minimum value say 5s) should be used while overriding before calling _search from plugin.

Ah got it. That makes sense!