Controls visualization should use time filter

[shaharmor]

Describe the feature:

Add a configuration option to the controls visualization to only query the last X indices.
When you have a huge number of indices it can take a long time to query all the terms

[epixa]

Can you explain a little more about what you're use case for this is? Beginning in 5.6, Elasticsearch will automatically optimize search requests when there are lots of potential indices to search through (it's technically shard-based), so you shouldn't get much of a performance penalty. Does that address your problem?

[shaharmor]

No it doesn't.

When you add a controls vis, choose "options list", select an index patten, select a field, click play -> it makes a request to ES without any time filter, which means that the search will go to all indices that match the index pattern.

In situations where you have a lot of indices (For example index per day/hour or something), it will take a lot of time for query to return.

An option to limit the time frame the query is using will make ES skip most of the shards

[nreese]

A filter by time configuration option could be added to each control. When this is enabled, the active kibana time range would be applied when fetching terms or min/max for range control. Would this solve your problem?

Would you expect the terms and min/max to update if the kibana time range is changed?

Nathan

[shaharmor]

Yup, exactly.

It would also be good to be able to set hardcoded limits on the date range for the controls vis, because if a user runs a query for the past year or so, I'd prefer to only show him filters from the past week for example and save the load from ES.

[nreese]

@stacey-gammon @kobelb What are your thoughts about setting a hardcoded date range in the controls vis? There is another open issue about allowing dashboard panels to specify a date range independent of the kibana time range.

Is this something that should be implemented specific to input controls or would it be worth pursuing panel specific time ranges to satisfy the issue highlighted by this request?

[stacey-gammon]

I think we should make date ranges configurable per panel, something we want to do anyway, and sounds like it would cover this use case.

[kobelb]

@nreese what was the original reasoning for us having the input controls not respect the global time-filter?

[nreese]

@kobelb Input controls were targeted at non technical users. I wanted to avoid confusion by making the list of gathered terms as static as possible. That way, regardless of kibana state - users were presented with the same list of terms and never got stumped because the term they expected to be there was missing due to a time filter.

[epixa]

That way, regardless of kibana state - users were presented with the same list of terms and never got stumped because the term they expected to be there was missing due to a time filter.

I think that's a reasonable idea, but it has some pretty significant problems from a scaling perspective. As a default behavior, this will not scale from a performance perspective to large data sets nor will it scale from a user experience perspective when dealing with data sets with a large number of fields. Perhaps the right solution here is to default to time based and then consider the ability to opt-out of time ranges on a panel by panel basis as @stacey-gammon mentioned.

[nreese]

#15852 added the option to apply the global time range.

Still to come is the ability to specify a time range per for the control panel

[nreese]

closed by #43153