[ML] Datafeed with aggregations that filter can go into an infinite loop

[droberts195]

When the ability to use aggregations was first added to datafeeds there was an assumption that the datafeed's query would be used to filter the input while its aggregations would simply group and summarise the input. As time has gone by, users are using more and more complex aggregations that also do filtering within the aggregation. Examples of sub-aggregations that could do this are filter and bucket_selector.

Datafeeds have functionality quickly skip periods of time when there is no data at all. They do this by adding a simple aggregation that gets the min and max timestamp onto the query from the datafeed config, and running this over the period from the last seen data to the datafeed's configured end time.

Unfortunately, the combination of these two things can cause a datafeed to go into an infinite loop. This happens if the following conditions all hold:

1. The datafeed is using aggregations
2. The datafeed has chunking enabled
3. The datafeed's aggregations filter data to a greater extent than its query
4. There is a particular chunk of time (as defined by the chunking_config) where data exists within the first bucket_span of that chunk of time that matches the query, but after the aggregations have been applied the entire chunk of time returns empty aggregations

In this scenario the datafeed will "skip" time back to the start of the current chunk, and hence go into an infinite loop.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)