Ability to align date histograms with current time automatically

[timroes]

tl;dr: Having buckets in date histogram like now to now-interval, now-interval to now-2*interval, and so on would be very nice!

If you use Kibana to monitor your values, you will often have a (let's say) line diagram that shows a value over time and you are interested if the value suddenly begins to change. If the date histogram has buckets of (again let's assume) 30 minutes, you might end up with a diagram as the following:

Let's assume that screenshot was taken at some hour and 14 minutes. Since the date histogram uses intervals of 30 minutes the last bucket is not yet complete and therefor has only half the value then the ones before had, even though that doesn't indicate any problem in the data itself. It's just not there yet.

Kibana correctly highlights this and outputs warnings about it. Unfortunately it makes it really hard to use that kind of graphs on dashboards for live monitoring, because you would see if something wasn't okay 30 minutes delayed, i.e. after the last bucket is completed and you can see if it really has an "OK" value (whatever that mean in your case).

For the count of documents (as in this example) it's still quite easy to recognize, since at x:15 o'clock the value of the last bucket should be around the half then the ones before (if you aim for continuity). But if you use other metric aggs like averages, sums, etc. it might be very hard or impossible to spot if the value is okay before the bucket is complete, making the diagram always 30 minutes delayed before you can spot any problems.

What could the solution be? A possible solution would be, if the buckets are not aligned to "round" values like :00 and :30 (in case of 30 minutes buckets), but they could be aligned to 30 minutes with the last bucket always be complete.

What the hell are you talking about? If we look at the above diagram and assume it is 10:14 o'clock, the last bucket shouldn't be from 10:00 to 10:30, but from 9:44 to 10:14, the one before from 9:14 to 9:44, and so on... That way your diagram would get meaningful again - only the first bucket in the diagram, which might be "long in the past" is not complete anymore and might not have meaningful data, what you may not care about, if you are using Kibana for live monitoring.

But how? Elasticsearch's date histogram aggregation supports an offset parameter. If you set this to minus the time we are in the not yet complete bucket, you will achieve the desired result. I.e. if we have 10:14, just set the offset to -14m which will shift the buckets, so that we achieve exactly the above described bucket situation:

That way the visualization would be usable for live monitoring.

Long story short, what's my idea? I would like to have an option in the date histogram aggregation for "aligning buckets to now" (yeah I haven't come up with a better name yet), which will cause Kibana to automatically calculate the offset it needs to apply to the date histogram aggregation every time data for that diagram is fetched - and of course applies it as an offset to the aggregation.

PS: I've tried to find if there is already an issue for that, but didn't find anything closely related. But please correct me if there is already an issue for that.

[Bargs]

++ would be very handy

[timroes]

Is addressed in TSVB via #15760

[timductive]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.