Interval field on DateHistograms will be deprecated

[polyfractal]

Today, Elasticsearch's date_histogram aggregation's interval field supports both "fixed" and "calendar" time simultaneously, choosing between the two depending on the interval's syntax (e.g. 1d is calendar, 2d is fixed). This is super confusing for users even if they read the documentation, and a very subtle bug for everyone else because intervals don't always work as they expect.

Elasticsearch is working on a PR to deprecate this functionality, and replace it with two explicit fields: fixed_interval and calendar_interval. This way the user knows exactly what they are getting.

The plan as it stands today:

• Add fixed_interval / calendar_interval fields to date_histogram aggregation.
• Deprecate interval field in 7.0+
• Remove interval field in 8.0
• All three fields are mutually exclusive
• Changes also affect rollup configurations and composite aggs (since they both use the date_histo)

More details can be found in the PR itself (elastic/elasticsearch#33727) but I wanted to open an issue in Kibana to track the deprecation since it'll affect anything touching a date histo.

Note: we're targeting 7.0, but there's no requirement it has to make 7.0 so it could slip to 7.x

TODOs

• Courier (AggConfigs) (Add dateHistogramInterval utility #39091)
• TSVB ([TSVB] Apply dateHistogramInterval utility #39665)
• Timelion (Use new date_histogram intervals in timelion #77160)
• APM

[elasticmachine]

Pinging @elastic/kibana-app

[polyfractal]

Note: this merged into master, backport for 7.x is in-progress. I'll update when that lands as well.

[elasticmachine]

Pinging @elastic/es-ui

[elasticmachine]

Pinging @elastic/stack-monitoring

[polyfractal]

Final note, elastic/elasticsearch#41906 just merged into 7.x and backporting is now done on the ES side.

[timroes]

Update: We currently have plans to provide a utility function that autodetects this into the new data core plugin, which all teams can use. I'll update here (and via mail) once this is available.

[chrisronline]

I'm not quite sure I understand how this works.

Today, Elasticsearch's date_histogram aggregation's interval field supports both "fixed" and "calendar" time simultaneously, choosing between the two depending on the interval's syntax (e.g. 1d is calendar, 2d is fixed).

You're saying 1d is calendar, but 2d is fixed? I don't know if I understand how the ES code differentiates those two as separate interval types entirely?

[polyfractal]

You're saying 1d is calendar, but 2d is fixed? I don't know if I understand how the ES code differentiates those two as separate interval types entirely?

This is correct. In the code, there's a map of "calendar" units and it literally looks to see if the supplied interval is one of the accepted calendar units (1d, 1w, etc). If it matches, it uses calendar-aware rounding for all the date math.

If the map doesn't contain the specified interval, it is assumed to be "fixed" and tries to parse it with the allowed non-calendar units. If this parsing works, fixed time rounding is used for the rest of the date math.

If that parsing fails, it throws an exception (either because the unit is an illegal fixed time like 2w, or just invalid foobar).

This is why we wanted explicit fields to disambiguate. Having two distinctly different behaviors overlapping with the same set of semi-shared syntax is complicated and confusing to everyone :)

[chrisronline]

@polyfractal

I don't think it's exactly related, but do you know if this interval is also affected by this change?

[polyfractal]

@chrisronline No, I don't think so. Should only affect things that touch the date_histogram interval directly (so the agg, composite agg, and rollups). Looks like that interval is always parsed as fixed time

[timroes]

I will be closing this issue since all linked TODOs are done. If any team still is using interval in their code, please open a separate issue for your team (so we can also have properly assigned team labels), reference this issue, and mark yoru issue as a blocker for 8.0. Thanks everyone.