Removing of implicit value scaling for time buckets

[timroes]

Kibana currently has some implicit value scaling. This works as following:

If a user selects a manual time range, like "minutes" or "hours" for their date histogram time interval, Kibana might detect that given the overall time range this would create too many buckets (more then histogram:maxBuckets setting). In that case Kibana uses a larger interval range instead, e.g. if you set "minutes" as interval and request to see data of the last 7 days we might actually request 30 minutes buckets, so we're not overflowing the chart with buckets. So far so good, now to the actual weird part.

If we scale your buckets up like this, we'll actually downscale the value shown on the y-axis on the chart (if it is a sum or a count) so it's shown "per minute" again. Meaning in the above case you got a bucket with a count of 15 in it, the chart will actually show 0.5 for it, since 15 / 30 minutes = 0.5 documents per minute.

This behavior turned out to be very confusing to users and we frequently see questions about that popping up. So I would like to suggest deprecating that feature and removing it in the next major version. In the long run I would suggest, that we instead build an explicit way in a chart (see #2010), that allows to configure showing your values "by seconds" or "by minutes" instead. That explicit way could pretty much solve the same issue than the implicit behavior, but would be less confusing.

Even if we don't build that explicit way until then I would suggest removing the implicit value scaling for now, since it's more confusing that beneficial. For clarification: I don't suggest removing scaling the bucket size to larger buckets (we still need to do that, to not overflow the chart), but only to show the real values in the chart and don't implicitly scale them down anymore.

cc @AlonaNadler @ppisljar

[elasticmachine]

Pinging @elastic/kibana-app

[AlonaNadler]

@timroes agree that the implicit value is not helping

It creates weird results where for example even though the metric is count there are a fraction on the y-axis.

Any idea why we did the implicit value in the first place?

[timroes]

It was created so that users can actually show something like "requests per second". If the user specify "seconds" manually as an interval we would just assume they are interested in the value per seconds and scale the value down to that. Though there are several use-cases where that's not true, and in general just confusing.

[cawoodm]

This is presented as a kind of oddity but it's a serious bug: charts are showing different averages depending on the level of zoom which results in false data in many (if not all) cases. If I have data every 5 min with an hourly average of say 50. It should then not matter whether I have buckets by hour or by 3 hours: the hourly average is 50 and should be displayed thusly.

https://discuss.elastic.co/t/average-changes-when-zoomed/204598

[timroes]

I totally agree. Also we basically mix querying concerns (how large will the buckets be) with presentation/styling concerns (I want to see kB per second not per bucket -- which is what most people that are using this, using this for).

With 7.5 we're removing the behavior by default in newly created visualizations, but leave the user the choice under advanced settings to enable this (and vice versa, disable it for already created visualizations). We're trying to implement the styling concern as soon as possible in Lens to have the styling functionality available before we'll remove this mixed behavior altogether in a upcoming major version.

[ghudgins]

plan is to leave this in aggregation based visualizations for now. No plans to migrate to lens for this setting as it is deprecated