Line visualization is messed up with Kibana 7.2.1

[Ron-70]

Kibana version: 7.2.1

Elasticsearch version: 7.2.1

Describe the bug:
The Line visualization in Kibana 7.2.1 is messed up.

I compared the Line visualization between Kibana 7.1.1 and 7.2.1. Based on the same elasticsearch data and the same "Saved Objects" used in 7.1.1. and in 7.2.1, the 7.2.1 Line visualization is totaly wrong according to the shown chart / values. See screenshots. Additionally if you zoom more into the chart in Kibana 7.2.1 then the values change and come more closer to the correct values.

It seems in 7.2.1 the real correct values (e.g. 1167 in the screenshot example) are divided by the current date interval (in screenshot example 30 min) and then shown as chart value (38.9). Thats very strange and wrong.

So at the moment Kibana 7.2.1 is unusable regarding Line visualizations.

Expected behavior: Correct Line visualization as in Kibana 7.1.1

Screenshots (if relevant):

[elasticmachine]

Pinging @elastic/kibana-app

[Ron-70]

The same problem occurs on the just released 7.3.0 version too.

[timroes]

Hi Ron,

that indeed looks like a bug. Kibana scales down values if you specify a manual interval that's smaller then what we actually use, and it seems this got triggered there without a reason. To reproduce that properly, could you please give us the configuration of your chart. Ideally just export the Saved Object of that chart from Management > Saved Object and upload this here. If there are sensitive field names or index names in that, feel free to blank them (but please mark what you changed manually in the saved object).

Also one more information: Could you give us a screenshot from the x-axis aggregation for that chart in the editor, i.e. what field and interval was selected for that x-axis (this should also be in the saved object, I just want to crosscheck if it's aligned with what you see in the editor).

Thanks a lot,
Tim

EDIT Just for posterity, my current suspicion is that removing customInterval might have caused the scaling logic to break, but we'll need to debug.

[Ron-70]

Hi @timroes,

here is a screenshot showing the x-axis parameters and the exported saved object of this visualization as zip archive. Both are from a Kibana 7.3.0 system.

heapchart.ndjson.zip

Best regards,
Ron

[Ron-70]

Hi,

I also add a screenshot of the chart and the related JSON response from elasticsearch. So the correct value (1167) is in the response but the chart is wrong.

[timroes]

Okay this is interesting, because 7.2 actually shows the correct (but confusing, I'll come to that later) behavior, and 7.1 seems to be broken, though I never have seen any other reports on that.

Let me shortly elaborate what's happening here. If you specify a manual interval (i.e. not auto) in a date histogram, this interval would maybe cause Kibana to create too many buckets (more then the histogram:maxBucket advanced setting), depending on the time range you're looking at. In your example you are looking at an overall time range of 4 days and requested an interval of "minutes". That would cause too many buckets, so Kibana decided to request a larger interval (in your case 30 minutes, as we can see in the x-axis label). BUT - now the magical part - since you selected "minute" manually, we assume you really want the value "per minute" and not "per 30 minutes", why Kibana actually divides the value by 30 (to scale from 30 minutes down to the minute you requested), and show you that value instead. So you actually only got one point on the x-axis per 30 minutes, but the value shown on the y-axis is scaled to "per minute".

So the 7.2 behavior is the one we would currently expect to see. I cannot tell you (and cannot recreate) why 7.1 did behave differently for you. Did you maybe not have "minute" selected that time, but "auto" Or were you looking at a smaller overall time range (in which case we might not have needed to apply scaling).

Nevertheless this behavior seems to confuse a lot of users, and we already talked about that a couple of time, and I think I will open a separate discussion issue for that tomorrow, to discuss if it makes more sense to just remove that scaling behavior.

Does this provide enough context for explaining that behavior for you?

[Ron-70]

Hi @timroes,

thanx for this explanation. But for me the Kibana >= 7.2 behavior seems faulty because you modify the values of the y-axis based on the time interval on x-axis. This makes no sense to me.

In my example I display the java Heap memory value over time. So in the current Kibana versions this heap memory value is modified and that's simply wrong. The heap memory was 1167 MB and not 38.9 MB at the shown point of time on the x-axis.

I used the same elasticsearch data and imported the same "Saved objects" into 7.1.1 and 7.2.1 / 7.3. So the chart parameters were equal also regarding the interval setting (Minute).

Here is a screenshot of the 7.1.1 x-axis parameters of this chart. The heap memory values are shown correctly as expected:

[timroes]

I agree that the behavior might be confusing, why I mentioned to discuss whether we remove this. The behavior in 7.2 is nevertheless the current expected behavior and the 7.1 was broken for some reason. Since the behavior is in recent Kibana versions working as expected I'll close this issue (and will later refer to the discussion issue about removing the time scaling in general).

[Ron-70]

Hi @timroes,

I just noticed also a small difference regarding the interval parameter naming in 7.1.1 and 7.3.0.
In 7.1.1 the parameter is just called "Interval" and in 7.3.0 "Minimum interval". So regarding this naming I would expect that in 7.3.0 it's like a kind of given minimum possible resolution. In my case "Minutes". So the values shown in 7.3.0 are wrong considering this resolution context.

[timroes]

Reopening this, since @flash1293 pointed out to me, that indeed this is a top hits metric, and despite my explanation for how we scale value, that actually should only happen for visualizations where it makes sense (currently Count and Sum), and not for any other metric. So indeed this is a bug, that the scale is applied to the "top hits" metric here. (And will still open an issue to discuss that behavior for Sum/Count)

[elasticmachine]

Pinging @elastic/kibana-app-arch

[timroes]

The root cause for this issue is, that the esaggs function instantiates AggConfigs without passing in the third parameter schemas to the constructor. It might also be very tricky passing this in, since by concept esaggs should not have any knowledge about the vis.type itself which is the one holding the schemas. But without schemas the following code in date_histogram is not able to properly does the scaling logic.

I checked the code and it seems this functionality is the only one that requires AggConfigs to have any knowledge about the schemas of the vis, meaning without that scaling we could remove that schemas link (that doesn't have any valid place in esaggs anymore.

@lukeelmers @ppisljar we need to sync about how we could fix this issue. Currently as long as we upscale your buckets to larger time buckets, we will ALWAYS scale down all metrics no matter which kind, which is wrong behavior.

[timroes]

@Ron-70 Thanks for opening this issue. We have a fix on the way for one of the next patch versions. You should be able to get this temporarily fixed by specifying interpreter.enableInVisualize: false in your kibana.yml. This will switch over to our old internal visualization rendering system, which does not have that issue. You should not have any (other) visible differences in Kibana, by setting this in your config.

[Ron-70]

Hi,

thanx for the fast reaction. I will wait for the next patch version.

[timroes]

I know also created #42611 to discuss the general removal of that feature.