[APM] Latency correlation: Display correlation value in relative or similarly intuitive representation

[formgeist]

Summary

Moved over from meta issue: #102499

Can we display the correlation value in an indicator like the impact indicator we have on the existing correlations feature? We could opt to show the actual value upon hover, or simply state it next to it in the table.

It's my understanding the correlation value can range from -1 to 1.

One proposal would be to use a visual indicator like the impact bar that APM uses in other views. It's a relative presentation in percentage. The range would be -1 (0%), 0 (50%) and 1 (100%).

The absolute correlation value could be displayed next to the indicator or in a tooltip for context.

Any other suggestions or thoughts about how to make the presentation of correlations more intuitive than the just the absolute value?

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[formgeist]

@sophiec20 @stevedodson @cyrille-leclerc Thoughts on this? We discussed this earlier in our sync on how to present the correlation value more intuitively.

[cyrille-leclerc]

There may be a risk of misunderstanding using a percentage as it could be understood as the percentage of slow transactions that have this field value.
Otherwise, I like the idea to have a purely positive scale. Maybe 0% to 100% or 0 to 10.
A range from -1 to +1 is hard to understand for me.

[stevedodson]

The range for slow values will be 0 to +1 (we will never display a negative value). The issue is we are scoring how much the field shifts a distribution to the right, and the value we use summarises this calculation.

Framing this as a percentage doesn't make sense as (for instance) 10% doesn't mean that the distribution is 10% 'slower'. Introducing a new value like percentage of transactions > 95% is also problematic, as a field may dominate latency in the 85-92% percentile and have a zero in percentage of transactions > 95%.

Fundamentally, the score is a Pearson Correlation coefficient and summarises how 'slow' the field makes the transactions well. Statistics can be harder for people to understand, but there isn't a simpler score we can use without issues.

My vote is to use only the correlation coefficient and describe and document it well. Fundamentally, the visuals are the best description, and now we should return only fields that make sense.

[sophiec20]

My preference is to leave as 0-1 for 7.14 (and until we have a worthwhile alternative). We should add a tool tip to explain what it is and its range.

Copying from another thread

If someone knows what correlations are, they would expect 0 - 1 and if they didn't I don't think it matters so I think just using the raw values seems sensible

[cyrille-leclerc]

Thanks, sounds good to me. I would then benefit in a kind of legend on the screen to help user understand that the range is 0-1.

[lcawl]

I've drafted a tooltip in walterra#3 for now. Will open another PR to go into more detail when we revamp the info in https://www.elastic.co/guide/en/kibana/master/correlations.html

[qn895]

Closing via #99905