[APM] Show alert annotations on charts

[sorenlouv]

Blocked by: #90627

Show alerts as annotations on latency chart. The alert instance should be displayed at the time it occurred. On hover it should be possible to see the threshold and the value that exceeded the threshold.

Only alerts matching the currently selected service.name and service.environment (via alert params) should be displayed.

[pmuellr]

On hover it should be possible to see the threshold and the value that exceeded the threshold.

We don't currently track the value or threshold at the time the alert executors run. Presumably the value would be available in the sourced data (blue line in the graph ^^^). But we also don't track the threshold - I suspect that this will become possible once we have a "alerts as data" story, where an alert will be able to store alert-specific data ... somewhere ... when the alert executors run. Today, we only store alert-generic data (so, not much).

Only alert instances matching the currently selected service.name and service.environment should be displayed.

I assume service.name etc are alert parameters? If so, we don't yet support any way of searching filtering based on the alert params - they aren't indexed - but are looking into right now via issue #50213

Show alert instances as annotations on latency chart.

Beyond what I mentioned above, the event log can provide timestamps for when alerts are active ("active-instance"), when they have just become active after not being active before ("new-instance"), when they are no longer active after they being active before ("recovered-instance"), errors (fields from the "execute" doc), and when actions have been invoked because the alert was active ("execute-action"). I think the timestamps will be the only interesting bit of info here (besides the alert id, etc).

Should be enough to show the vertical lines in the graph ^^^ ; If the pink-shaded areas are "alert was active", you can bound those by "new-instance" and "recovered-intstance" events, but keep in mind the query may not contain ALL the data, so you could have a string of "active-instance" events, with no "new-" or "recovered-" event, which would indicate the alert has been firing since before the data returned from the query, and/or beyond the data returned from the query (or possibly still active).

[sorenlouv]

Thanks for the feedback @pmuellr!

We don't currently track the value or threshold at the time the alert executors run

That's fine for for now.

I assume service.name etc are alert parameters? If so, we don't yet support any way of searching filtering based on the alert params - they aren't indexed - but are looking into right now via issue #50213

Yes, we are hoping to retrieve alerts by alert params.

the event log can provide timestamps for when alerts are [...]

I think timestamps for "execute-action" is the only useful one for us.
Out of curiosity: You mention timestamps for "active-instance", "new-instance", "recovered-instance". I've thought of these as implementation details - in what way would these be useful to consumers and/or end users? Sorry, it looks like you explain this further down: we can use those timestamps to determine when an alert started firing ("new-instance") and stopped again ("recovered-instance")

[sorenlouv]

Closing. @katrin-freihofner will create a new issue to replace this.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)