[Alerting] "View in App" for ES Query Alert Type

[ymao1]

For the ES query alert type #88528, we would like to explore a way of allowing users to access the raw documents that matched their query and threshold conditions, possible through a "View in App" view. The top 1000 hits are currently passed through the action context for use inside actions, so this is one way they can currently see the documents but (1) it is not ideal to be storing that much data even temporarily inside of a saved object and (2) if the query happened to match hundreds of thousands (millions!?!?!) of documents, the user would not be able to get them.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[ymao1]

Currently, during each execution of this alert type, when the threshold conditions are matched, the following is stored inside the alert instance state: dateStart, dateEnd, latestTimestamp. These, along with the query from the alert params, should be all we need to recreate the query that led to that active alert instance.

If we stored the alert instance state inside the event log when event.action: active-instance, we could base the View in App off of the event log by retrieving all active-instance events for that alert id. This would allow us to show a history of every time the alert was active, and for each entry, recreate the query and retrieve the documents.

Example of event log entry with alert instance state

{
	"_index": ".kibana-event-log-8.0.0-000001",
	"_id": "snhvRXcBW6kYwWa7bbzd",
	"_score": null,
	"_source": {
		"@timestamp": "2021-01-27T20:01:06.371Z",
		"event": {
			"provider": "alerting",
			"action": "active-instance"
		},
		"kibana": {
			"alerting": {
				"instance_id": "query matched",
				"action_group_id": "query matched",
				"alert_instance_state": """{"latestTimestamp ":1611777610424,"dateStart ":"2021 - 01 - 27 T19: 56: 06.365 Z ","dateEnd ":"2021 - 01 - 27 T20: 01: 06.365 Z "}"""
			},
			"saved_objects": [{
				"rel": "primary",
				"type": "alert",
				"id": "f13c3580-60c9-11eb-bc88-8173d2f513c6"
			}],
			"server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d"
		},
		"message": ".es-query:f13c3580-60c9-11eb-bc88-8173d2f513c6: 'test' active instance: 'query matched' in actionGroup: 'query matched'",
		"ecs": {
			"version": "1.6.0"
		}
	},
	"sort": [
		1611777666371
	]
}

[pmuellr]

If we stored the alert instance state inside the event log when event.action: active-instance ...

That's a big (as in bytes) ask! :-) Not because this specific data is too large, but because we don't know how big instance state is for other alerts. But in general, this feel like it should be part of the "alerts as data" work coming - not clear WHAT kind of data we'll be storing for alerts, or WHERE, but presumably once we have a mechanism, this alert could make use of it to store that state.

we could base the View in App off of the event log by retrieving all active-instance events for that alert id.

So the idea would be to get all the active-instance events, get the time ranges from those, build a filter that filters only within those dates, run the original query with that filter, that shows all the hits. I assume we'll need a new view for that - I think that was discussed in a recent call.

I'd love to have a super-easy version of this that just opened Discover with your query, forget about the time ranges. There would be noise - all the docs that didn't make the threshold limit would show. But perhaps that would be useful none-the-less - you might find that you wanted to decrease your threshold, for instance, if you saw situations where you did want to be notified, but the threshold didn't set the alarm off. Bigger problem is that we can't just throw this into Discover, AFAIK, since Discover works off of KQL | Lucene, not query DSL.

I guess that's a good question - why isn't there something like Discover in Kibana, that works off of query DSL? Or is there and I'm not aware of it? Or maybe we could morph Discover such that we can feed it query DSL for it to make the calls with, but not allow the "original query" (in query DSL) to be modified (maybe show it as read-only text?). You'd still want all the other control-ly bits to work - time ranges, additional filters, etc.

[pmuellr]

Just happened to think that we could likely construct the query we're going to run, and provide a way to paste that into the dev console, for the user to run. Should/could be paired with #84417

[ymao1]

@arisonl What do you think of closing this since there have been no requests for it since the ES Query type was introduced?

[arisonl]

hey @ymao1, for awareness there is a number of user requests around being able to access the query payload. Now, first they are not all necessarily around the ES query type, but I think that starting with the ES query type might go a long way. Second, we have a number of requests around being able to manipulate the payload in order to populate action parameters. Could the "view in app" be a first step towards that?

[ymao1]

@arisonl Gotcha thank you! We can keep this issue open for ES Query View in App if there are requests for it. I believe we have a separate issue for tracking payload manipulation in action variables - #89161

[arisonl]

@ymao1 when the time comes, I think that we should think about these issues as an end-to-end use case with the UX that we want to produce around it, rather than separate, independent increments.