Data formatting and manipulation support for Alert payloads

[alexfrancoeur]

We've received a request where it would be useful to use painless to transform the data returned before sending off to a 3rd party integration. This could be PagerDuty, ServiceNow, etc. This makes it much easier for the consumer of the incident to understand the data coming through.

Painless is a language that is already in use with Watcher, so there is an argument and preference for this syntax over others. Having to learn another language adds additional overhead. The alternative, and one I believe we've been leaning towards in the past, is to use Kibana expressions and functions to provide this level of transformation and flexibility.

This issue is meant to track the enhancement request and will need to be discussed and prioritized accordingly. This request has some similarities to #76910.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Creating alerts with Kibana expressions is still fairly out (#50270). I wonder if there's anything with expressions that could solve this requirement.

[pmuellr]

Curious about thoughts on executing arbitrary painless code from Kibana. Below is what the painless lab is doing - is this the recommended approach? That part seems pretty straight-forward, if so.

kibana/x-pack/plugins/painless_lab/server/routes/api/execute.ts

Lines 25 to 29 in 4b49e5a

	try {
	const callAsCurrentUser = ctx.core.elasticsearch.legacy.client.callAsCurrentUser;
	const response = await callAsCurrentUser('scriptsPainlessExecute', {
	body,
	});

The hard part would be figuring out how this all fits together. Do we allow action parameters to be "scripted" as well as supporting the built-in mustache templating? How would this work in the UI? How do the shape of the action parameters change to accommodate this?

I think we should start with some specific scenarios from customers here, if they're already chiming in. What kind of things are they expecting to be able to do?

[gmmorris]

@arisonl @alexfrancoeur I think this issue is a near-duplicate of #89161 but not a 1-to-1 overlap.

Should they be merged into one? Should we dedup the parts that are the same?

cc @mikecote

[pmuellr]

Since this was opened, we've extended mustache to support some data manipulation like date, number formatting, in the following PRs:

• [ResponseOps] adds mustache lambdas and array.asJSON #150572
• [ResponseOps] adds FormatNumber mustache lambda #159644

Presumably additional manipulations can be added in the same fashion.

Since no other specific manipulations were mentioned here, I'm going to close.

As a further note, we have done some experiments trying to use expressions somehow, but the problem is that our current path through mustache templating is synchronous, and using expressions would require async. As would painless support. Painless would be a bit painful given the extra network hop and potentially huge payloads of the alert context to process.

[pmuellr]

Also note that with ES|QL support, it should be possible to do some amount of "manipulation" within the queries sent to ES.