OpsGenie action type

[peterschretlen]

Describe the feature:

Send notifications to OpsGenie through their alert api.

• OpsGenie API documentation

Like other action types, the key field when binding to an alert will be message, but this action should be similar to PagerDuty in expressiveness, including as many JSON body fields as practical, like : alias, description, priority, source, entity etc.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[yevgenytrcloudzone]

Hi, is there any chance of releasing this feature in 7.10?

[mikecote]

@yevgenytrcloudzone

Hi, is there any chance of releasing this feature in 7.10?

This is currently targeting 7.12 for release. In the meantime, you may be able to integrate with OpsGenie using the webhook connector: https://www.elastic.co/guide/en/kibana/7.9/webhook-action-type.html.

[yevgenytrcloudzone]

@mikecote The webhook integration doesn't support passing request params for post that I need to OpsGenie.
Is that on your roadmap as well?

[mikecote]

@yevgenytrcloudzone I may need more context on your use case. Is this for having a parameterized URL? based on context.

There is an issue that tracks some findings when using the webhook connector for OpsGenie, you can find it here: #75299.

[yevgenytrcloudzone]

@mikecote I am trying to get Kibana Alerting to work with OpsGenie and (in opposed to Watcher) it does not seem to have support for REST request parameters. Is there a plan to provide an option to pass HTTP request parameters with the Kibana Alerting Webhook?
Hope that makes sense.

[mikecote]

@yevgenytrcloudzone request parameters could mean a couple of things, in your case which of the following are you trying to use?

• Path parameter (/foo/{id})
• Query string (/foo?id=1)
• Body
• Headers

Or if you can provide what your watcher config looks like, we can try and see what part is missing from the webhook action type.

[yevgenytrcloudzone]

To make things more clear, what I need is the same level of integration that exist today with PagerDuty.
OpsGenie expect few things that will be passed in a web request. Things such a token in a parameter, which exist for PagerDuty.

It's described for watcher integration here, but we use Elastic Cloud and I need this to be working with Kibana alerts.

[mikecote]

cc @arisonl

[mikecote]

Moving from 7.12 - Candidates to 7.x - Candidates.

[yevgenytrcloudzone]

BTW, if anyone is interested, the only work around is sending the alerts as e-mails to the e-mail integration at OpsGenie.
OpsGenie will create a dedicated e-mail address that will create the alerts with the subject and body from the e-mail message.

[mikecote]

Moving from 7.x - Candidates to 7.13 after the latest 7.x planning session. If we get time, we will implement this.

[scottdfedorov]

Moved to backlog? :(
Webhooks only help with opening alerts, as mentioned above. There seems to be no way to close alerts because we can't inject the parameter into the URL. This functionality is desperately needed.

It's nearly identical to Pagerduty, with different endpoints and config. Anything we can do help speed this up?

[nithril]

It is currently a major downside compared to the datadog OpsGenie integration. Our teams are using both Elastic and datadog, and migrating completely to Elastic is a nogo because of that.

[nithril]

Hello,

Would you accept a pull request for that feature (ie. an OpsGenie connector)?

[gmmorris]

Hi @nithril 👋 ,

The short answer is: Yes, absolutely!

The long answer is:

The team working on the alerting platform covers everything you can see in our project, which is quite a lot of stuff, and our capacity (like all engineering teams ;) ) is limited.
This means we don't have much capacity to guide a new contributor through the code base, review the PR and maintain the code after we accept the PR.

This means it might take us a while to get through the whole process, so I would urge you to only go to the effort if you have the time and patience to bear with us through that. 😄
In addition, we haven't done a good enough job documenting how to add a new connector, as this has been mostly leveraged internally thus far. So you might have to figure quite a bit out on your own (sorry about that!).

If you feel comfortable with those caveats, then, by all means, lets do it! 😍

Where to start?

To improve our chances I'd suggest we do the following: Start small

Any PR we review will have to include reliable unit tests and functional tests, documentation, and implementation that follow our contribution guidelines.
This is going to be easier for all of us if it starts out as a small PR, with an MVP in it, that we can then build on.
Once the first PR gets through the process, I can assure you follow-up PRs will be much easier and faster.

An example to look at:
If you're looking for a starting point, then it might help to look at the most recent PR adding a connector to Kibana.
This PR was contributed by the SwimLane team, but note that this PR didn't only add a connector, but also a variety of code to the Cases plugin (our case management offering) and also made some changes to other case management related connectors.

So, while that PR can help you, there's quite a bit of noise in there as well.
This example PR is far more than you would need to do (nor could my team accept contributions to Cases, as we do not own that product, and would have to loop in other reviewers from the Cases team).

I think the changes you'd need to make can be boiled down to (I might be missing something 😬):

1. Adding the connector itself to the builtin_action_types folder. Please add this inside of a dedicated folder like the Jira/Reslient/etc. connectors
2. Adding user facing docs for the OpsGenie connector
3. Adding the new connector to our telemetry so that we can track how often it's used in the wild
4. Add end-to-end tests that verify the behaviour of the new connector

Once you have those four, I think we can begin to review the PR and help you get it to the finish line. 😄

Good luck!! 🤞 🤞 🤞

[stephan-erb-by]

Hey! Questions to all the watchers on the ticket: Has anyone come up with a good strategy for auto-closing alerts in OpsGenie?

I am aware that OpsGenie has an auto-close policy. However, it always closes alerts after it has received 100 notifications from Elastic for it. In practice that means open alerts get auto-closed and re-opened every few hours which is rather annoying.

Anyone an idea how to solve that? (Besides having a specific OpsGenie action type that takes care of properly closing alerts :))

[beercialfo]

BTW, if anyone is interested, the only work around is sending the alerts as e-mails to the e-mail integration at OpsGenie. OpsGenie will create a dedicated e-mail address that will create the alerts with the subject and body from the e-mail message.

@yevgenytrcloudzone , I have a same use case as yours. I tried to use opsgenie alert api but it is difficult to pass dynamic variable in post api url in api connector.

Seems you used Email connector to achieve that, will the alert resolved automatically in opsgenie when it is resolved in elastic search or it creates new alert ? can you confirm that?

Thanks

[yevgenytrcloudzone]

@beercialfo unfortunately this method will create an alert for each action instead of aggregating all the actions in one alert like with any other API based integration for OpsGenie.
Far from being ideal in some ways, but better than nothing...

[beercialfo]

Thanks for your quick reply @yevgenytrcloudzone . Above you said is for alert api integration right,
How about email integration, is it smooth on creating and closing the alert?

@mikecote, When can we expect OpsGenie connector like pagerduty?

[yevgenytrcloudzone]

Email integration will only create an alert per email message received. Every action (critical/warning/recovered/no data) will create a new alert because there is no way to co-relate or distinguish alerts via email.

[beercialfo]

Thank you @yevgenytrcloudzone. FYI In Standard plan there is a way to avoid creating new alert for recovered one using filters.

[yevgenytrcloudzone]

This is not related to the subject, so please let's just stop here.
When you an email from Kibana alerts, there is no correlation ID that can point when a certain alert reached one of it's states and therefore every email action will be considered as a new alert.

[mikecote]

@mikecote, When can we expect OpsGenie connector like pagerduty?

We will start working on this in the coming weeks, but we don't have a promise to what release it will be included within.

[marcomusso]

I am testing this connector on 8.6.1 and found a bug with I already reported to support (pending confirmation).
So we have a connector that works... can this ticket be closed?

[jonathan-buttner]

@marcomusso thanks for pointing that out and providing feedback! I'm closing this issue and will create additional enhancement issues from your feedback.