Alerting strategy question #1413
-
|
Apologies if this has been asked before but I searched and couldn't find anything. My use case is I'll be the owner of a logging ES cluster deployed to k8s via helm, with other teams responsible for creating their own alerts. Because I don't want to have to manage .yaml files and deal with poorly made files resulting in crashes, redeploying all the time, etc, I'm looking for a more static approach when it comes to my end, in particular the alert .yaml files. My thought is to use the ES native functionality to allow 'customers' to configure alerts to write to a dedicated 'alerting' index, and then I'll just use a small subset of .yamls for searching that index and firing to opsgenie or email or whatever channel using elastalert2. I'll supply the customers templates that they can use to send data to the alerting index to ensure they supply the required info such as the alerter, recipients, etc. In theory this will allow tracking/auditing of alert history as well as sending different notifications based on the status of the alert, ie alerting or recovered, potentially flapping as well. Since I haven't been able to find others doing this I'm wondering if I'm missing something obvious as to why one should not do this approach? Any thoughts or recommendations are appreciated. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
|
I think I see what you're trying to do. You want a single instance of ElastAlert 2 to monitor N number of If that's the intention then yes it would handle this but it would seem that you'd only be making use of half of ElastAlert 2's capabilities. If you look at the project you could say it's half detection and half alerting. The detection capabilities would be mostly unused since it would be up to the customers to ensure they have their own methodology for detecting an issue and creating a record in their alert index. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, thanks for your response, this is correct. I'd have them create alerts using the native ES Query based alerts with templates depending on how they want to send an alert, with a template for smtp, opsgenie, or slack, and the templates pass on the alerter and other details ElastAlert2 will need to pass the alert along. Typing this out, I may be making things more complicated rather than less. Full disclosure I am aware of the Elastalert2 Server and Kibana plugins, but haven't had a chance to test these yet. Essentially I want to minimize other teams messing up the alert yaml files and having to re-deploy elastalert2 constantly, as they'll be the owners of their own alerts and we want to limit their destructive capabilities. In my initial testing, having a bad alert.yaml config resulted in some crashes of ElastAlert2 (sorry, this was a while back and I didn't think to put in a bug report, and can't recall what parameter was bad) and I want to be as hands-off as possible, though perhaps this isn't realistic. As it is, getting these folks to stick to the templated alerts I make is probably not going to be any better than getting them to make proper yaml files. I'll dig deeper into the detection capabilities of ElastAlert2 to see what we'd be missing out on and determine whether the trade-off is worthwhile. On a side note, I've been thinking about trying to get a solution in place for the opsgenie alerter to be able to close alerts by leveraging the opsgenie_alias and somehow manipulating opsgenie_addr to append '{{opsgenie_alias}}/close?identifierType=alias'. Maybe I'm overthinking this and wouldn't need any code changes actually... I'd just need to pass modified opsgenie_addr in the 'recovery' payload instead of relying on the default. |
Beta Was this translation helpful? Give feedback.
-
|
See #337 for a similar discussion. |
Beta Was this translation helpful? Give feedback.
-
|
In case anyone else stumbles down this path, I'll recommend against it. My initial testing was doing a bit of circular logic and as a result the notifications going out weren't linking back to the search results which triggered the alert, but rather back to the search of the alert index which triggered the notification, which is not of value to the person handling the alert. Instead I plan to use elastalert2 as intended, or potentially Karqls elastalert2-server/elastalert-kibana-plugin though in the limited testing I've done so far can result in crashing of the service if the YAML isn't just right. |
Beta Was this translation helpful? Give feedback.
I think I see what you're trying to do. You want a single instance of ElastAlert 2 to monitor N number of
customer_N_alertsindex for new records, and trigger outbound notifications using ElastAlert 2's numerous alerters. Is that correct?If that's the intention then yes it would handle this but it would seem that you'd only be making use of half of ElastAlert 2's capabilities. If you look at the project you could say it's half detection and half alerting. The detection capabilities would be mostly unused since it would be up to the customers to ensure they have their own methodology for detecting an issue and creating a record in their alert index.