how to configure with icinga-director #5

lcorsini · 2017-06-20T09:40:18Z

I've tried installing this component, but I'm stuck, since I'm using icinga-director.
I've installed and configured the deb package.
In icinga-director added the vars.slack_notifications to my hosts, but when I apply the config it says
critical/config: Error: Import references unknown template: 'slack-notifications-user-configuration-hosts' Location: in /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf: 47:3-47:55 /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(45): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(46): apply Notification "slack-notifications-notification-hosts" to Host { /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(47): import "slack-notifications-user-configuration-hosts" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
because director doesn't know anything about this new template

The text was updated successfully, but these errors were encountered:

staenker · 2017-06-20T10:15:02Z

I'll get to it this Friday,
cheers,
Richard

lcorsini · 2017-06-27T08:57:45Z

Hi, any update on this?

majales · 2017-12-08T09:57:01Z

Hi.. I did it on Director. I created Data Field slack_notifications and added it to host-template in Director and set it to "enabled" .. and added it to service-template in Director also with "enabled" .. It did the trick.. I had not modified hosts and service template file,s it is not necessary.. It was on icinga 2.7 and director 1.3.1 ( slightly newer then release.. It was taken from github repo in April )

rgevaert · 2017-12-08T10:01:22Z

Same here!

flaviomeyer · 2018-02-28T21:56:31Z

Hi @nisabek

Thanks for this nice notification plugin. I'm looking Icinga Director install guide. Is it possible to create a guide for Icinga Director?

@majales Maybe you can post your solution?

Thanks :)

Flavio

nisabek · 2018-03-01T12:06:37Z

Hi all,

Thanks for addressing this, and sorry for late response.
I just got to try icinga-director, and in order for the director to recognize the changes in the icinga core I run the command icingacli director kickstart run

That should be run after the usual installation and final restart of the icinga2 service.
After that the slack-notifications-command is visible in the command list of the icinga director, and if your hosts have the vars.slack_notifications = "enabled" set it will fire the notifications.

@lcorsini @meyerf99 - I would appreciate if you give it a try and report feedback.

Thanks and Regards,
Nune

flaviomeyer · 2018-03-01T15:41:39Z

Hey @nisabek

Thanks for your fast answer. I got this error with timeperiod.

critical/config: Error: Validation failed for object 'x.ch!disk_check!slack-notifications-notification-services' of type 'Notification'; Attribute 'period': Object '24x7' of type 'TimePeriod' does not exist. Location: in /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf: 8:5-8:19 /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(6): user_groups = [ "icingaadmins" ] /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(7): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(8): period = "24x7" ^^^^^^^^^^^^^^^ /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(9): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(10): interval = 5m
Maybe you know how i can fix it?

Kind regards,
Flavio

rgevaert · 2018-03-01T15:53:14Z

It's possible to manage the custom attributes on host/service level through director. However configuring the other things isn't possible. Or at least I didn't find how. I added them to the configuration files.

flaviomeyer · 2018-03-01T19:23:29Z

I've fixed my problem, but it was very hard. @nisabek Maybe u can documentation with timeperiod and icingaadmins group as prerequisite. Is it possible to test notifications?

From my point of view, I have everything set up, but unfortunately I think no message in Slack.
Does the script have a logfile? If yes, where?

Kind regards,

Flavio

PS: In my Debug Log i see only this:
[2018-03-01 20:58:04 +0100] debug/slack-notifications: Sending notification...read user configuration successfully: {"vars.slack_notifications_botname":null,"vars.slack_notifications_channel":null,"vars.slack_notifications_icinga2_base_url":null,"vars.slack_notifications_icon_dictionary":null,"vars.slack_notifications_plugin_output_max_length":0.0,"vars.slack_notifications_webhook_url":null}

I think thats false, coz all infos have null as string, is that right?

lcorsini · 2018-03-02T13:21:09Z

In the meantime, I've actually reinstalled my infrastructure without director so I cannot really test this anymore in the short term.

flaviomeyer · 2018-03-02T15:31:07Z

@nisabek I've fixed my problem.

Set in Icinga Director Notification Interval to 60 and not 0.

See this link in: https://monitoring-portal.org/woltlab/index.php?thread/41134-slack-notification-im-director-per-user-zuweisung/

nisabek · 2018-09-08T13:25:42Z

seems like this is fixed. The current master contains the reference to a docker image that can be used for testing.

nisabek · 2019-06-18T15:15:36Z

It looks like it's still unclear how to configure the plugin in Icinga Director.
@majales @rgevaert - could you clarify if you had any luck configuring the webhook_url and making it work in Icinga Director? If so, I'd be very thankful for a README contribution with steps how to achieve that.

Thanks!
Nune

rgevaert · 2019-06-18T15:59:47Z

Sorry, I'm not using icinga2 anymore (different job) so can't help unfortunately.

buggsdummy · 2019-09-04T15:29:00Z

Can someone update this? Thanks!

flaviomeyer · 2019-09-04T15:32:43Z

Interesting...
I have looked for this notification addon and found out that is not working anymore.
Maybe I build in the next few days a new slack notification plugin which works with Icinga Director.

NC-Yungd · 2019-12-12T18:56:29Z

After running the kickstart wizard in Icinga2 Director, is this the expected output for the NotificationCommand?

I am unable to get any messages in my Slack channel. Below are from my debug logs:

# tail -f /var/log/icinga2/debug.log | grep "warning/PluginNotificationTask\|slack-notifications"
[2019-12-12 17:45:15 +0000] notice/Notification: Attempting to send notifications of type 'Custom' for notification object 'server1.com!slack-notifications-notification-hosts'.
[2019-12-12 17:48:22 +0000] notice/NotificationComponent: Attempting to send reminder notification 'server2.com!linux_apt!slack-notifications-notification-services'.
[2019-12-12 17:48:22 +0000] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'server3.com!linux_apt!slack-notifications-notification-services'.
[2019-12-12 17:48:22 +0000] notice/NotificationComponent: Attempting to send reminder notification 'server4.com!linux_apt!slack-notifications-notification-services'.

nisabek · 2019-12-12T19:06:51Z

@NC-Yungd can't verify right now, if that's the expected output, but I can tell you that those logs are not from the plugin(your grep is correct, just it matches some other lines...)...are you sure everything is enabled on the plugin side?

flaviomeyer · 2019-12-12T21:03:53Z

@NC-Yungd A few weeks ago I have rebuilt the Slack notification. If I find time tomorrow I'il create a new gist and I'il let you know.

NC-Yungd · 2019-12-12T22:14:19Z

Here are my current configurations:

/etc/icinga2/zones.conf

object Endpoint "icinga2.xyz.com" {
}

object Zone "master" {
	endpoints = [ "icinga2.xyz.com" ]
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

/etc/icinga2/zones.d# tree -d

.
├── global-templates
│   ├── commands
│   └── templates
└── master
    ├── archive
    └── slack-notifications

/etc/icinga2/zones.d/master/slack-notifications# cat slack-notifications-user-configuration.conf

template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/secret1/secret2"
    vars.slack_notifications_icinga2_base_url = "http://icinga2.xyz.com"
    vars.slack_notifications_channel = "#icinga2_alerts"
    vars.slack_notifications_botname = "icinga2_bot"
}

template Notification "slack-notifications-user-configuration-hosts" {
    import "slack-notifications-default-configuration-hosts"
}

template Notification "slack-notifications-user-configuration-services" {
    import "slack-notifications-default-configuration-services"
}

In Icinga2 Director, I've created the custom variable for slack_notifications as a string. Afterwards, I've add ed slack_notifications as a field to my Host and Services template and included the string "enabled".

nisabek · 2019-12-13T07:13:15Z

hmm, from the fact that there are no logs of the plugin, I suppose that the slack_notifications variable is not understood by the director...
when you say "custom variable for slack_notifications as a string" do you mean a data field? Cause @majales mentioned creating a data_field and I'm not sure if that's the same as "custom variable"

Sorry I can't do more at the moment :/

NC-Yungd · 2019-12-13T19:22:36Z

@nisabek Initially, I installed this plugin by using the "Debian package" method, but after installing with git and copying the slack-notification conf files into my Icinga2 zones.d/master folder, I was able to get notifications in Slack.

I'm going to assume that the Debian package is out-of-date since I wasn't able to get Slack notifications with Director.

nisabek · 2019-12-15T07:15:55Z

Thanks @NC-Yungd for getting back to us and making it work!
I'm having a bit of a trouble updating the dbian repo, in the meantime I updated the README

https://github.com/nisabek/icinga2-slack-notifications#installation-using-debian-package
https://github.com/nisabek/icinga2-slack-notifications#running-with-icinga-director

If you feel like there's information missing regarding making it work on icinga director, I'd appreciate a a PR with more details.

Thanks!
Nune

Edit: updated wrong links.

guzaho · 2020-03-27T15:41:05Z

Hi there!

Thanks for creating such a useful plugin.

I recently installed icinga2 with icingaweb2 and director, and I need to integrate is to my company's slack.

I've followed the howto and installed from git (and discovered a few mistakes in the howto), as suggested, but I cannot make it work.

This is what I have done:

ONE) Created a slack webhook and verified with curl that I do indeed get messages into the right channel.

TWO) Logged into the icinga2 master machine as root, and

cd /etc/icinga2/conf.d/
git clone https://github.com/nisabek/icinga2-slack-notifications.git
cd icinga2-slack-notifications
cp src/slack-notifications/slack-notifications-user-configuration.conf.template slack-notifications-user-configuration.conf
chown -R nagios:nagios /etc/icinga2/conf.d

THREE) Customized /etc/icinga2/conf.d/templates.conf

template Host "generic-host" {
max_check_attempts = 3
check_interval = 1m
retry_interval = 30s

check_command = "hostalive"
vars.slack_notifications = "enabled"
}

/**
* Provides default settings for services. By convention
* all services should import this template.
*/
template Service "generic-service" {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s

vars.slack_notifications = "enabled"
}

FOUR) Customized /etc/icinga2/conf.d/icinga2-slack-notifications/slack-notifications-user-configuration.conf

template Notification "slack-notifications-user-configuration" {
import "slack-notifications-default-configuration"

vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/secret1/secret2"
vars.slack_notifications_icinga2_base_url = "https://my.company.com/icingaweb2"
vars.slack_notifications_channel = "#service-health"
vars.slack_notifications_botname = "service_health_bot"
}

template Notification "slack-notifications-user-configuration-hosts" {
import "slack-notifications-default-configuration-hosts"
}

template Notification "slack-notifications-user-configuration-services" {
import "slack-notifications-default-configuration-services"
}

FIVE) Enable Icinga2 debug logging
icinga2 feature enable debuglog

SIX) Restarted icinga2
systemctl restart icinga2

SEVEN) Triggered an alert by stopping a service, and verified that icingaweb2 shows the alert

But I don't see any slack in the debug file.
grep "warning/PluginNotificationTask\|slack-notifications" /var/log/icinga2/debug.log

I have not configured anything in the director, because I'm new to Icinga2, and the discussions are not clear on what to do in what order.

I don't have zones configured, do I need to?

Can anyone please help?

nisabek · 2020-03-27T16:16:52Z

hi @guzaho and thanks for using the plugin
It's getting hard to solve these issues, since everyone's setup looks different, but I'm gonna try.

question - does the service you stopped to test, use the generic-service template?...
Can you show how it is configured?...
another question - Do you see any logs in the debug.log?..

guzaho · 2020-03-27T18:35:07Z

Hi @nisabek !

Thanks for the prompt reply!

I do see plenty of logs in the debug.log.

Actually, right before I saw your message now, I got a message in slack! Yay!

It seems however, that I am only getting messages for the master node.

The service I stopped is checked by check_systemd, and my service template looks like this in director:

zones.d/director-global/service_templates.conf
template Service "check_systemd" {
check_command = "check_systemd"
command_endpoint = host_name
vars.slack_notification = "enabled"
}

The service set that uses the service template is like this:

apply Service "check_systemd" {
import "check_systemd"

assign where "sentinel-template" in host.templates

import DirectorOverrideTemplate

}

So I guess it does not use the generic-service template?

nisabek · 2020-03-27T18:37:15Z

oh wait, that's a template on it's own, but do I see a typo?...
vars.slack_notifications = "enabled"

guzaho · 2020-03-27T18:41:01Z

ok, fixed the missing s

zones.d/director-global/service_templates.conf
template Service "check_systemd" {
check_command = "check_systemd"
command_endpoint = host_name
vars.slack_notifications = "enabled"
}

nisabek · 2020-03-27T18:43:11Z

and?...works?.. :)

guzaho · 2020-03-27T18:46:04Z

No, not for other hosts.. just the master... which does only use default services like apt, procs, etc. My nodes only use check_apt and check_procs, etc.

The particular service I'm testing does not post any msgs to slack, it seems...

guzaho · 2020-03-27T18:47:27Z

I've followed the icinga2 official docs to the point. I've set up service templates, then service groups and the assigned that to hosts. So any service that inherits the template for check_systemd has now gotten the slack_notifications set to "enabled":

nisabek · 2020-03-27T18:54:08Z

alrighty that looks correct...questions

you say, you get notifications from your 'master' node - which template does the master node use?..
on a side note - you must be then seeing logs from the master node...
in the comment by @NC-Yungd it is mentioned that
"In Icinga2 Director, I've created the custom variable for slack_notifications as a string. Afterwards, I've add ed slack_notifications as a field to my Host and Services template and included the string "enabled"."

can you try that?...

guzaho · 2020-03-27T19:01:25Z

Of course, I can try! Thank you!

Yes, I do see slack messages in the debug log now, but only for the master node.

I am not sure which template the master node uses, it just showed up after installing icinga2, icingaweb2 and the director.. How can I check?

Yes, I did as @NC-Yungd suggested. I've created the slack_notifications as a string in data fields:

I have now also set the slack_notifications to enabled for the host, and it works!!

guzaho · 2020-03-27T19:02:38Z

Ok, so just to sum up, the missing piece what that misspelling that check_notifications should be in plural form, and that the hosts did not have the slack_notifications set to "enabled"!

thank you so much!

nisabek · 2020-03-27T19:02:41Z

I looked through the docs again, and your configuration, and I don't seem to find if you have this piece somewhere

apply Notification "slack-notifications-notification-services" to Service {
  import "slack-notifications-user-configuration-services"

  assign where service.vars.slack_notifications == "enabled"
}

guzaho · 2020-03-27T19:03:11Z

Hm, that's true. Where does that go?

nisabek · 2020-03-27T19:04:33Z

ah, ignore that, that's in the source code! so you must have checked it out :D my memory of the project is not that fresh sorry :)

nisabek · 2020-03-27T19:05:12Z

so - works now?

guzaho · 2020-03-27T19:05:12Z

Yes, looks like it. I'm gonna do some more testing, just in case. But all good for now! Thank you so much. 😄

nisabek · 2020-03-27T19:08:07Z

awesome, enjoy!

I really want to close this issue cause it's turned into debugging session of different issues.

Anybody having trouble with icinga2 director configuration please refer to last part of the docs
https://github.com/nisabek/icinga2-slack-notifications#running-with-icinga-director

If that doesn't help - please open a new issue. And of course, contribute - with more docs, more tests, ...! ;)

nisabek closed this as completed Sep 8, 2018

nisabek reopened this Jun 18, 2019

nisabek changed the title ~~how to configure with icinga-director~~ [Help wanted] how to configure with icinga-director Mar 27, 2020

nisabek added the help wanted label Mar 27, 2020

nisabek changed the title ~~[Help wanted] how to configure with icinga-director~~ how to configure with icinga-director Mar 27, 2020

nisabek closed this as completed Mar 27, 2020

cherazo mentioned this issue Jun 9, 2020

Icingacli director kickstart run - error for slack_notification setup #31

Open

how to configure with icinga-director #5

how to configure with icinga-director #5

Comments

lcorsini commented Jun 20, 2017 • edited Loading

staenker commented Jun 20, 2017

lcorsini commented Jun 27, 2017

majales commented Dec 8, 2017 • edited Loading

rgevaert commented Dec 8, 2017

flaviomeyer commented Feb 28, 2018

nisabek commented Mar 1, 2018

flaviomeyer commented Mar 1, 2018 • edited Loading

rgevaert commented Mar 1, 2018

flaviomeyer commented Mar 1, 2018 • edited Loading

lcorsini commented Mar 2, 2018

flaviomeyer commented Mar 2, 2018 • edited Loading

nisabek commented Sep 8, 2018

nisabek commented Jun 18, 2019

rgevaert commented Jun 18, 2019

buggsdummy commented Sep 4, 2019

flaviomeyer commented Sep 4, 2019

NC-Yungd commented Dec 12, 2019

nisabek commented Dec 12, 2019 • edited Loading

flaviomeyer commented Dec 12, 2019

NC-Yungd commented Dec 12, 2019

nisabek commented Dec 13, 2019

NC-Yungd commented Dec 13, 2019

nisabek commented Dec 15, 2019 • edited Loading

guzaho commented Mar 27, 2020

nisabek commented Mar 27, 2020

guzaho commented Mar 27, 2020 • edited Loading

nisabek commented Mar 27, 2020

guzaho commented Mar 27, 2020

nisabek commented Mar 27, 2020

guzaho commented Mar 27, 2020

guzaho commented Mar 27, 2020 • edited Loading

nisabek commented Mar 27, 2020

guzaho commented Mar 27, 2020

guzaho commented Mar 27, 2020

nisabek commented Mar 27, 2020

guzaho commented Mar 27, 2020

nisabek commented Mar 27, 2020

nisabek commented Mar 27, 2020

guzaho commented Mar 27, 2020 • edited Loading

nisabek commented Mar 27, 2020

lcorsini commented Jun 20, 2017 •

edited

Loading

majales commented Dec 8, 2017 •

edited

Loading

flaviomeyer commented Mar 1, 2018 •

edited

Loading

flaviomeyer commented Mar 1, 2018 •

edited

Loading

flaviomeyer commented Mar 2, 2018 •

edited

Loading

nisabek commented Dec 12, 2019 •

edited

Loading

nisabek commented Dec 15, 2019 •

edited

Loading

guzaho commented Mar 27, 2020 •

edited

Loading

guzaho commented Mar 27, 2020 •

edited

Loading

guzaho commented Mar 27, 2020 •

edited

Loading