Send email and/or IRC notification if pipeline starts failing #41
Comments
|
Would it be fine to use a plugin for this? The plugin https://github.com/jenkinsci/email-ext-plugin seems to be referred to a lot when searching on the topic. Would just need to make sure the version of the plugin works with the version of the Jenkins console we are using. Also looks to be an IRC plugin https://github.com/jenkinsci/ircbot-plugin, but not as recently updated. |
|
Yup, using a plugin is definitely the way to go! I have a slight preference for IRC over email for this, but if there's blockers there for whatever reason, email is fine to start. |
|
Re. versioning, that shouldn't be an issue. The master is more or less up to date with the latest LTS release. :) |
|
👍 I like the sound of IRC as well, avoids the extra noise to a mailing list (though wouldn't expect it to be too noisy itself). |
|
For a channel to log to, would we eventually want a separate channel for the Jenkins pipeline rather than |
|
I could give this a try if we'd like it soon - could try getting a POC working locally with the Jenkins IRC plugin. |
Depends on how chatty it would be. We could start with |
|
Did some looking into this last week - I could install the IRC plugin [1] using the Jenkins web console. However, there isn't a tab in the web console for configuring
The configuration I have done (in my local developer pipeline) is:
Right now, the IRC bot joins the specified channel when Jenkins starts, but doesn't send notifications when a build starts. Looking for a way to configure the IRC plugin through the pipeline, it seems support for this is not available [2]. An alternative which may be worth considering is [3], sending the IRC notification through the Groovy script. This may be enough for what we need - and it'd be nice to define everything in the Jenkinsfile. An example implementation is available [4], if we can make it work with freenode. [1] https://wiki.jenkins.io/display/JENKINS/IRC+Plugin |
|
This also looks promising: jenkinsci/ircbot-plugin#21 |
Hmm yeah, though as per the discussions there and https://plugins.jenkins.io/ircbot itself, it doesn't seem like the plugin is maintained much anymore.
Oh man, that one made me giggle. I guess that'd work, though not sure if we want to resort to that. Hmm maybe for now, let's at least do Slack notifications? It has native pipeline support, and is actively maintained. Even once we add IRC support, I think having it on both could be useful? For IRC, once we get fedora messaging working, I wonder if there's a bot in the Fedora infra which could do what we want already (i.e. fedmsg to IRC). We should ask the infra folks about this. |
|
In RHCOS land we've discussed using the Slack plugin as well, but decided we'd rather not get involved with Jenkins plugins if possible. We'll probably implement something like this instead: https://www.geekytidbits.com/til-post-to-slack-from-jenkins/ |
yes. notifications from fedora messaging /fedmsg go to IRC channels all the time. You can also configure them to be sent to you directly. See https://apps.fedoraproject.org/notifications |
|
Hi all, just FYI : I recently joined the list of maintainers for the Jenkins instant-messaging-plugin (the core doing grunt work) and the ircbot-plugin (the protocol adapter for IRC in particular), did some cleanup and merges of outstanding PRs, and had cut releases of both. So it should now be possible to use an On a side note, something perhaps lacking conceptually with Jenkins notifications about job results, is that you'd have to add notification steps (whether calling existing plugins, or making your own) separately for each protocol (IRC, jabber, mail, etc.), in each job definition. There is no magic "notify-everything-configured" silver bullet currently (a shared library defining the magic notifier for your build farm, so you only call one routine and maintain it in one place, might come close however). |
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we want both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure Jenkins using YAML instead of XML. For now, we're simply using it to configure the Slack plugin. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we want both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure Jenkins using YAML instead of XML. For now, we're simply using it to configure the Slack plugin. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in coreos#41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
We need to add more monitoring on the pipeline. Otherwise it's going to be very easy to go red for a few days without noticing. My initial goal was to hook up to IRC, but the reality is that I think we'll need both Slack and IRC because many folks mostly live in Slack nowadays. (That said, I still fully intend to add IRC support via fedmsgs as discussed in #41). Now, how this patch works is that we add two new Jenkins plugins: - configuration-as-code - slack The first one is used to configure the second one. More broadly, it's able to configure almost all of Jenkins and its plugins via YAML instead of dropping to XML, so we'll likely be leveraging it some more in the future. The actual configuration bit here is more or less the same as RHCOS, with some minor variations (we use secrets instead of defining the token at template generation time, and we just set a default channel instead of passing it through an env var) Of course, this is all still optional. The local developer workflow should still work fine.
|
There now exists a matrix plugin for jenkins: https://github.com/jenkinsci/matrix-communication-plugin since we are moving more and more away from IRC because of the libera/matrix bridge being down maybe we should explore this as an option. |
|
Yeah, if the outcome of coreos/fedora-coreos-tracker#1566 is to fully cutover communications to Matrix, then I think targeting Matrix here too seems like the most reasonable path now. It also supports features that we've come accustomed to having for these notifications (like threading). |
E.g. to freenode/#fedora-coreos. Otherwise, it's too easy to miss when status goes red.
The text was updated successfully, but these errors were encountered: