I think you should add whether you have confirmed the operation to the checklist of the pull request.

[nsano-rururu]

Even if you check the operation, it will occur when a bug occurs, but it cannot be prevented by just adding test code, so I think it is necessary to check the operation.
If you haven't confirmed the operation, it's a good idea to have someone in the maintainer confirm the operation after the review before merging.

By the way, it is possible to check the operation of alert notifications of Slack, MS Teams, Email, Telegram, Line Notify, Mattermost, Command, Gitter, Amazon SNS, Amazon SES, Zabbix, Stomp, Chatwork, Discord, Rocket.Chat, Dingtalk, Other alert notifications cannot be confirmed because there is no environment.

[ferozsalam]

I think this is a good idea - a couple of other thoughts on this:

1. Should we define formal owners for certain alerters? For example, I can look after verifying changes to TheHive (and a couple of others), because I have access to these platforms and can easily check whether a change does what it says on the tin for simple cases. In those cases, the owner(s) of an alerter can be responsible for verifying and approving a proposed change to their alerter.
2. Should we also consider implementing some sort of end-to-end testing for ElastAlert? I have a small project that does something of the sort - I'm happy to improve it where necessary and integrate it with the testing we run, if this would help guarantee that we're always building something that works?

I am aware that no one is looking after ElastAlert full-time so we need to think of solutions that will not be too time-consuming. Would be interested to know what people think.

[nsano-rururu]

Should we define formal owners for certain alerters? For example, I can look after verifying changes to TheHive (and a couple of others), because I have access to these platforms and can easily check whether a change does what it says on the tin for simple cases. In those cases, the owner(s) of an alerter can be responsible for verifying and approving a proposed change to their alerter.

The explanation was insufficient. I don't usually use ElastAlert2, and I run ElastAlert2 via Praeco and ElastAlertServer fork, so I can only check alert notifications.

[nsano-rururu]

I think this is a good idea - a couple of other thoughts on this:

1. Should we define formal owners for certain alerters? For example, I can look after verifying changes to TheHive (and a couple of others), because I have access to these platforms and can easily check whether a change does what it says on the tin for simple cases. In those cases, the owner(s) of an alerter can be responsible for verifying and approving a proposed change to their alerter.
2. Should we also consider implementing some sort of end-to-end testing for ElastAlert? I have a small project that does something of the sort - I'm happy to improve it where necessary and integrate it with the testing we run, if this would help guarantee that we're always building something that works?

I am aware that no one is looking after ElastAlert full-time so we need to think of solutions that will not be too time-consuming. Would be interested to know what people think.

I think it's worth considering if there is something that can be automated in the part that the test code does not support

[jertel]

I've added the new step to the PR checklist. Thanks for offering to help test the PRs. Anything we can do to automate additional testing is appreciated, but I know how much work is involved in getting new tests and test infrastructure setup.