[BUG] Problem Sending Mail Without Smtp Auth on OpenSearch

[espala]

Hello everyone,

I am using Opensearch cluster and my cluster is running smoothly. I don’t get any error messages or logs in the standard run cycle.

I wanted to define an alarm specifically, so I defined a new “Monitors” definition from the “Alerting” menu. Then I define “Trigger” and select my “Smtp Channel” definition.

Everything works correctly in the Alerting mechanism, it finds the word I filtered and runs the trigger function. But it cannot send an email using my smtp information I entered in the “Notification” section. Because it is trying to do smtp authentication.

But I selected the “Encryption Method” as “none” in the “Email Senders” section under the “Notifications” menu. When I click the “send test email” button both in the “Channel” I defined and in the Alerting > Monitors > Triggers I defined, I see that it sends emails. I can also see the emails in my inbox.

But when “Alerting > Monitors > Triggers > Action” automatically sends an email, it tries to establish smtp authentication connection.

I want to send email without smtp auth. There is also “none” option on OpenSearch Panel. I think we have such a possibility. But I see in the log that I get a 503 error. 503 here is an error code representing smtp auth.

I am using an internal smtp relay server and I am not using the smtp auth mechanism. I need to be able to send emails without smtp auth as the Opensearch system allows me.

I think this problem is a bug. How can I solve this problem?

My first OpenSearch forum case;
https://forum.opensearch.org/t/about-mail-sending-problem-on-alerting-notification-section/17786

My second case detailing the problem;
https://forum.opensearch.org/t/problem-sending-mail-without-smtp-auth-on-opensearch-critical/17993

OpenSearch-Dashboards Github Projects Bug Report case;
opensearch-project/alerting-dashboards-plugin#890

screenshot of my smtp definition;

screenshot of my Alerting/Monitors definition;

Log;
Error: Failed running action: OpenSearchStatusException[{"event_status_list": [{"config_id":"Qzjsho0B2TBnxA0NWL9r","config_type":"email","config_name":"Test Email Notification Channel","email_recipient_status":[{"recipient":"[email protected]","delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}],"delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}]}]

---

Opensearch and my linux versions

[root@server6 ~]# yum list installed |grep opensearch
opensearch.x86_64                             2.11.1-1                                  @opensearch-2.x
opensearch-dashboards.x86_64                  2.11.1-1                                  @opensearch-dashboards-2.x

[root@server6 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)

Related sections in my logs

...
[2024-02-21T07:59:52,113][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.112Z, periodEnd: 2024-02-21T04:59:52.112Z, dryrun: true, executionId: _2024-02-21T04:59:52.113348729_cf70ca80-2682-4222-936f-b830f3ba42c1
[2024-02-21T07:59:52,162][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.161Z, periodEnd: 2024-02-21T04:59:52.161Z, dryrun: true
[2024-02-21T07:59:52,163][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.161Z, periodEnd: 2024-02-21T04:59:52.161Z, dryrun: true, executionId: _2024-02-21T04:59:52.163613799_ed76123f-6835-4285-8d35-3967a9c53af0
[2024-02-21T07:59:52,302][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.301Z, periodEnd: 2024-02-21T04:59:52.301Z, dryrun: true
[2024-02-21T07:59:52,302][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.301Z, periodEnd: 2024-02-21T04:59:52.301Z, dryrun: true, executionId: _2024-02-21T04:59:52.302626749_4da076b3-d59b-4128-9666-5af0273ce234
[2024-02-21T07:59:52,339][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.339Z, periodEnd: 2024-02-21T04:59:52.339Z, dryrun: true
[2024-02-21T07:59:52,340][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.339Z, periodEnd: 2024-02-21T04:59:52.339Z, dryrun: true, executionId: _2024-02-21T04:59:52.339981829_405f8627-c4d2-45c8-95be-9f636852f566
[2024-02-21T07:59:56,935][INFO ][o.o.a.t.TransportIndexMonitorAction] [server6] Updating monitor, hq0Xl40BzHB4GwYe75E5, from: {"monitor":{"type":"monitor","schema_version":8,"name":"opensearch Disaster  Error Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707914606685,"schedule":{"period":{"interval":5,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["opensearch-disaster-*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-3h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"level":{"query":"error","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"CzSzp40BPraQSw1iuOEJ","name":"Email Trigger","severity":"1","condition":{"script":{"source":"ctx.results[0].hits.total.value > 1","lang":"painless"}},"actions":[{"id":"notification915423","name":"act","destination_id":"Qzjsho0B2TBnxA0NWL9r","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"Alerting Notification action","lang":"mustache"}}]}}],"last_update_time":1708424762064,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":5},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":3,"timeField":"@timestamp","bucketUnitOfTime":"h","filters":[{"fieldName":[{"label":"level","type":"text"}],"fieldValue":"error","operator":"is"}],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"Email Trigger":{"value":1,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
 to: {"monitor":{"type":"monitor","schema_version":8,"name":"opensearch Disaster  Error Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707914606685,"schedule":{"period":{"interval":5,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["opensearch-disaster-*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-7h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"level":{"query":"error","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"CzSzp40BPraQSw1iuOEJ","name":"Email Trigger","severity":"1","condition":{"script":{"source":"ctx.results[0].hits.total.value > 1","lang":"painless"}},"actions":[{"id":"notification915423","name":"act","destination_id":"Qzjsho0B2TBnxA0NWL9r","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"Alerting Notification action","lang":"mustache"}}]}}],"last_update_time":1708491595830,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":5},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":7,"timeField":"@timestamp","bucketUnitOfTime":"h","filters":[{"fieldName":[{"label":"level","type":"text"}],"fieldValue":"error","operator":"is"}],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"Email Trigger":{"value":1,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
[2024-02-21T07:59:58,104][INFO ][o.o.a.r.RestSearchMonitorAction] [server6] The monitor parsing failed. Will return response as is.
...

2. Subject
At the same time, when it can’t find the word I filtered for the alarm in “Alerting” > “Monitor”, it says that the “Trigger” and “Action” functions are working in the Alerts section and shows that it sent the email. But this is not true either.

My filter works correctly. I am looking for the word “error” in the log data for the last 3 hours. When it really can’t find the word “error” in the last 3 hours, the “History” section goes to green “no alerts”. This works correctly.

But when I change the status of the alarms under the “Alerts” heading at the bottom of the dashboard to “Completed”, it shows it as “Email Trigger”.

In this case, it does not seem to reflect an true result. Because even though there was an alarm trigger, the action gave an error and could not send the email.