Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Praeco Docker smtp settings error #245

Closed
S733 opened this issue Sep 9, 2020 · 14 comments
Closed

Praeco Docker smtp settings error #245

S733 opened this issue Sep 9, 2020 · 14 comments
Labels
question Further information is requested

Comments

@S733
Copy link

S733 commented Sep 9, 2020

Hi, I am getting the following error when setting up email alerts using praeco docker setup.

ERROR elastalert-server: ProcessController: ERROR:root:Error while running alert email: Error connecting to SMTP host:[Errno -2] Name does not resolve

config/elastalert.yaml

alert:
- "email"

email:
- "[email protected]"
smtp_host: "smtp.gmail.com"
smtp_port: 465
smtp_ssl : true
from_addr: "[email protected]"
smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

is this how the smtp settings are configured ?.

From inside the praeco_elastalter container curl smtp.gmail.com return results so something else is the error, or the above config is not using by the server. Any solution to this issue.

@nsano-rururu
Copy link
Collaborator

@nsano-rururu nsano-rururu added the question Further information is requested label Sep 9, 2020
@vtdat
Copy link
Contributor

vtdat commented Sep 10, 2020

Hi, I am getting the following error when setting up email alerts using praeco docker setup.

ERROR elastalert-server: ProcessController: ERROR:root:Error while running alert email: Error connecting to SMTP host:[Errno -2] Name does not resolve

config/elastalert.yaml

alert:
- "email"

email:
- "[email protected]"
smtp_host: "smtp.gmail.com"
smtp_port: 465
smtp_ssl : true
from_addr: "[email protected]"
smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

is this how the smtp settings are configured ?.

From inside the praeco_elastalter container curl smtp.gmail.com return results so something else is the error, or the above config is not using by the server. Any solution to this issue.

Have you tried nslookup/curl inside elastalert-server container? please check permission and configurations on your /etc/resolv.conf and /etc/hosts files.

@S733
Copy link
Author

S733 commented Sep 11, 2020

Got it working, the smtp settings given in config/elastalert.yaml not working. SMTP settings working after inputing the following inside rules/BaseRule.config

smtp_host: "smtp.gmail.com"
smtp_port: 465
smtp_ssl : true
smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Sep 11, 2020

Got it working, the smtp settings given in config/elastalert.yaml not working. SMTP settings working after inputing the following inside rules/BaseRule.config

smtp_host: "smtp.gmail.com"
smtp_port: 465
smtp_ssl : true
smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

It works according to Praeco's specifications.
https://github.com/johnsusek/praeco/blob/master/src/store/config/index.js#L927

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Sep 12, 2020

Google account in advance → Apps that can access the account → Allow less secure apps: Disable → Enable
I specified the Gmail address with from_addr and confirmed the operation and it worked without problems.

キャプチャ

/home/sano/dkwork2/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--kibana
|  |--config
|  |  |--kibana.yml
|--praeco
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--nginx_config
|  |  |--default.conf
|  |  |--nginx.conf
|  |--public
|  |  |--praeco.config.json
|  |--pass
|  |  |--smtp_auth_user.yaml
|  |--rule_templates
|  |--rules
|  |  |--a.yaml
|  |  |--BaseRule.config

Dockerfiles/Dockerfile.elastalert

FROM praecoapp/elastalert-server:latest

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx512m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.9.0
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
 
  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert-server:3.0.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./praeco/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./praeco/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./praeco/rules:/opt/elastalert/rules
      - ./praeco/rule_templates:/opt/elastalert/rule_templates
      - ./praeco/pass/smtp_auth_user.yaml:/opt/elastalert/pass/smtp_auth_user.yaml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  praeco:
    container_name: praeco
    image: praecoapp/praeco:latest
    ports:
      - 8080:8080
    depends_on:
      - elastalert
    restart: always
    volumes:
      - ./praeco/public/praeco.config.json:/var/www/html/praeco.config.json
      - ./praeco/nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./praeco/nginx_config/default.conf:/etc/nginx/conf.d/default.conf
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

praeco/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

praeco/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

praeco/config/api.config.json

Default settings

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "praeco_elastalert_status"
}

praeco/config/elastalert.yaml

Default settings

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: praeco_elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

profile: default

praeco/public/praeco.config.json

Default settings

{
  "appUrl": "http://127.0.0.1:8080",
  "errorLoggerUrl": "",
  "hidePreconfiguredFields": []
}

praeco/pass/smtp_auth_user.yaml

"xxxx" in the email address and password part is not the value actually set

user: '[email protected]'
password: 'xxxx' 

praeco/rules/BaseRule.config

smtp_host: 'smtp.gmail.com'
smtp_port: 587
smtp_auth_file: '/opt/elastalert/pass/smtp_auth_user.yaml'

praeco/rules/a.yaml

"xxxx" in the email address and password part is not the value actually set

__praeco_query_builder: >-
  {"query":{"logicalOperator":"all","children":[{"type":"query-builder-rule","query":{"rule":"message","selectedOperator":"contains","selectedOperand":"message","value":"Quit"}}]}}
alert:
  - email
alert_subject: a
alert_subject_args: []
alert_text: b
alert_text_args: []
email:
  - [email protected]
filter:
  - query:
      query_string:
        query: 'message:Quit'
from_addr: [email protected]
import: BaseRule.config
index: mariadblog-*
is_enabled: true
name: a
num_events: 1
realert:
  minutes: 1
terms_size: 50
timeframe:
  minutes: 1
timestamp_field: '@timestamp'
timestamp_type: iso
type: frequency
use_strftime_index: false

setting

cd /home/sano/dkwork2/es
chmod 777 es/data
chmod -R 777 praeco/rules praeco/rule_templates
docker-compose up -d

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Sep 12, 2020

When I set from_addr to a non-gmail address (in this case, set it to yahoo.co.jp) and to to the gmail address, an email arrived at the gmail address.
キャプチャ

/home/sano/dkwork2/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--kibana
|  |--config
|  |  |--kibana.yml
|--praeco
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--nginx_config
|  |  |--default.conf
|  |  |--nginx.conf
|  |--public
|  |  |--praeco.config.json
|  |--pass
|  |  |--smtp_auth_user.yaml
|  |--rule_templates
|  |--rules
|  |  |--a.yaml
|  |  |--BaseRule.config

Dockerfiles/Dockerfile.elastalert

FROM praecoapp/elastalert-server:latest

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx512m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.9.0
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
 
  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert-server:3.0.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./praeco/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./praeco/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./praeco/rules:/opt/elastalert/rules
      - ./praeco/rule_templates:/opt/elastalert/rule_templates
      - ./praeco/pass/smtp_auth_user.yaml:/opt/elastalert/pass/smtp_auth_user.yaml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  praeco:
    container_name: praeco
    image: praecoapp/praeco:latest
    ports:
      - 8080:8080
    depends_on:
      - elastalert
    restart: always
    volumes:
      - ./praeco/public/praeco.config.json:/var/www/html/praeco.config.json
      - ./praeco/nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./praeco/nginx_config/default.conf:/etc/nginx/conf.d/default.conf
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

praeco/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

praeco/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

praeco/config/api.config.json

Default settings

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "es_username": "",
  "es_password": "",
  "es_ssl": false,
  "writeback_index": "praeco_elastalert_status"
}

praeco/config/elastalert.yaml

Default settings

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: praeco_elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

profile: default

praeco/public/praeco.config.json

Default settings

{
  "appUrl": "http://127.0.0.1:8080",
  "errorLoggerUrl": "",
  "hidePreconfiguredFields": []
}

praeco/pass/smtp_auth_user.yaml

"xxxx" in the email address and password part is not the value actually set

user: '[email protected]'
password: 'xxxx' 

praeco/rules/BaseRule.config

smtp_host: 'smtp.mail.yahoo.co.jp'
smtp_port: 465 
smtp_ssl: true
smtp_auth_file: '/opt/elastalert/pass/smtp_auth_user.yaml'

praeco/rules/a.yaml

"xxxx" in the email address and password part is not the value actually set

__praeco_query_builder: >-
  {"query":{"logicalOperator":"all","children":[{"type":"query-builder-rule","query":{"rule":"message","selectedOperator":"contains","selectedOperand":"message","value":"Quit"}}]}}
alert:
  - email
alert_subject: a
alert_subject_args: []
alert_text: b
alert_text_args: []
email:
  - [email protected]
filter:
  - query:
      query_string:
        query: 'message:Quit'
from_addr: [email protected]
import: BaseRule.config
index: mariadblog-*
is_enabled: true
name: a
num_events: 1
realert:
  minutes: 1
terms_size: 50
timeframe:
  minutes: 1
timestamp_field: '@timestamp'
timestamp_type: iso
type: frequency
use_strftime_index: false

setting

cd /home/sano/dkwork2/es
chmod 777 es/data
chmod -R 777 praeco/rules praeco/rule_templates
docker-compose up -d

@nsano-rururu
Copy link
Collaborator

As far as I can see from the ElastAlert docs, it should be like this ...

If the specified Gmail account uses the two-step authentication process, you need to enter the app password generated in Google's account settings in the password field for SMTP authentication.

alert:
 - email
email:
 - "<mailId>@gmail.com"
smtp_host: "smtp.gmail.com" 
smtp_port: 587 
smtp_ssl: false
from_addr: "<mailId>@gmail.com"
smtp_auth_file: '/opt/elastalert/smtp/smtp_auth_user.yaml"

smtp_auth_user.yaml

user: "<mailId>@gmail.com"
password: "<password>"

If your Gmail account doesn't use two-step verification, you'll need to enable insecure app access in your Google account settings.

alert:
 - email
email:
 - "<mailId>@gmail.com"
smtp_host: "smtp.gmail.com" 
smtp_port: 465 
smtp_ssl: true
from_addr: "<mailId>@gmail.com"
smtp_auth_file: '/opt/elastalert/smtp/smtp_auth_user.yaml"

smtp_auth_user.yaml

user: "<mailId>@gmail.com"
password: "<password>"

@S733
Copy link
Author

S733 commented Sep 15, 2020

I got the same error in AWS SES smtp settings, so it's not related to gmail insecure app. As I said earlier I got it working when I moved the smtp settings to the praeco/rules/BaseRule.config insted of praeco/config/elastalert.yaml. If the BaseRule.config is where the smtp settings is required please update the document accordingly, as I couldn't find any relavent documentation for the email smtp configuration.

@nsano-rururu
Copy link
Collaborator

Is that so. I don't know anymore so I won't investigate for you

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Sep 15, 2020

elastalert is not compatible with AWS SES. Therefore, praeco does not support it either.
Yelp/elastalert#1594

@vtdat
Copy link
Contributor

vtdat commented Sep 15, 2020

I got the same error in AWS SES smtp settings, so it's not related to gmail insecure app. As I said earlier I got it working when I moved the smtp settings to the praeco/rules/BaseRule.config insted of praeco/config/elastalert.yaml. If the BaseRule.config is where the smtp settings is required please update the document accordingly, as I couldn't find any relavent documentation for the email smtp configuration.

Did you try remove smtp options in BaseRule.config, I believe those in elastalert.yml got overrode

@nsano-rururu
Copy link
Collaborator

@S733

Is this issue still going on?

@nsano-rururu
Copy link
Collaborator

Close. Reopens when communication resumes

@nsano-rururu
Copy link
Collaborator

nsano-rururu commented Nov 14, 2020

I got the same error in AWS SES smtp settings, so it's not related to gmail insecure app. As I said earlier I got it working when I moved the smtp settings to the praeco/rules/BaseRule.config insted of praeco/config/elastalert.yaml. If the BaseRule.config is where the smtp settings is required please update the document accordingly, as I couldn't find any relavent documentation for the email smtp configuration.

It was necessary to enable Gmail's insecure app permissions. Please understand it even if you do not understand it because it is an actual movement.

1
2
キャプチャ

Repository owner locked and limited conversation to collaborators Oct 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants