Email Alerts Using Elastalert

Email-Alerting-with-Elastalert/README.md

@@ -0,0 +1,261 @@
+# Email Alerts example :
+
+Basically one needs a gold or platinum package to activate/send email alerts via ELK using X-pack watchers but in this example i'm going to demonstrate how to configure ELK stack(basic license) to send free email alerts using [ElastAlert](https://github.com/Yelp/elastalert).
+
+`Note: This whole example is demostrated assuming that you are using Linux/Ubuntu based operating system.`
+
+### Tested Enivironment:
+
+* Ubuntu 18/20
+* Elastic Stack 7.8.0
+* ElastAlert (Latest Version)
+
+
+### Dependencies you need have in your host :
+
+* Java 8 or above 
+
+To check which version you have 
+
+```cmd
+    java --version
+```
+
+Output : (In my case i'm having java 11)
+
+```cmd 
+    openjdk 11.0.8 2020-07-14
+    OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
+    OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
+```
+
+* Python (version 3+ recommended)
+
+```cmd
+    sudo apt-get install -y python3 
+    sudo apt-get install -y python3-pip python3-dev libffi-dev libssl-dev 
+```
+To verify the installation 
+
+```cmd
+        python3 --version
+
+```
+Output :  
+
+``` cmd 
+        Python 3.8.2 (version no can be varied) 
+```
+The above output tells that python has been installed successfully.
+
+* ElasticSearch 7.8.0 (Linux x86_64)  : [Download from here](https://www.elastic.co/downloads/past-releases/elasticsearch-7-8-0)
+
+
+* Kibana 7.8.0 (Linux 64-bit)          : [Download from here](https://www.elastic.co/downloads/past-releases/kibana-7-8-0)
+
+
+* Logstash 7.8.0 (TAR.GZ)               : [Download from here](https://www.elastic.co/downloads/past-releases/logstash-7-8-0)
+
+
+Now visit the directory where you have cloned , downloaded and 
+execute the below command to find your tar files
+
+```cmd 
+ls -lh | grep tar.gz
+```
+
+Output :
+
+```cmd
+-rwxrwxrwx 1 vvk vvk 305M Jul  7 19:38 elasticsearch-7.8.0-linux-x86_64.tar.gz
+-rwxrwxrwx 1 vvk vvk 319M Jul  7 19:39 kibana-7.8.0-linux-x86_64.tar.gz
+-rwxrwxrwx 1 vvk vvk 160M Jul  7 19:39 logstash-7.8.0.tar.gz
+```
+
+Extract them one by one :
+
+```cmd 
+    Format :  tar -xvf <tar file>
+
+    tar -xvf elasticsearch-7.8.0-linux-x86_64.tar.gz
+    tar -xvf kibana-7.8.0-linux-x86_64.tar.gz
+    tar -xuf logstash-7.8.0.tar.gz
+```
+
+* ElastAlert (Download Latest code) :
+
+```cmd 
+     git clone https://github.com/Yelp/elastalert.git
+```
+Now your directory should have the following files :
+
+![Downloaded Dependencies](https://raw.githubusercontent.com/vvvk-gh/examples/master/Email-Alerting-with-Elastalert/Images/Downloads.png)
+
+
+# Introduction 
+
+Before making any new changes, let's understand why, where and how Elastalert is useful and configured.
+
+## What is ElastAlert ?
+
+ElastAlert is an opensource framework for alerting duplicates, system spikes and for many other patterns present in the data/documents of Elasticsearch.
+
+## How it works ?
+
+We define a rule in Elastalert (which is basically a query) -> if a match found in Elasticsearch data -> Elastalert sends an alert to your gmail  
+
+# Configuration changes :
+
+1. Elasticsearch :
+    Replace your `elasticsearch-7.8.0-linux-x86_64/config/elasticsearch.yml` file with `elasticsearch.yml` 
+
+    - Save it
+
+    - Run it 
+    ```cmd
+        ./bin/elasticsearch
+    ```
+    - Verify it 
+        by opening [localhost:9200](http://localhost:9200) in your browser
+        which will be showing your cluster details
+
+2. Kibana :
+    Replace your `kibana-7.8.0-linux-x86_64/config/kibana.yml` file with `kibana.yml` 
+
+    - Save it
+
+    - Run it
+
+    ```cmd
+        ./bin/kibana
+    ```
+
+    - Verify it 
+        by opening [localhost:5601](http://localhost:5601) in your browser
+        which will open your kibana
+
+3. ElastAlert 
+    * go to cloned project
+
+    ```cmd
+        cd elastalert
+    ```
+    do the following
+
+    ```cmd
+        sudo pip3 install "setuptools>=11.3" 
+        sudo pip3 install pyOpenSSL 
+        sudo python3 setup.py install 
+        sudo pip3 install "elasticsearch>=5.0.0" 
+    ```
+
+    * Copy config.yaml.example into config.yaml  
+
+    ```cmd
+
+        cp config.example.yaml config.yaml
+
+    ```
+    and replace new copied `config.yaml` with `config.yaml` in this project and save it.
+
+    * Create Elastalert Indices 
+
+        ```cmd
+            elastalert-create-index
+        ```
+       Output :
+
+        ```cmd
+            Elastic Version: 7.8.0
+            Reading Elastic 6 index mappings:
+            Reading index mapping 'es_mappings/6/silence.json'
+            Reading index mapping 'es_mappings/6/elastalert_status.json'
+            Reading index mapping 'es_mappings/6/elastalert.json'
+            Reading index mapping 'es_mappings/6/past_elastalert.json'
+            Reading index mapping 'es_mappings/6/elastalert_error.json'
+            New index elastalert_status created
+            Done!
+         ```
+
+    * Writing the test rules  
+
+        rules are defined in example_rules folder and we are going to use only `frequency based` test rule in this example which means
+
+        > Alert an email if a match found at X events/documents in Y time 
+
+        replace the `./example_rules/example_frequency.yaml` with `example_frequency.yaml` in this project and also download and add 'stmp_auth_file.txt' in the same directory `./example_rules/`
+
+        Now, modify the both files in a way that serves your needs
+
+    example_frequency.yaml
+
+    ```YAML
+        email:
+            - "[email protected]"
+        smtp_host: "smtp.gmail.com."
+        smtp_port: 465
+        smtp_ssl: true
+        from_addr: "[email protected]"
+        smtp_auth_file: '/path/to/file/smtp_auth_file.txt'
+    ```
+    smtp_auth_file.txt
+
+    ```txt
+        user : [email protected]
+        password: yourgmailpassword
+    ```
+
+4. Logstash 
+
+    Add the `elasalert_logstash.conf` into your `logstash-7.8.0/config/` and also
+    download the sample logs file `cpustruck_syslogs.log` in the same path
+
+    - Save it
+
+    - Run it
+
+    ```cmd
+         ./bin/logstash -f /path/to/elastalert_logstash.conf
+    ```
+
+    This will push the sample logs to elasticsearch and also prints them to console
+
+5. Test Run Elastalert 
+
+    ```cmd
+         elastalert-test-rule example_rules/example_frequency.yaml
+    ```
+
+    Output :
+
+    ![Image of TestRun](https://raw.githubusercontent.com/vvvk-gh/examples/master/Email-Alerting-with-Elastalert/Images/Testrun.png)
+
+6. Running ElastAlert
+
+    ```teriminal
+        python3 -m elastalert.elastalert --verbose --rule example_frequency.yaml
+    ```
+
+If you get a match it will alert to configured email.
+
+
+![Emailnotification](https://raw.githubusercontent.com/vvvk-gh/examples/master/Email-Alerting-with-Elastalert/Images/Emailnotificatin.png)
+
+![Image of EmailNotification](https://raw.githubusercontent.com/vvvk-gh/examples/master/Email-Alerting-with-Elastalert/Images/Emailalert.png)
+
+
+Its additional supports alerts via 
+   * Command
+   * JIRA
+   * OpsGenie
+   * SNS
+   * HipChat
+   * Slack
+   * Telegram
+   * GoogleChat
+   * Debug
+   * Stomp
+   * theHive
+
+For more details 
+visit the [documentation](https://elastalert.readthedocs.io/en/latest/elastalert.html)

Email-Alerting-with-Elastalert/config.yaml

@@ -0,0 +1,115 @@
+# This is the folder that contains the rule yaml files
+# Any .yaml file will be loaded as a rule
+rules_folder: example_rules
+
+# How often ElastAlert will query Elasticsearch
+# The unit can be anything from weeks to seconds
+run_every:
+  minutes: 1
+
+# ElastAlert will buffer results from the most recent
+# period of time, in case some log sources are not in real time
+buffer_time:
+  minutes: 15
+
+# The Elasticsearch hostname for metadata writeback
+# Note that every rule can have its own Elasticsearch host
+es_host: "localhost"
+
+# The Elasticsearch port
+es_port: 9200
+
+# The AWS region to use. Set this when using AWS-managed elasticsearch
+#aws_region: us-east-1
+
+# The AWS profile to use. Use this if you are using an aws-cli profile.
+# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
+# for details
+#profile: test
+
+# Optional URL prefix for Elasticsearch
+#es_url_prefix: elasticsearch
+
+# Connect with TLS to Elasticsearch
+#use_ssl: True
+
+# Verify TLS certificates
+#verify_certs: True
+
+# GET request with body is the default option for Elasticsearch.
+# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
+# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
+# for details
+#es_send_get_body_as: GET
+
+# Option basic-auth username and password for Elasticsearch
+#es_username: someusername
+#es_password: somepassword
+
+# Use SSL authentication with client certificates client_cert must be
+# a pem file containing both cert and key for client
+#verify_certs: True
+#ca_certs: /path/to/cacert.pem
+#client_cert: /path/to/client_cert.pem
+#client_key: /path/to/client_key.key
+
+# The index on es_host which is used for metadata storage
+# This can be a unmapped index, but it is recommended that you run
+# elastalert-create-index to set a mapping
+writeback_index: elastalert_status
+writeback_alias: elastalert_alerts
+
+# If an alert fails for some reason, ElastAlert will retry
+# sending the alert until this time period has elapsed
+alert_time_limit:
+  days: 2
+
+# Custom logging configuration
+# If you want to setup your own logging configuration to log into
+# files as well or to Logstash and/or modify log levels, use
+# the configuration below and adjust to your needs.
+# Note: if you run ElastAlert with --verbose/--debug, the log level of
+# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
+#logging:
+#  version: 1
+#  incremental: false
+#  disable_existing_loggers: false
+#  formatters:
+#    logline:
+#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
+#
+#    handlers:
+#      console:
+#        class: logging.StreamHandler
+#        formatter: logline
+#        level: DEBUG
+#        stream: ext://sys.stderr
+#
+#      file:
+#        class : logging.FileHandler
+#        formatter: logline
+#        level: DEBUG
+#        filename: elastalert.log
+#
+#    loggers:
+#      elastalert:
+#        level: WARN
+#        handlers: []
+#        propagate: true
+#
+#      elasticsearch:
+#        level: WARN
+#        handlers: []
+#        propagate: true
+#
+#      elasticsearch.trace:
+#        level: WARN
+#        handlers: []
+#        propagate: true
+#
+#      '':  # root logger
+#        level: WARN
+#          handlers:
+#            - console
+#            - file
+#        propagate: false

Email-Alerting-with-Elastalert/cpustruck_syslogs.log

@@ -0,0 +1,13 @@
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24 
+Aug 24 00:11:29 KCA-MYSYS-CS07 : [20418418.755494] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s!  [ksoftirqd/26:140] host.name:filebeat-xsgjn @timestamp:Aug 24, 2020 @ 01:41:30.806 ecs.version:1.0.0 input.type:log  agent.id:ef31817b-54c1-48d5-ae62-1ba90a9d9aab agent.version:7.1.1 agent.type:filebeat agent.hostname:filebeat-xsgjn agent.ephemeral_id:ce5f1571-7bda-4d18-8bb8-fdfc266e02cc log.offset:297,607,057 log.file.path:/var/log/syslog @version:1  tags:syslog, beats_input_codec_plain_applied _id:pQDO4nIB0xt2OZ2Qofa8 _type:_doc _index:syslog-7.1.1-2020.08.24