Merge pull request #1104 from akusei/http_post2_alternate_jinja_root

Http post2 alternate jinja root

CHANGELOG.md

@@ -7,6 +7,9 @@
 ## New features
 - [Graylog GELF] Alerter added. [#1050](https://github.com/jertel/elastalert2/pull/1050) - @malinkinsa
 - [TheHive] Format `title`, `type`, and `source` with dynamic lookup values - [#1092](https://github.com/jertel/elastalert2/pull/1092) - @fandigunawan
+- [HTTP POST2] `http_post2_payload` and `http_post2_headers` now support multiline JSON strings for better control over jinja templates - @akusei
+- [HTTP POST2] This alerter now supports the use of `jinja_root_name` - @akusei
+- [Rule Testing] The data file passed with `--data` can now contain a single JSON document or a list of JSON objects - @akusei
 
 ## Other changes
 - [Docs] Clarify Jira Cloud authentication configuration - [94f7e8c](https://github.com/jertel/elastalert2/commit/94f7e8cc98d59a00349e3b23acd8a8549c80dbc8) - @jertel
@@ -16,7 +19,10 @@
 - Modify schema to allow string and boolean for `*_ca_certs` to allow for one to specify a cert bundle for SSL certificate verification - [#1082](https://github.com/jertel/elastalert2/pull/1082) - @goggin
 - Fix UnicodeEncodeError in PagerDutyAlerter - [#1091](https://github.com/jertel/elastalert2/pull/1091) - @nsano-rururu
 - The scan_entire_timeframe setting, when used with use_count_query or use_terms_query will now scan entire timeframe on subsequent rule runs - [#1097](https://github.com/jertel/elastalert2/pull/1097) - @rschirin
-
+- Add new unit tests to cover changes in the HTTP POST2 alerter - @akusei  
+- [Docs] Updated HTTP POST2 documentation to outline changes with payloads, headers and multiline JSON strings - @akusei
+- [HTTP POST2] Additional error checking around rendering and dumping payloads/headers to JSON - @akusei
+
 # 2.9.0
 
 ## Breaking changes

docs/source/ruletypes.rst

@@ -2445,11 +2445,11 @@ Required:
 
 Optional:
 
-``http_post2_payload``: List of keys:values to use for the payload of the HTTP Post. You can use {{ field }} (Jinja2 template) in the key and the value to reference any field in the matched events (works for nested ES fields and nested payload keys). If not defined, all the Elasticsearch keys will be sent. Ex: `"description_{{ my_field }}": "Type: {{ type }}\\nSubject: {{ title }}"`.
+``http_post2_payload``: A JSON string or list of keys:values to use for the payload of the HTTP Post. You can use {{ field }} (Jinja2 template) in the key and the value to reference any field in the matched events (works for nested ES fields and nested payload keys). If not defined, all the Elasticsearch keys will be sent. Ex: `"description_{{ my_field }}": "Type: {{ type }}\\nSubject: {{ title }}"`. When field names use dot notation or reserved characters, _data can be used to access these fields. If _data conflicts with your top level data, use jinja_root_name to change its name.
 
 ``http_post2_raw_fields``: List of keys:values to use as the content of the POST. Example - ip:clientip will map the value from the clientip field of Elasticsearch to JSON key named ip. This field overwrite the keys with the same name in `http_post2_payload`.
 
-``http_post2_headers``: List of keys:values to use for as headers of the HTTP Post. You can use {{ field }} (Jinja2 template) in the key and the value to reference any field in the matched events (works for nested fields). Ex: `"Authorization": "{{ user }}"`. Headers `"Content-Type": "application/json"` and `"Accept": "application/json;charset=utf-8"` are present by default, you can overwrite them if you think this is necessary.
+``http_post2_headers``: A JSON string or list of keys:values to use for as headers of the HTTP Post. You can use {{ field }} (Jinja2 template) in the key and the value to reference any field in the matched events (works for nested fields). Ex: `"Authorization": "{{ user }}"`. Headers `"Content-Type": "application/json"` and `"Accept": "application/json;charset=utf-8"` are present by default, you can overwrite them if you think this is necessary. When field names use dot notation or reserved characters, _data can be used to access these fields. If _data conflicts with your top level data, use jinja_root_name to change its name.
 
 ``http_post2_proxy``: URL of proxy, if required. only supports https.
 
@@ -2461,6 +2461,32 @@ Optional:
 
 ``http_post2_ignore_ssl_errors``: By default ElastAlert 2 will verify SSL certificate. Set this option to ``True`` if you want to ignore SSL errors.
 
+.. note:: Due to how values are rendered to JSON, the http_post2_headers and http_post2_payload fields require single quotes where quotes are required for Jinja templating. This only applies when using the YAML key:value pairs. Any quotes can be used with the new JSON string format. See below for examples of how to properly use quotes as well as an example of the new JSON string formatting.
+
+Incorrect usage with double quotes::
+
+    alert: post2
+    http_post2_url: "http://example.com/api"
+    http_post2_payload:
+      # this will result in an error as " is escaped to \"
+      description: 'hello {{ _data["name"] }}'
+      # this will result in an error as " is escaped to \"
+      state: '{{ ["low","medium","high","critical"][event.severity] }}'
+    http_post2_headers:
+      authorization: Basic 123dr3234
+      X-custom-type: '{{type}}'
+
+Correct usage with single quotes::
+
+    alert: post2
+    http_post2_url: "http://example.com/api"
+    http_post2_payload:
+      description: hello {{ _data['name'] }}
+      state: "{{ ['low','medium','high','critical'][event.severity] }}"
+    http_post2_headers:
+      authorization: Basic 123dr3234
+      X-custom-type: '{{type}}'
+
 Example usage::
 
     alert: post2
@@ -2474,6 +2500,27 @@ Example usage::
       authorization: Basic 123dr3234
       X-custom-type: {{type}}
 
+Example usage with json string formatting::
+
+    alert: post2
+    jinja_root_name: _new_root
+    http_post2_url: "http://example.com/api"
+    http_post2_payload: |
+      {
+        "description": "An event came from IP {{ _new_root["client.ip"] }}",
+        "username": "{{ _new_root['username'] }}"
+        {%- for k, v in some_field.items() -%}
+        ,"{{ k }}": "changed_{{ v }}"
+        {%- endfor -%}
+      }
+    http_post2_raw_fields:
+      ip: clientip
+    http_post2_headers: |
+      {
+        "authorization": "Basic 123dr3234",
+        "X-custom-{{key}}": "{{type}}"
+      }
+
 Jira
 ~~~~
 

elastalert/alerters/httppost2.py

@@ -1,15 +1,18 @@
 import json
+from json import JSONDecodeError
 
 import requests
-from jinja2 import Template
+from jinja2 import Template, TemplateSyntaxError
 from requests import RequestException
 
 from elastalert.alerts import Alerter, DateTimeEncoder
 from elastalert.util import lookup_es_key, EAException, elastalert_logger
 
+
 def _json_escape(s):
     return json.encoder.encode_basestring(s)[1:-1]
 
+
 def _escape_all_values(x):
     """recursively rebuilds, and escapes all strings for json, the given dict/list"""
     if isinstance(x, dict):
@@ -21,6 +24,14 @@ def _escape_all_values(x):
     return x
 
 
+def _render_json_template(template, match):
+    if not isinstance(template, str):
+        template = json.dumps(template)
+    template = Template(template)
+
+    return json.loads(template.render(**match))
+
+
 class HTTPPost2Alerter(Alerter):
     """ Requested elasticsearch indices are sent by HTTP POST. Encoded with JSON. """
     required_options = frozenset(['http_post2_url'])
@@ -39,15 +50,40 @@ def __init__(self, rule):
         self.post_ca_certs = self.rule.get('http_post2_ca_certs')
         self.post_ignore_ssl_errors = self.rule.get('http_post2_ignore_ssl_errors', False)
         self.timeout = self.rule.get('http_post2_timeout', 10)
+        self.jinja_root_name = self.rule.get('jinja_root_name', None)
 
     def alert(self, matches):
         """ Each match will trigger a POST to the specified endpoint(s). """
         for match in matches:
             match_js_esc = _escape_all_values(match)
-            payload = match if self.post_all_values else {}
-            payload_template = Template(json.dumps(self.post_payload))
-            payload_res = json.loads(payload_template.render(**match_js_esc))
-            payload = {**payload, **payload_res}
+            args = {**match_js_esc}
+            if self.jinja_root_name:
+                args[self.jinja_root_name] = match_js_esc
+
+            try:
+                field = 'payload'
+                payload = match if self.post_all_values else {}
+                payload_res = _render_json_template(self.post_payload, args)
+                payload = {**payload, **payload_res}
+
+                field = 'headers'
+                header_res = _render_json_template(self.post_http_headers, args)
+                headers = {
+                    "Content-Type": "application/json",
+                    "Accept": "application/json;charset=utf-8",
+                    **header_res
+                }
+            except TemplateSyntaxError as e:
+                raise ValueError(f"HTTP Post 2: The value of 'http_post2_{field}' has an invalid Jinja2 syntax. "
+                                 f"Please check your template syntax: {e}")
+
+            except JSONDecodeError as e:
+                raise ValueError(f"HTTP Post 2: The rendered value for 'http_post2_{field}' contains invalid JSON. "
+                                 f"Please check your template syntax: {e}")
+
+            except Exception as e:
+                raise ValueError(f"HTTP Post 2: An unexpected error occurred with the 'http_post2_{field}' value. "
+                                 f"Please check your template syntax: {e}")
 
             for post_key, es_key in list(self.post_raw_fields.items()):
                 payload[post_key] = lookup_es_key(match, es_key)
@@ -59,14 +95,6 @@ def alert(self, matches):
             if self.post_ignore_ssl_errors:
                 requests.packages.urllib3.disable_warnings()
 
-            header_template = Template(json.dumps(self.post_http_headers))
-            header_res = json.loads(header_template.render(**match_js_esc))
-            headers = {
-                "Content-Type": "application/json",
-                "Accept": "application/json;charset=utf-8",
-                **header_res
-            }
-
             for key, value in headers.items():
                 if type(value) in [type(None), list, dict]:
                     raise ValueError(f"HTTP Post 2: Can't send a header value which is not a string! "

elastalert/test_rule.py

@@ -14,7 +14,6 @@
 
 from elastalert.config import load_conf
 from elastalert.elastalert import ElastAlerter
-from elastalert.util import EAException
 from elastalert.util import elasticsearch_client
 from elastalert.util import lookup_es_key
 from elastalert.util import ts_now
@@ -354,6 +353,8 @@ def run_elastalert(self, rule, conf):
             if not self.data:
                 return None
             try:
+                if isinstance(self.data, dict):
+                    self.data = [self.data]
                 self.data.sort(key=lambda x: x[timestamp_field])
                 self.starttime = self.str_to_ts(self.data[0][timestamp_field])
                 self.endtime = self.str_to_ts(self.data[-1][timestamp_field]) + datetime.timedelta(seconds=1)