Hive alerter update

CHANGELOG.md

@@ -18,6 +18,8 @@
 	- Dockerfile now runs as a non-root user "elastalert". Ensure your volumes are accessible by this non-root user.
 	- System packages removed from the Dockerfile: All dev packages, cargo, libmagic. Image size reduced to 250Mb.
 	- `tmp` files and dev packages removed from the final container image.
+- TheHive alerter refactoring - [#142](https://github.com/jertel/elastalert2/pull/142) - @ferozsalam  
+  - See the updated documentation for changes required to alert formatting
 
 ## New features
 - Added support for alerting via Amazon Simple Email System (SES) - [#105](https://github.com/jertel/elastalert2/pull/105) - @nsano-rururu

docs/source/ruletypes.rst

@@ -2600,21 +2600,40 @@ Example usage::
 TheHive
 ~~~~~~~
 
-theHive alert type will send JSON request to theHive (Security Incident Response Platform) with TheHive4py API. Sent request will be stored like Hive Alert with description and observables.
+TheHive alerter can be used to create a new alert in TheHive. The alerter supports adding tags,
+custom fields, and observables from the alert matches and rule data.
 
 Required:
 
-``hive_connection``: The connection details as key:values. Required keys are ``hive_host``, ``hive_port`` and ``hive_apikey``.
+``hive_connection``: The connection details to your instance (see example below for the required syntax).
+Only ``hive_apikey`` is required, ``hive_host`` and ``hive_port`` default to ``http://localhost`` and
+``9000`` respectively.
 
-``hive_alert_config``: Configuration options for the alert.
+``hive_alert_config``: Configuration options for the alert, see example below for structure.
+
+If not supplied, the alert title and description will be populated from the ElastAlert default
+``title`` and ``alert_text`` fields, including any defined ``alert_text_args``.
 
 Optional:
 
-``hive_proxies``: Proxy configuration.
+``tags`` can be populated from the matched record, using the same syntax used in ``alert_text_args``.
+If a record doesn't contain the specified value, the rule itself will be examined for the tag. If
+this doesn't contain the tag either, the tag is attached without modification to the alert. For
+aggregated alerts, all matches are examined individually, and tags generated for each one. All tags
+are then attached to the same alert.
 
-``hive_verify``: Wether or not to enable SSL certificate validation. Defaults to False.
+``customFields`` can also be populated from rule fields as well as matched results. Custom fields
+are only populated once. If an alert is an aggregated alert, the custom field values will be populated
+using the first matched record, before checking the rule. If neither matches, the ``customField.value``
+will be used directly.
 
-``hive_observable_data_mapping``: If needed, matched data fields can be mapped to TheHive observable types using python string formatting.
+``hive_observable_data_mapping``: If needed, matched data fields can be mapped to TheHive
+observable types using the same syntax as ``tags``, described above. The algorithm used to populate
+the observable value is also the same, including the behaviour for aggregated alerts.
+
+``hive_proxies``: Proxy configuration.
+
+``hive_verify``: Whether or not to enable SSL certificate validation. Defaults to False.
 
 Example usage::
 
@@ -2629,20 +2648,24 @@ Example usage::
         https: ''
 
     hive_alert_config:
-      title: 'Title'  ## This will default to {rule[index]_rule[name]} if not provided
-      type: 'external'
-      source: 'elastalert'
-      description: '{match[field1]} {rule[name]} Sample description'
+      customFields:
+        - name: example
+          type: string
+          value: example
+      follow: True
       severity: 2
-      tags: ['tag1', 'tag2 {rule[name]}']
-      tlp: 3
       status: 'New'
-      follow: True
+      source: 'elastalert'
+      description: 'Sample description'
+      tags: ['tag1', 'tag2']
+      title: 'Title'
+      tlp: 3
+      type: 'external'
 
     hive_observable_data_mapping:
-      - domain: "{match[field1]}_{rule[name]}"
-      - domain: "{match[field]}"
-      - ip: "{match[ip_field]}"
+      - domain: agent.hostname
+      - domain: response.domain
+      - ip: client.ip
 
 Twilio
 ~~~~~~
@@ -2709,4 +2732,4 @@ Example usage::
     zbx_sender_host: "zabbix-server"
     zbx_sender_port: 10051
     zbx_host: "test001"
-    zbx_key: "sender_load1"
+    zbx_key: "sender_load1"

elastalert/alerts.py

@@ -3,7 +3,6 @@
 import datetime
 import json
 import os
-import re
 import subprocess
 import sys
 import time
@@ -2043,74 +2042,115 @@ class HiveAlerter(Alerter):
     """
     Use matched data to create alerts containing observables in an instance of TheHive
     """
-
     required_options = set(['hive_connection', 'hive_alert_config'])
 
+    def lookup_field(self, match: dict, field_name: str, default):
+        """Populates a field with values depending on the contents of the Elastalert match
+        provided to it.
+
+        Uses a similar algorithm to that implemented to populate the `alert_text_args`.
+        First checks any fields found in the match provided, then any fields defined in
+        the rule, finally returning the default value provided if no value can be found.
+        """
+        field_value = lookup_es_key(match, field_name)
+        if field_value is None:
+            field_value = self.rule.get(field_name, default)
+
+        return field_value
+
+    # Iterate through the matches, building up a list of observables
+    def load_observable_artifacts(self, match: dict):
+        artifacts = []
+        for mapping in self.rule.get('hive_observable_data_mapping', []):
+            for observable_type, mapping_key in mapping.items():
+                data = self.lookup_field(match, mapping_key, '')
+                artifact = {'tlp': 2,
+                            'tags': [],
+                            'message': None,
+                            'dataType': observable_type,
+                            'data': data}
+                artifacts.append(artifact)
+
+        return artifacts
+
+    def load_custom_fields(self, custom_fields_raw: list, match: dict):
+        custom_fields = {}
+        position = 0
+
+        for field in custom_fields_raw:
+            if (isinstance(field['value'], str)):
+                value = self.lookup_field(match, field['value'], field['value'])
+            else:
+                value = field['value']
+
+            custom_fields[field['name']] = {'order': position, field['type']: value}
+            position += 1
+
+        return custom_fields
+
+    def load_tags(self, tag_names: list, match: dict):
+        tag_values = set()
+        for tag in tag_names:
+            tag_value = self.lookup_field(match, tag, tag)
+            if isinstance(tag_value, list):
+                for sub_tag in tag_value:
+                    tag_values.add(sub_tag)
+            else:
+                tag_values.add(tag_value)
+
+        return tag_values
+
     def alert(self, matches):
+        # Build TheHive alert object, starting with some defaults, updating with any
+        # user-specified config
+        alert_config = {
+            'artifacts': [],
+            'customFields': {},
+            'date': int(time.time()) * 1000,
+            'description': self.create_alert_body(matches),
+            'sourceRef': str(uuid.uuid4())[0:6],
+            'tags': [],
+            'title': self.create_title(matches),
+        }
+        alert_config.update(self.rule.get('hive_alert_config', {}))
 
+        # Iterate through each match found, populating the alert tags and observables as required
+        tags = set()
+        artifacts = []
+        for match in matches:
+            artifacts = artifacts + self.load_observable_artifacts(match)
+            tags.update(self.load_tags(alert_config['tags'], match))
+
+        alert_config['artifacts'] = artifacts
+        alert_config['tags'] = list(tags)
+
+        # Populate the customFields
+        alert_config['customFields'] = self.load_custom_fields(alert_config['customFields'],
+                                                               matches[0])
+
+        # POST the alert to TheHive
         connection_details = self.rule['hive_connection']
 
-        for match in matches:
-            context = {'rule': self.rule, 'match': match}
+        api_key = connection_details.get('hive_apikey', '')
+        hive_host = connection_details.get('hive_host', 'http://localhost')
+        hive_port = connection_details.get('hive_port', 9000)
+        proxies = connection_details.get('hive_proxies', {'http': '', 'https': ''})
+        verify = connection_details.get('hive_verify', False)
 
-            artifacts = []
-            for mapping in self.rule.get('hive_observable_data_mapping', []):
-                for observable_type, match_data_key in mapping.items():
-                    try:
-                        match_data_keys = re.findall(r'\{match\[([^\]]*)\]', match_data_key)
-                        rule_data_keys = re.findall(r'\{rule\[([^\]]*)\]', match_data_key)
-                        data_keys = match_data_keys + rule_data_keys
-                        context_keys = list(context['match'].keys()) + list(context['rule'].keys())
-                        if all([True if k in context_keys else False for k in data_keys]):
-                            artifact = {'tlp': 2, 'tags': [], 'message': None, 'dataType': observable_type,
-                                        'data': match_data_key.format(**context)}
-                            artifacts.append(artifact)
-                    except KeyError:
-                        raise KeyError('\nformat string\n{}\nmatch data\n{}'.format(match_data_key, context))
-
-            alert_config = {
-                'artifacts': artifacts,
-                'caseTemplate': None,
-                'customFields': {},
-                'date': int(time.time()) * 1000,
-                'description': self.create_alert_body(matches),
-                'sourceRef': str(uuid.uuid4())[0:6],
-                'title': '{rule[index]}_{rule[name]}'.format(**context),
-            }
-            alert_config.update(self.rule.get('hive_alert_config', {}))
-            custom_fields = {}
-            for alert_config_field, alert_config_value in alert_config.items():
-                if alert_config_field == 'customFields':
-                    n = 0
-                    for cf_key, cf_value in alert_config_value.items():
-                        cf = {'order': n, cf_value['type']: cf_value['value'].format(**context)}
-                        n += 1
-                        custom_fields[cf_key] = cf
-                elif isinstance(alert_config_value, str):
-                    alert_value = alert_config_value.format(**context)
-                    if alert_config_field in ['severity', 'tlp']:
-                        alert_value = int(alert_value)
-                    alert_config[alert_config_field] = alert_value
-                elif isinstance(alert_config_value, (list, tuple)):
-                    formatted_list = []
-                    for element in alert_config_value:
-                        try:
-                            formatted_list.append(element.format(**context))
-                        except (AttributeError, KeyError, IndexError):
-                            formatted_list.append(element)
-                    alert_config[alert_config_field] = formatted_list
-            if custom_fields:
-                alert_config['customFields'] = custom_fields
-
-            alert_body = json.dumps(alert_config, indent=4, sort_keys=True)
-            req = '{}:{}/api/alert'.format(connection_details['hive_host'], connection_details['hive_port'])
-            headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer {}'.format(connection_details.get('hive_apikey', ''))}
-            proxies = connection_details.get('hive_proxies', {'http': '', 'https': ''})
-            verify = connection_details.get('hive_verify', False)
-            response = requests.post(req, headers=headers, data=alert_body, proxies=proxies, verify=verify)
-
-            if response.status_code != 201:
-                raise Exception('alert not successfully created in TheHive\n{}'.format(response.text))
+        alert_body = json.dumps(alert_config, indent=4, sort_keys=True)
+        req = f'{hive_host}:{hive_port}/api/alert'
+        headers = {'Content-Type': 'application/json',
+                   'Authorization': f'Bearer {api_key}'}
+
+        try:
+            response = requests.post(req,
+                                     headers=headers,
+                                     data=alert_body,
+                                     proxies=proxies,
+                                     verify=verify)
+            response.raise_for_status()
+        except RequestException as e:
+            raise EAException(f"Error posting to TheHive: {e}")
 
     def get_info(self):
 

tests/alerts_test.py

@@ -23,6 +23,7 @@
 from elastalert.alerts import EmailAlerter
 from elastalert.alerts import GitterAlerter
 from elastalert.alerts import GoogleChatAlerter
+from elastalert.alerts import HiveAlerter
 from elastalert.alerts import HTTPPostAlerter
 from elastalert.alerts import JiraAlerter
 from elastalert.alerts import JiraFormattedMatchString
@@ -7073,3 +7074,85 @@ def test_mattermost_ea_exception():
             alert.alert([match])
     except EAException:
         assert True
+
+
+def test_thehive_alerter():
+    rule = {'alert': [],
+            'alert_text': '',
+            'alert_text_type': 'alert_text_only',
+            'description': 'test',
+            'hive_alert_config': {'customFields': [{'name': 'test',
+                                                    'type': 'string',
+                                                    'value': 'test.ip'}],
+                                  'follow': True,
+                                  'severity': 2,
+                                  'source': 'elastalert',
+                                  'status': 'New',
+                                  'tags': ['test.ip'],
+                                  'tlp': 3,
+                                  'type': 'external'},
+            'hive_connection': {'hive_apikey': '',
+                                'hive_host': 'https://localhost',
+                                'hive_port': 9000},
+            'hive_observable_data_mapping': [{'ip': 'test.ip'}],
+            'name': 'test-thehive',
+            'tags': ['a', 'b'],
+            'type': 'any'}
+    rules_loader = FileRulesLoader({})
+    rules_loader.load_modules(rule)
+    alert = HiveAlerter(rule)
+    match = {
+        "test": {
+          "ip": "127.0.0.1"
+        },
+        "@timestamp": "2021-05-09T14:43:30",
+    }
+    with mock.patch('requests.post') as mock_post_request:
+        alert.alert([match])
+
+    expected_data = {
+        "artifacts": [
+            {
+                "data": "127.0.0.1",
+                "dataType": "ip",
+                "message": None,
+                "tags": [],
+                "tlp": 2
+            }
+        ],
+        "customFields": {
+            "test": {
+                "order": 0,
+                "string": "127.0.0.1"
+            }
+        },
+        "description": "\n\n",
+        "follow": True,
+        "severity": 2,
+        "source": "elastalert",
+        "status": "New",
+        "tags": [
+            "127.0.0.1"
+        ],
+        "title": "test-thehive",
+        "tlp": 3,
+        "type": "external"
+    }
+
+    conn_config = rule['hive_connection']
+    alert_url = f"{conn_config['hive_host']}:{conn_config['hive_port']}/api/alert"
+    mock_post_request.assert_called_once_with(
+        alert_url,
+        data=mock.ANY,
+        headers={'Content-Type': 'application/json',
+                 'Authorization': 'Bearer '},
+        verify=False,
+        proxies={'http': '', 'https': ''}
+    )
+
+    actual_data = json.loads(mock_post_request.call_args_list[0][1]['data'])
+    # The date and sourceRef are autogenerated, so we can't expect them to be a particular value
+    del actual_data['date']
+    del actual_data['sourceRef']
+
+    assert expected_data == actual_data