Merge branch 'master' into master

CHANGELOG.md

@@ -7,7 +7,23 @@
 - [Alertmanager] Add alertmanager resolve timeout configuration option - [#1187](https://github.com/jertel/elastalert2/pull/1187) - @eveningcafe
 
 ## Other changes
-- TBD
+- [Docs] Clarify docs to state that alert_time_limit should not be 0 - [#1208](https://github.com/jertel/elastalert2/pull/1208) - @jertel
+
+# 2.12.0
+
+## Breaking changes
+- None
+
+## New features
+- Add initial support for EQL - [#1189](https://github.com/jertel/elastalert2/pull/1189) - @jertel
+- Add `fields` parameter to rules to be able to pull in runtimes fields, and more. [#1193](https://github.com/jertel/elastalert2/pull/1193) - @Goggin
+- Add EQL support to elastalert-test-rule utility - [#1195](https://github.com/jertel/elastalert2/pull/1195) - @jertel
+
+## Other changes
+- Add support for Kibana 8.8 for Kibana Discover - [#1184](https://github.com/jertel/elastalert2/pull/1184) - @nsano-rururu
+- Upgrade pylint 2.17.0 to 2.17.4, pytest 7.2.2 to 7.3.1, pytest-xdist 3.2.0 to 3.3.1, sphinx 6.1.3 to 6.2.1, sphinx_rtd_theme == 1.2.2 - [#1194](https://github.com/jertel/elastalert2/pull/1194) - @nsano-rururu
+- Upgrade to Tox 4 - [#1196](https://github.com/jertel/elastalert2/pull/1196) - @jertel
+- Log message when exiting due to --end param being in the past - [#1199](https://github.com/jertel/elastalert2/pull/1199) - @jertel
 
 # 2.11.0
 

chart/elastalert2/Chart.yaml

@@ -1,8 +1,8 @@
 apiVersion: v1
 description: Automated rule-based alerting for Elasticsearch
 name: elastalert2
-version: 2.11.0
-appVersion: 2.11.0
+version: 2.12.0
+appVersion: 2.12.0
 home: https://github.com/jertel/elastalert2
 sources:
 - https://github.com/jertel/elastalert2

chart/elastalert2/README.md

@@ -49,7 +49,7 @@ The command removes all the Kubernetes components associated with the chart and
 | Parameter                                    | Description                                                                                                                   | Default                                                  |
 |----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
 | `image.repository`                           | docker image                                                                                                                  | jertel/elastalert2                                       |
-| `image.tag`                                  | docker image tag                                                                                                              | 2.11.0                                                    |
+| `image.tag`                                  | docker image tag                                                                                                              | 2.12.0                                                    |
 | `image.pullPolicy`                           | image pull policy                                                                                                             | IfNotPresent                                             |
 | `image.pullSecret`                           | image pull secret                                                                                                             | ""                                                       |
 | `podAnnotations`                             | Annotations to be added to pods                                                                                               | {}                                                       |

chart/elastalert2/values.yaml

@@ -39,7 +39,7 @@ image:
   # docker image
   repository: jertel/elastalert2
   # docker image tag
-  tag: 2.11.0
+  tag: 2.12.0
   pullPolicy: IfNotPresent
   pullSecret: ""
 

docs/source/recipes/writing_filters.rst

@@ -125,3 +125,30 @@ Below is a more complex example for Elasticsearch 7.x, provided by a `community
                        uid: charlie
                    - match:
                        menu_item: "burrito pasta salad pizza"
+
+EQL (Event Query Language)
+**************************
+
+EQL is partially supported as of version 2.12.0. To use EQL, include a filter item as follows::
+
+    filter:
+    - eql: any where machine.os == "win 8"
+
+Note that only one ``eql`` filter can be defined in a filter.
+
+It is also possible to use standard query filters in combination with EQL filters::
+
+    filter:
+    - eql: any where machine.os == "win 8"
+    - query:
+        query_string:
+            query: "test.field: 123"
+
+EQL is only partially supported due to the following limitations:
+
+- Cannot be used with aggregation rule types.
+- Cannot be used with blacklist/whitelist rule types.
+- Cannot be used with percentage match rule types.
+- Cannot be used with ``use_count_query`` property.
+- Does not support scrolling, so large result sets may have unexpected results. Be sure to filter your queries thoroughly to avoid returning excessive numbers of events.
+- Not supported with OpenSearch

docs/source/ruletypes.rst

@@ -104,6 +104,8 @@ Rule Configuration Cheat Sheet
 +--------------------------------------------------------------+           |
 | ``include`` (list of strs, default ["*"])                    |           |
 +--------------------------------------------------------------+           |
+| ``fields`` (list of strs, no default)                        |           |
++--------------------------------------------------------------+           |
 | ``filter`` (ES filter DSL, no default)                       |           |
 +--------------------------------------------------------------+           |
 | ``max_query_size`` (int, default global max_query_size)      |           |
@@ -584,6 +586,13 @@ include
 fields, along with '@timestamp', ``query_key``, ``compare_key``, and ``top_count_keys``  are included, if present.
 (Optional, list of strings, default all fields)
 
+fields
+^^^^^^
+
+``fields``: A list of fields that should be included in query results and passed to rule types and alerts. If ``_source_enabled`` is False,
+only these fields and those from ``include`` are included.  When ``_source_enabled`` is True, these are in addition to source.  This is used
+for runtime fields, script fields, etc.  This only works with Elasticsearch version 7.11 and newer.  (Optional, list of strings, no default)
+
 top_count_keys
 ^^^^^^^^^^^^^^
 
@@ -716,7 +725,7 @@ kibana_discover_version
 The currently supported versions of Kibana Discover are:
 
 - `7.0`, `7.1`, `7.2`, `7.3`, `7.4`, `7.5`, `7.6`, `7.7`, `7.8`, `7.9`, `7.10`, `7.11`, `7.12`, `7.13`, `7.14`, `7.15`, `7.16`, `7.17`
-- `8.0`, `8.1`, `8.2`, `8.3`, `8.4`, `8.5`, `8.6`, `8.7`
+- `8.0`, `8.1`, `8.2`, `8.3`, `8.4`, `8.5`, `8.6`, `8.7`, `8.8` 
 
 ``kibana_discover_version: '7.15'``
 
@@ -978,9 +987,17 @@ and missing or misconfigured fields.
 
 ``--count-only``: Only find the number of matching documents and list available fields. ElastAlert 2 will not be run and documents will not be downloaded.
 
-``--days N``: Instead of the default 1 day, query N days. For selecting more specific time ranges, you must run ElastAlert 2 itself and use ``--start``
+``--days N``: Instead of the default 1 day, query N days. For selecting more specific time ranges, use ``--start``
 and ``--end``.
 
+``--start <timestamp>`` The starting date/time of the search filter's time range. The timestamp is formatted as
+``YYYY-MM-DDTHH:MM:SS`` (UTC) or with timezone ``YYYY-MM-DDTHH:MM:SS-XX:00``
+(UTC-XX). If ``timeframe`` is specified, defaults to the ending time - timeframe. Otherwise defaults to ending time - 1 day.
+
+``--end <timestamp>`` The ending date/time of the search filter's time range. The timestamp is formatted as
+``YYYY-MM-DDTHH:MM:SS`` (UTC) or with timezone ``YYYY-MM-DDTHH:MM:SS-XX:00``
+(UTC-XX). Defaults to the current time.
+
 ``--save-json FILE``: Save all documents downloaded to a file as JSON. This is useful if you wish to modify data while testing or do offline
 testing in conjunction with ``--data FILE``. A maximum of 10,000 documents will be downloaded.
 
@@ -997,6 +1014,8 @@ guaranteed to have the exact same results as with Elasticsearch. For example, an
    and flatline require a minimum elapsed time before they begin alerting, based on their timeframe. In addition, use_count_query and
    use_terms_query rely on run_every to determine their resolution. This script uses a fixed 5 minute window, which is the same as the default.
 
+   Also, EQL filters do not support counts, so the output relating to counts may show N/A (Not Applicable).
+
 
 .. _ruletypes:
 

docs/source/running_elastalert.rst

@@ -80,7 +80,7 @@ elastalert2 container image on `Docker Hub <https://hub.docker.com/r/jertel/elas
 
 Be aware that the ``latest`` tag of the image represents the latest commit into
 the master branch. If you prefer to upgrade more slowly you will need utilize a
-versioned tag, such as ``2.11.0`` instead, or ``2`` if you are comfortable with
+versioned tag, such as ``2.12.0`` instead, or ``2`` if you are comfortable with
 always using the latest released version of ElastAlert 2.
 
 A properly configured config.yaml file must be mounted into the container during
@@ -292,7 +292,7 @@ connections
 ``writeback_index`` is the name of the index in which ElastAlert 2 will store
 data. We will create this index later.
 
-``alert_time_limit`` is the retry window for failed alerts.
+``alert_time_limit`` is the retry window for failed alerts. Must be greater than zero.
 
 Save the file as ``config.yaml``
 

elastalert/__init__.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 import copy
-import time
+import elastalert.eql as eql
 
 from elasticsearch import Elasticsearch
 from elasticsearch import RequestsHttpConnection
@@ -74,3 +74,84 @@ def resolve_writeback_index(self, writeback_index, doc_type):
         elif doc_type == 'elastalert_error':
             return writeback_index + '_error'
         return writeback_index
+
+
+    @query_params(
+        "_source",
+        "_source_excludes",
+        "_source_includes",
+        "allow_no_indices",
+        "allow_partial_search_results",
+        "analyze_wildcard",
+        "analyzer",
+        "batched_reduce_size",
+        "ccs_minimize_roundtrips",
+        "default_operator",
+        "df",
+        "docvalue_fields",
+        "expand_wildcards",
+        "explain",
+        "from_",
+        "ignore_throttled",
+        "ignore_unavailable",
+        "lenient",
+        "max_concurrent_shard_requests",
+        "pre_filter_shard_size",
+        "preference",
+        "q",
+        "request_cache",
+        "rest_total_hits_as_int",
+        "routing",
+        "scroll",
+        "search_type",
+        "seq_no_primary_term",
+        "size",
+        "sort",
+        "stats",
+        "stored_fields",
+        "suggest_field",
+        "suggest_mode",
+        "suggest_size",
+        "suggest_text",
+        "terminate_after",
+        "timeout",
+        "track_scores",
+        "track_total_hits",
+        "typed_keys",
+        "version",
+    )
+    def search(self, body=None, index=None, doc_type=None, params=None, headers=None):
+        # This implementation of search is nearly identical to the base class with the following exceptions:
+        # 1. If the request body contains an EQL query, the body will be restructured to support the EQL API.
+        # 2. The path will be set to the EQL API endpoint, if #1 is true.
+        # 3. The scroll and _source_includes params will be dropped if #1 is true, since the EQL API doesn't support them.
+        # 4. The size param will be moved to a body parameter instead of a top-level param if #1 is true.
+        # 5. The results will be converted from EQL API format into the standard search format.
+
+        # from is a reserved word so it cannot be used, use from_ instead
+        if "from_" in params:
+            params["from"] = params.pop("from_")
+
+        path = _make_path(index, doc_type, "_search")
+        eql_body = eql.format_request(body)
+        if eql_body is not None:
+            path = path.replace('/_search', '/_eql/search')
+            body = eql_body
+            if 'size' in params:
+                body['size'] = int(params.pop('size'))
+            if 'scroll' in params:
+                params.pop('scroll')
+            if '_source_includes' in params:
+                params.pop('_source_includes')
+
+        results = self.transport.perform_request(
+            "POST",
+            path,
+            params=params,
+            headers=headers,
+            body=body,
+        )
+
+        eql.format_results(results);
+
+        return results

elastalert/elastalert.py

@@ -363,6 +363,9 @@ def get_hits(self, rule, starttime, endtime, index, scroll=False):
             query['stored_fields'] = rule['include']
             extra_args = {}
 
+        if rule.get('fields', None) is not None:
+            query['fields'] = rule['fields']
+
         try:
             if scroll:
                 res = self.thread_data.current_es.scroll(scroll_id=rule['scroll_id'], scroll=scroll_keepalive)
@@ -1129,7 +1132,9 @@ def start(self):
             if self.args.end:
                 endtime = ts_to_dt(self.args.end)
 
-                if next_run.replace(tzinfo=dateutil.tz.tzutc()) > endtime:
+                next_run_dt = next_run.replace(tzinfo=dateutil.tz.tzutc())
+                if next_run_dt > endtime:
+                    elastalert_logger.info("End time '%s' falls before the next run time '%s', exiting." % (endtime, next_run_dt))
                     exit(0)
 
             if next_run < datetime.datetime.utcnow():

elastalert/eql.py

@@ -0,0 +1,52 @@
+# A dict containing EQL would resemble the following:
+# {'query': {'bool': {'filter': {'bool': {'must': [{'range': {'@timestamp': {'gt': 'yyyy...', 'lte': 'yyyy...'}}}, {'eql': 'process where process.name == "regsvr32.exe"'}]}}}}, 'sort': [{'@timestamp': {'order': 'asc'}}]}
+def format_request(body):
+    query = body.get('query')
+    if not query:
+        return None
+
+    query_bool = query.get('bool')
+    if not query_bool:
+        return None
+
+    filter = query_bool.get('filter')
+    if not filter:
+        return None
+
+    filter_bool = filter.get('bool')
+    if not filter_bool:
+        return None
+
+    filter_bool_must = filter_bool.get('must')
+    if not filter_bool_must:
+        return None
+
+    other_filters = []
+    eql = None
+    for f in filter_bool_must:
+        if f.get('eql'):
+            eql = f['eql']
+        else:
+            other_filters.append(f)
+
+    if eql:
+        new_body = {'filter': { 'bool': { 'must': other_filters }}, 'query': eql}
+        return new_body
+
+    return None
+
+def format_results(results):
+    hits = results.get('hits')
+    if not hits:
+        return results
+
+    events = hits.get('events')
+    if events is None:
+        return results
+
+    # relabel events as hits, for consistency
+    events = hits.pop('events')
+    hits['hits'] = events
+    results['eql'] = True
+
+    return results

elastalert/kibana_discover.py

@@ -16,7 +16,7 @@
 
 kibana_versions = frozenset([
         '7.0', '7.1', '7.2', '7.3', '7.4', '7.5', '7.6', '7.7', '7.8', '7.9', '7.10', '7.11', '7.12', '7.13', '7.14', '7.15', '7.16', '7.17', 
-        '8.0', '8.1', '8.2', '8.3', '8.4', '8.5', '8.6', '8.7'
+        '8.0', '8.1', '8.2', '8.3', '8.4', '8.5', '8.6', '8.7', '8.8'
         ])
 
 def generate_kibana_discover_url(rule, match):

elastalert/loaders.py

@@ -347,6 +347,7 @@ def load_options(self, rule, conf, filename, args=None):
         rule.setdefault('description', "")
         rule.setdefault('jinja_root_name', "_data")
         rule.setdefault('query_timezone', "")
+        rule.setdefault('fields', None)
 
         # Set timestamp_type conversion function, used when generating queries and processing hits
         rule['timestamp_type'] = rule['timestamp_type'].strip().lower()
@@ -393,6 +394,9 @@ def _dt_to_ts_with_format(dt):
         if 'include' in rule and type(rule['include']) != list:
             raise EAException('include option must be a list')
 
+        if 'fields' in rule and rule['fields'] is not None and type(rule['fields']) != list:
+            raise EAException('fields option must be a list')
+
         raw_query_key = rule.get('query_key')
         if isinstance(raw_query_key, list):
             if len(raw_query_key) > 1:

elastalert/schema.yaml

@@ -255,6 +255,7 @@ properties:
     items: *filter
 
   include: {type: array, items: {type: string}}
+  fields: {type: array, item: {type: string}}
   top_count_keys: {type: array, items: {type: string}}
   top_count_number: {type: integer}
   raw_count_keys: {type: boolean}
@@ -280,7 +281,7 @@ properties:
   generate_kibana_discover_url: {type: boolean}
   shorten_kibana_discover_url: {type: boolean}
   kibana_discover_app_url: {type: string}
-  kibana_discover_version: {type: string, enum: ['8.7', '8.6', '8.5', '8.4', '8.3', '8.2', '8.1', '8.0', '7.17', '7.16', '7.15', '7.14', '7.13', '7.12', '7.11', '7.10', '7.9', '7.8', '7.7', '7.6', '7.5', '7.4', '7.3', '7.2', '7.1', '7.0']}
+  kibana_discover_version: {type: string, enum: ['8.8', '8.7', '8.6', '8.5', '8.4', '8.3', '8.2', '8.1', '8.0', '7.17', '7.16', '7.15', '7.14', '7.13', '7.12', '7.11', '7.10', '7.9', '7.8', '7.7', '7.6', '7.5', '7.4', '7.3', '7.2', '7.1', '7.0']}
   kibana_discover_index_pattern_id: {type: string, minLength: 1}
   kibana_discover_columns: {type: array, items: {type: string, minLength: 1}, minItems: 1}
   kibana_discover_from_timedelta: *timedelta