Error finding recent pending alerts..."No mapping found for [alert_time] in order to sort on"

[kmfreder1]

We moved to Elastalert2 from Elastalert quite some time ago, (thank you for keeping this project going!) and have implemented it across just shy of a dozen different SIEM systems where we are running Wazuh-Indexer 4.7.5, which is based on Opensearch 2.8.0.

It seems to be working as expected from what we can tell as we get email and slack alerts from our rules, however, I notice that there is an error upon every 5 minute run through the rules in /var/log/elastalert.log on each of our implementations of Elastalert2 as follows:

Oct 16 10:16:50 sample elastalert[1411]: INFO:elastalert:background configuration change check run at 2024-10-16 15:16 UTC
Oct 16 10:16:50 sample elastalert[1411]: INFO:elastalert:Disabled rules are: []
Oct 16 10:16:50 sample elastalert[1411]: INFO:elastalert:Sleeping for 299.999685 seconds
Oct 16 10:16:50 sample elastalert[1411]: WARNING:elasticsearch:POST https://localhost:9201/elastalert_status/_search?size=1000 [status:400 request:0.007s]
Oct 16 10:16:50 sample elastalert[1411]: ERROR:elastalert:Error finding recent pending alerts: RequestError(400, 'search_phase_execution_exception', 'No mapping found for [alert_time] in order to sort on') {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2024-10-14T15:16:50.018033Z', 'to': '2024-10-16T15:16:50.018068Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
Oct 16 10:16:50 sample elastalert[1411]: Traceback (most recent call last):
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elastalert/elastalert.py", line 1483, in find_recent_pending_alerts
Oct 16 10:16:50 sample elastalert[1411]:     res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
Oct 16 10:16:50 sample elastalert[1411]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
Oct 16 10:16:50 sample elastalert[1411]:     return func(*args, params=params, headers=headers, **kwargs)
Oct 16 10:16:50 sample elastalert[1411]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elastalert/__init__.py", line 147, in search
Oct 16 10:16:50 sample elastalert[1411]:     results = self.transport.perform_request(
Oct 16 10:16:50 sample elastalert[1411]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elasticsearch/transport.py", line 392, in perform_request
Oct 16 10:16:50 sample elastalert[1411]:     raise e
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elasticsearch/transport.py", line 358, in perform_request
Oct 16 10:16:50 sample elastalert[1411]:     status, headers_response, data = connection.perform_request(
Oct 16 10:16:50 sample elastalert[1411]:                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elasticsearch/connection/http_requests.py", line 199, in perform_request
Oct 16 10:16:50 sample elastalert[1411]:     self._raise_error(response.status_code, raw_data)
Oct 16 10:16:50 sample elastalert[1411]:   File "/usr/local/lib/python3.11/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
Oct 16 10:16:50 sample elastalert[1411]:     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
Oct 16 10:16:50 sample elastalert[1411]: elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'No mapping found for [alert_time] in order to sort on')
Oct 16 10:16:50 sample elastalert[1411]: INFO:elastalert:background alerts thread 0 pending alerts sent at 2024-10-16 15:16 UTC

Directly after this error, it cycles through the rules and triggers alerts as we would otherwise expect.

I've done some looking around via Google and on this forum and I found some suggestion about adding timestamp_field: indicating what we are using as the timestamp field, timestamp in our case. However, after implementing this across our rules, the error continues to occur at the beginning of every rule check cycle. I did not see anything else in the documentation or discussions that seemed to apply to our case.

Any help or suggestions would be much appreciated.

[jertel]

It looks like the elastalert_* indices are incorrect. You can diagnose this and inspect the indices to see what type of field alert_time is. Or you can delete the elastalert_* indices, and restart ElastAlert 2 container. Upon restart, it will attempt to recreate the indices. You will lose any pending alerts (alerts that either failed to send and are awaiting retry, or alerts that haven't reached their aggregation threshold), but otherwise it's not a significant loss in most cases.

If you're not using the container you will need to manually run the elastalert_create_index script as described in the docs.

[kmfreder1]

I'll give that a shot and report back my findings here shortly. Thank you for your quick response!

[kmfreder1]

Unfortunately, that didn't work. I removed all elastalert* indices from Wazuh Indexer and on the second run, the same error occurred. How likely is it that this is actually impacting overall functionality?

We are using the container and are on the newest version of Elastalert2.

[kmfreder1]

A bit more information, when I look at one of the implementations where we are not getting these errors, I see mappings for quite a number of fields for the elastalert_status index, but on implementations where the errors occur, there are no field mappings at all. This is even after I stopped the elastalert container, deleted all elastalert* indices and restarted the elastalert container. This is what I get in Dev Tools when I look at the elastalert_status:

GET elastalert_status

{
  "elastalert_status": {
    "aliases": {},
    "mappings": {},
    "settings": {
      "index": {
        "creation_date": "1729111017125",
        "number_of_shards": "1",
        "number_of_replicas": "0",
        "uuid": "J6bzXy2pRmeXePu7W1tvjQ",
        "version": {
          "created": "136297827"
        },
        "provided_name": "elastalert_status"
      }
    }
  }
}

[kmfreder1]

So, not only were mappings not in place for the elastalert_status index, but we also found that no documents were being written to the index when alerts were triggered, as we observe on other hosts where Elastalert is working properly.

So, I pulled the mappings from a host were Elastalert2 was not giving these errors and manually put the mappings in place. After doing this, restarting the Elastalert docker container and waiting a few minutes for the second check of the rules, the error I first mentioned above no longer occurred.

However, even with that, events were still not being written to the elastalert_status index when an alert was generated.

Then I found, our email facility, Postfix, was not working properly. We also use Slack for alerting, so we had not double-checked to see if emails were coming, nor would we have known this was related to these errors.

Once I fixed Postfix so it was configured properly and emailing successfully, suddenly, events were written to the elastalert_status index when one of the rules triggered an alert.

I am not 100% sure how writing mappings and/or documents to the elastalert_status index would be dependent upon email functionality, but these are the steps that we took to correct the issue.

For just in case this may help someone else, I have attached a file with the command I ran in Dev Tools to put the mappings in place.

elastalert_status-mappings-command.txt