Rule with flatline and use_terms_query doesn't work anymore after elastalert upgrade (->v0.1.12)

[Trustys]

Hello everybody,

ES version : 2.4.2

After have upgraded from elastalert v0.0.93 to elastalert v0.1.12, the following alert fail :

name: "##System_Log_Collection_Stopped"
index: graylog_*
type: flatline
query_key: event_src
use_terms_query: true
doc_type: message
threshold: 1
timeframe:
  minutes: 3

filter:
- query:
    query_string:
      query: "is_system_device:true"

realert:
    minutes: 5

When the first event_src stop to send logs, an alert is raised.
But , when a another event_src stop to send log (i.e five minutes later), no alert is raised.

The buckets (two for my test) are well seen by elastalert. elastalert see the two buckets at the beginning, then he notice the first missing bucket, and finally the second (two bucket -> one bucket -> zero bucket)

Now, the strange thing : I tested the same alert on elastalert v0.0.93 (ES version is also 2.4.2) and it works fine : one alert when the first event_src stop to send logs and another one when the second event_src stop to send logs.

With elastalert v0.0.93 ,it seems that realert is not working as I don't have any other alert than the first two alerts .

With elastalert v0.1.12, it seems that realert is working well as I have new alerts every five minutes

So, I would suspect a bug in v0.1.12 around flatline and use_temrs_query but I'm asking if somebody else has noticed the same behavior as the issue could be due to my own specific environment

(without use_terms_query, alert is fine but I have to use use_terms_query for performance reasons)

Thanks for your feedback ! :-)

[Qmando]

I think I see what the issue is. The behavior that changed is that now flatline alerts will keep alerting (#751)

The bug is that silence stashes are created not for a single query key, but for all alerts. Flatline stores the query_key value in a field literally called "key": https://github.com/Yelp/elastalert/blob/master/elastalert/ruletypes.py#L527

But the elastalert code looks for it in the name of the query_key field, https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L782.

I'll have a PR up for this soon.

[Trustys]

OK understood. Waiting for the PR. Thanks for having a look on this @Qmando ! :-)