writing rules by using aggregation of a field

[abhisheksurpur2]

I want to write an elastalert rule where, there is a spike in any particular user's activity i.e., say user xyz, on average logs in 20 times a day but on some days if he crosses double of average, there should be an alert.
Please help me writing this rule. I'm very much new to elastalert.

[jertel]

Have you looked at the example_spike.yaml file yet?

[abhisheksurpur2]

If you're asking to detect for any arbitrary username, not just user xyz, then again that sounds more like a machine learning fit.

yes I'm working on arbitrary username.
If flatline works on single user, why just I can't use query_key to perform aggregate operation on username & make flatline rule work for any arbitrary users?

[abhisheksurpur2]

It sounds like you're asking to detect a specific username of xyz not appearing in the logs for 7 days. How about a flatline rule, where the timeline is set to 7 days, and threshold is set to 0?

If you're asking to detect for any arbitrary username, not just user xyz, then again that sounds more like a machine learning fit.

yes I'm working on arbitrary username.
If flatline works on single user, why just I can't use query_key to perform aggregate operation on username & make flatline rule work for any arbitrary users?

[jertel]

I've not tried that but it sounds reasonable. Let us know if it works. Would be nice to add that as an example.

[abhisheksurpur2]

es_host: .....
es_port: 9200

name: long period gap

type: flatline

index: elastalertdata2

timeframe:
days: 7

threshold: 1

query_key: username

filter:

• query:
  query_string:
  query: "*"

alert:

• "email"

This is the output I'm getting, which shows 0 matches..
elastalert_status - {'rule_name': 'long period gap', 'endtime': datetime.datetime(2021, 5, 4, 4, 52, 18, 780287, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 4, 27, 3, 11, 30, 780287, tzinfo=tzutc()), 'matches': 0, 'hits': 54, '@timestamp': datetime.datetime(2021, 5, 4, 4, 52, 22, 464042, tzinfo=tzutc()), 'time_taken': 3.655336856842041}

Where I'm going wrong?
PS : I've no log entries in last 7 days except yesterday's 54 matches

[nsano-rururu]

When did the rules start working?

https://elastalert2.readthedocs.io/en/latest/elastalert.html?highlight=start#running-elastalert

--start will force ElastAlert to begin querying from the given time, instead of the default, querying from the present. The timestamp should be ISO8601, e.g. YYYY-MM-DDTHH:MM:SS (UTC) or with timezone YYYY-MM-DDTHH:MM:SS-08:00 (PST). Note that if querying over a large date range, no alerts will be sent until that rule has finished querying over the entire time period. To force querying from the current time, use “NOW”.

--end will cause ElastAlert to stop querying at the specified timestamp. By default, ElastAlert will periodically query until the present indefinitely.

Read the document

type: Flatline

Count the number of events at the set interval, and alert if the number of events is less than the threshold

timeframe

Interval to aggregate values used in the logic of the selected type
With this setting, exceptions that occurred during the 7 days will be aggregated.

query_key

If you set the field name, the event will be counted for each set field.

threshold

Threshold to alert