Adding use_terms_query for cardinality rules

[DrewHorrigan]

My current use case is a rule to detect port scans, running on a lot of traffic data. Currently we use a cardinality rule with a query_key containing 2 fields, but it downloads full events and counts the cardinality on the side of Elastalert, which makes the rule very heavy, sometimes to the point where it crashes or times out (I tried to use "include" to only retrieve the required fields, but this does not seem to help much).

This seems to be an appropriate use for a Terms aggregation; if query_key is used, it can be a series of embedded Terms aggregations, or, alternatively, a Composite aggregation. I tested the performance of the latter, and it seems to be faster than the full query by orders of magnitude (it also has an added advantage of a possibly unlimited number of terms). I've been wondering why it wasn't present in the original Elastalert - is there a technical reason preventing this?

[jertel]

Metric Aggregations is available for similar purposes. Have you looked to see if this can meet your needs?

[DrewHorrigan]

I haven't, actually - I guess I overlooked it. Will definitely try it, thanks.

[JeffAshton]

Maybe related:

#699