Build inclusive "OR" filters from lists in the UI

[jimcheetham]

As per the question in https://discuss.elastic.co/t/hunting-for-bad-actors-searching-for-multiple-ip-addresses/27095, I often have a dashboard that provides a useful list of values in response to a query; I'd like to be able to simply build these values into a complex "OR/should" filter from the UI.

The example use case is related to hunting bad actor IP addresses; when I find a compromised user account I often get a list of IP addresses that have accessed it; a quick and useful query is to see if any other accounts have been accessed from the same source addresses.

Currently I can manually build a query with "srcip:1.2.3.4 OR srcip:2.3.4.5 ..." but that takes time, and as a query probably isn't the most efficient; this should be a filter.

[rashidkpc]

Duplicate of #3693

[rashidkpc]

Sort of not a dupe. What we'd really want here is a compound filter that acts on an array of values, eg, put array.length terms filters into a bool, but keep it as a single filter object.

[darrenjsykes]

Are we talking about a subquery? i.e. filter where ip matches field_x in subquery(filter to get a list of ips)

[jimcheetham]

That sounds like it would be a great feature in general, and improve
functionality.

It wouldn't be fully flexible for the sort of forensic investigations I get
involved in, I'd rather see each element in the subquery independently
listed, so I can select/deselect them as I work through them deciding if
they are 'ok' or if they represent more study.
On 9 Sep 2015 01:38, "darrenjsykes" [email protected] wrote:

Are we talking about a subquery? i.e. filter where ip matches field_x in
subquery(filter to get a list of ips)

—
Reply to this email directly or view it on GitHub
#4673 (comment).

[l34n]

+1, trying to match up a list of known company owned IPs with intelligence feeds of bad actors.

[polarpal99]

Perhaps the expression language of Canvas can be expanded to add a checkboxfilter and checkboxfilterControl and element. These could be OR-based filters that would could be ANDed like the dropdownfilter and timefilter elements do today. Would this be a reasonable way to implement what people have been crying about for several releses now? #HowLongToImplement ?

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[archon810]

Just tried to query nginx logs in Kibana by the top 1000 IPs in https://www.abuseipdb.com, using source.ip:IP1 or source.ip:IP2... format, and the search is still going after many minutes. It'd be great to implement not just this feature but also any related speed optimizations so that querying for 1000 IPs doesn't run endlessly and pretty much never return.

Here's an example of what Inspect shows for such a query with only 4 IPs. No wonder ES has trouble with it when we get to 1000 values due to all the nesting: https://gist.github.com/6dc0915601699a37a763d4dc99263a16.

Is there really no equivalent of MySQL's IN parameter so that we could say something like source.ip:(IP1, IP2, IP3, etc)?

[timroes]

@archon810 This seems like a separate issue from this here, around how we generate the Elasticsearch DSL from the KQL you enter, while this issue is about adding UI capabilities in the filter UI for it. Could you please open a separate issue for this, with the information you shared here?

[archon810]

Opened up a separate ticket above. I'm not sure if it's an issue with how DSL is built from KQL or if ES isn't capable of a more optimized search, but I'm all ears to test any solutions.

[ppisljar]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)