[KQL] Should wildcard queries default to case-insensitive search?

[wylieconlon]

Starting in 7.10, Elasticsearch supports an option to set case_insensitive: true on the wildcard search query. This works internally by rewriting the searches to regular expressions that match upper and lower case characters.

Options for how to expose this

a. Set this flag to be the default in all KQL wildcard searches, without changing the KQL grammar. This has some potential performance issues as described in a comment by @markharwood in the related issue about wildcard fields:

For example - if they support a *foo* style query in the KQL bar and assume, like normal whole-term based queries, that can be run across multiple fields then it may result in slow results or timeouts. Wildcard fields will be fast but hitting other fields which are keyword will involve an expensive linear scan.

b. Only set this flag by default when the user is running wildcard query on wildcard type fields. This would be the most performant option, but it would potentially be confusing to have two different behaviors.

c. Add something to the KQL grammar, like this request to add an UPPER() function to KQL. This could let users enable case insensitive queries as needed. I don't have a proposed grammar.

cc @elastic/kibana-app-arch @markharwood

[markharwood]

No easy answers here:

option a)
Case sensitivity is not important - until it is. Where would the "off" switch be?
I had some benchmarks for case insensitive vs case sensitive on keyword fields here

option b)
Same point about the lack of off switch.
We have tried to make wildcard field == keyword field in terms of behaviour so in this respect it would not be good.

option c)
Adding a switch to the syntax is hard. If it's any consolation I'm struggling with adding this to regexes in Lucene's query parser too.

I don't think KQL/Lucene Query string should seek to expose all the search options available in Lucene - there are ways to resolve these syntaxes' shortcomings which I will address elsewhere.

I'd root for option A - or maybe make case insensitivity an index-time decision by adding the option for normalizers to the wildcard field (cc @jimczi )

[wylieconlon]

@markharwood thanks for the response. It seems like your main concern is that users should be able to choose when to use case-insensitive search, or at least to disable it. What if, instead of adding it to KQL, we added a wildcard query to the Kibana filters editor? It's a more form-based way of building filters and could have a checkbox asking if users want to do a case-insensitive search.

[markharwood]

Hard agree that graphical clause editors (aka filter panels) are the way to simplify the complexity in clause options.

Sadly, the challenge for Kibana is that filter pills can only be ANDed together. The only way of orchestrating custom assemblies of AND/OR logic is currently through text-based KQL. This is why I've been advocating graphical Boolean query builders for some time

[AlonaNadler]

I got multiple issues with the fact Kibana is case sensitive. There are few use cases where it is important but for most users it's an obstacle requires them to know how exactly the value was entered.

I prefer option A

a. Set this flag to be the default in all KQL wildcard searches, without changing the KQL grammar. This has some potential performance issues as described in a comment by @markharwood in the related issue about wildcard fields:

And introduce an advance setting that allows to turn that off

cc: @elastic-jb

[markharwood]

And introduce an advance setting that allows to turn that off

If we can't make the off switch an inline per-query-clause we need to consider the other possible scopes:

1. An option on the user's KQL bar
2. A specific field in the mapping
3. A specific index pattern
4. Kibana-wide

While case sensitivity is not typically important for text it can matter for machine-facing content like

• Base64 encoded values
• Passwords
• Unix-based file names
• Cookies
• Variable names in code or command line switches

We may want to be careful about simply introducing a Kibana-wide switch.

[elastic-jb]

I am option A also. I think the scope question is interesting. Could it be a per-space setting instead of Kibana wide? I was hit by this quite a few times recently working on a demo using NBA data. LeBron James' capital B kept tripping me up. Even after I was aware of it, I still sometimes didn't capitalize the B here and there, and the results were empty. More inclusive feels like a better default.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[GeorgeGkinis]

Hello!

Is this issue related that i can do this in dev console:

GET logs-my-index-grokking/_search
{
  "query": {
    "wildcard": {
      "message": {
        "value":  "*certs chain from Key*"
      }
    }
  }
}

but not in kibana in the KQL bar:
message: "*certs chain from Key*"?

the message field is set as wilcard type.

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.