Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KQL] Add support for case-insensitive searches #55378

Closed
Tracked by #166068
joswr1ght opened this issue Jan 21, 2020 · 12 comments
Closed
Tracked by #166068

[KQL] Add support for case-insensitive searches #55378

joswr1ght opened this issue Jan 21, 2020 · 12 comments
Labels
enhancement New value added to drive a business result Feature:KQL KQL Feature:Search Querying infrastructure in Kibana impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL.

Comments

@joswr1ght
Copy link

Describe the feature:

Requesting the implementation of an UPPER() function in KQL to be applied to fields, converting the record content to all uppercase.

UPPER(url.original):*SELECT*

KQL currently lacks a mechanism to perform case-insensitive searches. When using Kibana for system log analysis, attackers can evade detection by mixing case in data that would show up in web logs. For example, a search to identify a SQL injection attack utilizing the SQL UNION keyword might be:

url.original:*UNION*

An attacker who constructs an attack where the HTTP request uses the SQL keyword UnioN would evade detection with this KQL query.

Describe a specific use case for the feature:

  • Ease of use: Analysts can construct KQL queries without knowing the case expectation by converting content to uppercase as desired.
  • Log analysis: Analysts can search for keywords to identify evidence of an attack in system logs while mitigating the opportunity for an adversary to evade detection through mixed case use.

Thank you for your consideration.

-Josh
@Bargs Bargs added Feature:KQL KQL Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Jan 29, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app (Team:KibanaApp)

@Bargs
Copy link
Contributor

Bargs commented Jan 29, 2020

This depends more on how your field was analyzed at ingestion time. If your field is configured to use the Lowercase token filter then your KQL queries on that field will be case insensitive.

@joswr1ght
Copy link
Author

The use case I’m looking for is following the ingestion of Linux auth.log or access.log files using the Apache or Nginx modules with Filebeat. I’m not aware of a Filebeat configure mechanism to change the status of case-sensitivity during import, though I could be mistaken here.

Thanks!

@joswr1ght
Copy link
Author

Following up on this, I don't believe the Filebeat module for Apache/Nginx/IIS web log ingestion offers an option to use the Lowercast token filter. The ability to query data with a KQL UPPER() function would be a useful addition, allowing analysts some post-import flexibility in how the data is evaluated.

@timroes timroes added Team:AppArch and removed Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Feb 20, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-arch (Team:AppArch)

@joswr1ght
Copy link
Author

Just my semi-regular follow-up on this request. Thank you!

@lukasolson lukasolson added the enhancement New value added to drive a business result label Jun 23, 2020
@joswr1ght
Copy link
Author

Hey, just checking in again! You guys do great work, and this would be a fantastic feature to add to Kibana for log analysis. Thank you! 🙏

@joswr1ght
Copy link
Author

Just checking in, any status on getting case-insensitive KQL match functionality?

@wylieconlon wylieconlon mentioned this issue Oct 14, 2020
Closed
@wylieconlon
Copy link
Contributor

@joswr1ght There's been no direct work on this, but in the future release of 7.10, Elasticsearch will offer a case-insensitive option for wildcard searches. I'm proposing that we change KQL to use this by default in this issue, if you want to follow that discussion.

@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort labels Jun 2, 2021
@lukasolson lukasolson changed the title Add UPPER() Function to KQL for Selected Case Insensitive Searches [KQL] Add support for case-insensitive searches Dec 15, 2021
@exalate-issue-sync exalate-issue-sync bot added loe:medium Medium Level of Effort impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. and removed loe:small Small Level of Effort impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. labels Mar 29, 2022
@exalate-issue-sync exalate-issue-sync bot added loe:large Large Level of Effort and removed loe:medium Medium Level of Effort labels Apr 6, 2022
@stratoula stratoula mentioned this issue Apr 18, 2022
Closed
@rwaight rwaight mentioned this issue Jun 9, 2022
Open
@exalate-issue-sync exalate-issue-sync bot added loe:medium Medium Level of Effort and removed loe:large Large Level of Effort labels Jul 5, 2022
@petrklapka petrklapka added the Feature:Search Querying infrastructure in Kibana label Nov 23, 2022
@petrklapka petrklapka added Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. and removed Team:AppServicesSv labels Nov 23, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@willemdh
Copy link

When will we be able to query case insensitive in Kibana KQL? I also see #134143
This seems like very basic search functionality?

@lukasolson lukasolson mentioned this issue Jan 13, 2023
Merged
2 tasks
lukasolson added a commit that referenced this issue Jan 20, 2023
@lukasolson @kibanamachine
[KQL] Add case-insensitive config for term/wildcard queries (#148916)
81d5764 
## Summary

Part of #55378.

Adds a `caseInsensitive` option to the KQL parsing options
(`KueryQueryOptions`). When enabled, keyword queries (both
wildcard<sup>[1]</sup> and term<sup>[2]</sup>) will add the
`case_insensitive` option to enable case-insensitive search.

[1]
https://www.elastic.co/guide/en/elasticsearch/reference/8.6/query-dsl-wildcard-query.html#wildcard-query-field-params
[2]
https://www.elastic.co/guide/en/elasticsearch/reference/8.6/query-dsl-term-query.html#term-field-params


### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <[email protected]>
@davismcphee davismcphee added impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. and removed impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. labels Sep 8, 2023
@ebeahan ebeahan mentioned this issue Jul 17, 2024
Merged
@kertal kertal mentioned this issue Oct 1, 2024
Open
@kertal
Copy link
Member

kertal commented Oct 1, 2024

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.

This works with ES|QL btw:
https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-to_upper

@kertal kertal closed this as not planned Won't fix, can't repro, duplicate, stale Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New value added to drive a business result Feature:KQL KQL Feature:Search Querying infrastructure in Kibana impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@kertal @wylieconlon @timroes @lukasolson @joswr1ght @Bargs @willemdh @elasticmachine @davismcphee @petrklapka