[ML] [AIOps] Uses standard analyzer in log pattern analysis to ensure filter in Discover matches correct documents #172188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jgowdyelastic
requested review from
walterra,
alvarezmelissa87 and
peteharverson
jgowdyelastic
added
release_note:fix
:ml
Feature:ML/AIOps
|
Pinging @elastic/ml-ui (:ml) |
droberts195
approved these changes
Dec 5, 2023
|
|
||
| const categorizationAnalyzer: AggregationsCustomCategorizeTextAnalyzer = { | ||
| char_filter: ['first_line_with_letters'], | ||
| tokenizer: 'standard', |
There was a problem hiding this comment.
It would be good to add a comment here saying:
- This is basically the default categorization analyzer but with the
standardtokenizer instead ofml_standard. - The
ml_standardtokenizer splits tokens in a way that was observed to give better categories in testing many years ago, however, the downside of these better categories is then potential failures to find the original documents when using the category tokens to search for them. - Ideally we'd use the tokenizer from the mappings of the field being categorized, but that's too hard, so using
standardis a quick compromise.
There was a problem hiding this comment.
Comment added in 287e868
💚 Build Succeeded
Metrics [docs]Async chunks
History
To update your PR or re-run it, just comment with: |
jgowdyelastic
changed the title
peteharverson
changed the title
3 tasks
walterra
added a commit
that referenced
this pull request
Feb 12, 2024
## Summary Fixes #176387. The `standard` analyser for log pattern analysis introduced in #172188 might return patterns that mess with the identifying of significant patterns across time ranges, for example if a pattern matches different parts of a date or time. This adds an update that allows to set the analyser for log rate analysis to `ml_standard` but keep `standard` for log pattern analysis. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this pull request
Feb 15, 2024
…ic#176587) ## Summary Fixes elastic#176387. The `standard` analyser for log pattern analysis introduced in elastic#172188 might return patterns that mess with the identifying of significant patterns across time ranges, for example if a pattern matches different parts of a date or time. This adds an update that allows to set the analyser for log rate analysis to `ml_standard` but keep `standard` for log pattern analysis. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
fkanout
pushed a commit
to fkanout/kibana
that referenced
this pull request
Mar 4, 2024
…ic#176587) ## Summary Fixes elastic#176387. The `standard` analyser for log pattern analysis introduced in elastic#172188 might return patterns that mess with the identifying of significant patterns across time ranges, for example if a pattern matches different parts of a date or time. This adds an update that allows to set the analyser for log rate analysis to `ml_standard` but keep `standard` for log pattern analysis. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #169523
The
categorize_textagg uses theml_standardtokenizer by default which produces slightly different tokens compared to thestandardtokenizer, which is the default used for search.This means the category key (which is comprised of these tokens) will occasionally not match any documents when it is used as a filter in Discover to find docs in a category.
This PR ensures the
standardtokenizer is always used in the pattern analysis query.A future enhancement would be to check which analyzer is specified in the mappings for the source field and to use that instead of unconditionally using
standard. However for an initial fix, using thestandardanalyzer will be more likely to match the results from the majority of searches.