Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] [AIOps] Uses standard analyzer in log pattern analysis to ensure filter in Discover matches correct documents #172188

Merged
merged 11 commits into from
Dec 5, 2023
Merged

[ML] [AIOps] Uses standard analyzer in log pattern analysis to ensure filter in Discover matches correct documents #172188

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6913d0f
[ML] [AIOps] Using standard analyser in pattern analysis
jgowdyelastic Nov 29, 2023
2cae512
fixing test
jgowdyelastic Nov 30, 2023
03c01fc
fixing test
jgowdyelastic Nov 30, 2023
5ca1774
fixing test
jgowdyelastic Nov 30, 2023
e951e68
Merge branch 'main' into using-standard-analyzer-in-pattern-analysis-8
jgowdyelastic Nov 30, 2023
87707d9
fixing test
jgowdyelastic Nov 30, 2023
a56ee25
Merge branch 'main' into using-standard-analyzer-in-pattern-analysis-8
jgowdyelastic Dec 1, 2023
bf2d840
fixing tests
jgowdyelastic Dec 1, 2023
77dac0e
Merge branch 'main' into using-standard-analyzer-in-pattern-analysis-8
jgowdyelastic Dec 5, 2023
ce7a019
Merge branch 'main' into using-standard-analyzer-in-pattern-analysis-8
jgowdyelastic Dec 5, 2023
287e868
adding comment
jgowdyelastic Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions x-pack/plugins/aiops/common/__mocks__/artificial_logs/significant_log_patterns.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ export const significantLogPatterns: SignificantItem[] = [
normalizedScore: 0,
pValue: 0.000001,
score: -13.815510557964274,
total_bg_count: 1975,
total_doc_count: 4669,
total_bg_count: 2528,
total_doc_count: 6650,
type: 'log_pattern',
},
];
64 changes: 63 additions & 1 deletion x-pack/plugins/aiops/common/api/log_categorization/create_category_request.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@
* 2.0.
*/

import type { QueryDslQueryContainer } from '@elastic/elasticsearch/lib/api/types';
import type {
QueryDslQueryContainer,
AggregationsCustomCategorizeTextAnalyzer,
} from '@elastic/elasticsearch/lib/api/types';

import { createRandomSamplerWrapper } from '@kbn/ml-random-sampler-utils';

Expand All @@ -30,6 +33,7 @@ export function createCategoryRequest(
categorize_text: {
field,
size: CATEGORY_LIMIT,
categorization_analyzer: categorizationAnalyzer,
},
aggs: {
hit: {
Expand Down Expand Up @@ -64,3 +68,61 @@ export function createCategoryRequest(
},
};
}

const categorizationAnalyzer: AggregationsCustomCategorizeTextAnalyzer = {
char_filter: ['first_line_with_letters'],
tokenizer: 'standard',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to add a comment here saying:

  1. This is basically the default categorization analyzer but with the standard tokenizer instead of ml_standard.
  2. The ml_standard tokenizer splits tokens in a way that was observed to give better categories in testing many years ago, however, the downside of these better categories is then potential failures to find the original documents when using the category tokens to search for them.
  3. Ideally we'd use the tokenizer from the mappings of the field being categorized, but that's too hard, so using standard is a quick compromise.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment added in 287e868

filter: [
// @ts-expect-error filter type in AggregationsCustomCategorizeTextAnalyzer is incorrect
{
type: 'stop',
stopwords: [
'Monday',
'Tuesday',
'Wednesday',
'Thursday',
'Friday',
'Saturday',
'Sunday',
'Mon',
'Tue',
'Wed',
'Thu',
'Fri',
'Sat',
'Sun',
'January',
'February',
'March',
'April',
'May',
'June',
'July',
'August',
'September',
'October',
'November',
'December',
'Jan',
'Feb',
'Mar',
'Apr',
'May',
'Jun',
'Jul',
'Aug',
'Sep',
'Oct',
'Nov',
'Dec',
'GMT',
'UTC',
],
},
// @ts-expect-error filter type in AggregationsCustomCategorizeTextAnalyzer is incorrect
{
type: 'limit',
max_token_count: '100',
},
],
};
60 changes: 59 additions & 1 deletion x-pack/plugins/aiops/server/routes/log_rate_analysis/queries/fetch_categories.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,65 @@ describe('getCategoryRequest', () => {
random_sampler: { probability: 0.1, seed: 1234 },
aggs: {
categories: {
categorize_text: { field: 'the-field-name', size: 1000 },
categorize_text: {
categorization_analyzer: {
char_filter: ['first_line_with_letters'],
tokenizer: 'standard',
filter: [
{
type: 'stop',
stopwords: [
'Monday',
'Tuesday',
'Wednesday',
'Thursday',
'Friday',
'Saturday',
'Sunday',
'Mon',
'Tue',
'Wed',
'Thu',
'Fri',
'Sat',
'Sun',
'January',
'February',
'March',
'April',
'May',
'June',
'July',
'August',
'September',
'October',
'November',
'December',
'Jan',
'Feb',
'Mar',
'Apr',
'May',
'Jun',
'Jul',
'Aug',
'Sep',
'Oct',
'Nov',
'Dec',
'GMT',
'UTC',
],
},
{
type: 'limit',
max_token_count: '100',
},
],
},
field: 'the-field-name',
size: 1000,
},
aggs: {
hit: {
top_hits: { size: 1, sort: ['the-time-field-name'], _source: 'the-field-name' },
Expand Down