Skip to content

Commit

Permalink
[ML] AIOps Log Rate Analysis: Fix text field selection (elastic#186176)
Browse files Browse the repository at this point in the history
If we analyse all detected text fields, we might run into performance
issues with the `categorize_text` aggregation. Until this is resolved,
we will rely on a predefined white list of supported text fields, for
now set to `message` and `error.message`.

(cherry picked from commit d3b8123)
  • Loading branch information
walterra committed Jun 18, 2024
1 parent 8fb9cb8 commit 967c5c1
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@
export const fieldCapsPgBenchMock = {
indices: ['.ds-filebeat-8.2.0-2022.06.07-000082'],
fields: {
// The next two fields are not in the original field caps response,
// but are added here to test the logic to ignore fields that are not
// in the white list. It's based on a real world example where the mapping
// included a double mapping of text+integer.
ignore_this_text_field: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'ignore_this_text_field.int': {
integer: { type: 'integer', metadata_field: false, searchable: true, aggregatable: true },
},
'kubernetes.node.uid': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ import { getTotalDocCountRequest } from './get_total_doc_count_request';
// TODO Consolidate with duplicate `fetchPValues` in
// `x-pack/plugins/observability_solution/apm/server/routes/correlations/queries/fetch_duration_field_candidates.ts`

// Supported field names for text fields for log rate analysis.
// If we analyse all detected text fields, we might run into performance
// issues with the `categorize_text` aggregation. Until this is resolved, we
// rely on a predefined white list of supported text fields.
const TEXT_FIELD_WHITE_LIST = ['message', 'error.message'];

const SUPPORTED_ES_FIELD_TYPES = [
ES_FIELD_TYPES.KEYWORD,
ES_FIELD_TYPES.IP,
Expand Down Expand Up @@ -76,7 +82,7 @@ export const fetchIndexInfo = async (
acceptableFields.add(key);
}

if (isTextField) {
if (isTextField && TEXT_FIELD_WHITE_LIST.includes(key)) {
acceptableTextFields.add(key);
}

Expand Down

0 comments on commit 967c5c1

Please sign in to comment.