Skip to content

Commit

Permalink
[8.12] [Bug][Investigations] - Fix slow timeline queries (elastic#176838
Browse files Browse the repository at this point in the history
) (elastic#176956)

# Backport

This will backport the following commits from `main` to `8.12`:
- [[Bug][Investigations] - Fix slow timeline queries
(elastic#176838)](elastic#176838)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Michael
Olorunnisola","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-02-14T21:02:20Z","message":"[Bug][Investigations]
- Fix slow timeline queries (elastic#176838)\n\n## Summary\r\n\r\n**Version
Affected: 8.11.x, 8.12.0, 8.12.1**\r\n\r\n### Background\r\n\r\nThe ID
field necessary to track long running timeline search
strategy\r\nqueries was no longer being passed to ES search after work
in 8.11. This\r\nled to what looked like long running timeline queries,
but in reality\r\nwere queries being repeated due to the ID not being
tracked. This pr\r\nre-introduces the ID field necessary for long
running timeline search\r\nstrategies in security
solution\r\n\r\n**Views Affected:**\r\n - Timeline tabs (query,
correlation, pinned)\r\n - Explore events tables (hosts, users,
network)\r\n - Rule preview table\r\n \r\n\r\nPre-fix:\r\n\r\nObserver
the changing ID's for the `timelineSearchStrategy`
`eventsAll`\r\nqueries.\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/5731d310-d3ed-452d-8c34-783b2cfe76e1\r\n\r\n\r\nPost-fix:\r\n\r\nObserver
the same ID for the `timelineSearchStrategy`
`eventsAll`\r\nqueries.\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/a20d4b28-2748-4475-a257-96133bb8efc7\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"68bdd7cb275bdcbc41c0b6bba86a56d954fd5496","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Threat
Hunting:Investigations","v8.12.2","v8.14.0"],"title":"[Bug][Investigations]
- Fix slow timeline
queries","number":176838,"url":"https://github.com/elastic/kibana/pull/176838","mergeCommit":{"message":"[Bug][Investigations]
- Fix slow timeline queries (elastic#176838)\n\n## Summary\r\n\r\n**Version
Affected: 8.11.x, 8.12.0, 8.12.1**\r\n\r\n### Background\r\n\r\nThe ID
field necessary to track long running timeline search
strategy\r\nqueries was no longer being passed to ES search after work
in 8.11. This\r\nled to what looked like long running timeline queries,
but in reality\r\nwere queries being repeated due to the ID not being
tracked. This pr\r\nre-introduces the ID field necessary for long
running timeline search\r\nstrategies in security
solution\r\n\r\n**Views Affected:**\r\n - Timeline tabs (query,
correlation, pinned)\r\n - Explore events tables (hosts, users,
network)\r\n - Rule preview table\r\n \r\n\r\nPre-fix:\r\n\r\nObserver
the changing ID's for the `timelineSearchStrategy`
`eventsAll`\r\nqueries.\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/5731d310-d3ed-452d-8c34-783b2cfe76e1\r\n\r\n\r\nPost-fix:\r\n\r\nObserver
the same ID for the `timelineSearchStrategy`
`eventsAll`\r\nqueries.\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/a20d4b28-2748-4475-a257-96133bb8efc7\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"68bdd7cb275bdcbc41c0b6bba86a56d954fd5496"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.2","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.14.0","branchLabelMappingKey":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/176838","number":176838,"mergeCommit":{"message":"[Bug][Investigations]
- Fix slow timeline queries (elastic#176838)\n\n## Summary\r\n\r\n**Version
Affected: 8.11.x, 8.12.0, 8.12.1**\r\n\r\n### Background\r\n\r\nThe ID
field necessary to track long running timeline search
strategy\r\nqueries was no longer being passed to ES search after work
in 8.11. This\r\nled to what looked like long running timeline queries,
but in reality\r\nwere queries being repeated due to the ID not being
tracked. This pr\r\nre-introduces the ID field necessary for long
running timeline search\r\nstrategies in security
solution\r\n\r\n**Views Affected:**\r\n - Timeline tabs (query,
correlation, pinned)\r\n - Explore events tables (hosts, users,
network)\r\n - Rule preview table\r\n \r\n\r\nPre-fix:\r\n\r\nObserver
the changing ID's for the `timelineSearchStrategy`
`eventsAll`\r\nqueries.\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/5731d310-d3ed-452d-8c34-783b2cfe76e1\r\n\r\n\r\nPost-fix:\r\n\r\nObserver
the same ID for the `timelineSearchStrategy`
`eventsAll`\r\nqueries.\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/17211684/a20d4b28-2748-4475-a257-96133bb8efc7\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"68bdd7cb275bdcbc41c0b6bba86a56d954fd5496"}}]}]
BACKPORT-->

Co-authored-by: Michael Olorunnisola <[email protected]>
  • Loading branch information
kibanamachine and michaelolo24 authored Feb 14, 2024
1 parent e446672 commit f5bd489
Show file tree
Hide file tree
Showing 8 changed files with 434 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { timelineEqlRequestOptionsSchema } from './eql';
import { mockBaseTimelineRequest } from './mocks/base_timeline_request';

const mockEqlRequestOptions = {
...mockBaseTimelineRequest,
filterQuery: 'sequence\n[any where true]\n[any where true]',
eventCategoryField: 'event.category',
tiebreakerField: '',
fieldRequested: [
'@timestamp',
'message',
'event.category',
'event.action',
'host.name',
'source.ip',
'destination.ip',
'user.name',
'@timestamp',
'kibana.alert.workflow_status',
'kibana.alert.workflow_tags',
'kibana.alert.workflow_assignee_ids',
'kibana.alert.group.id',
'kibana.alert.original_time',
'kibana.alert.building_block_type',
'kibana.alert.rule.from',
'kibana.alert.rule.name',
'kibana.alert.rule.to',
'kibana.alert.rule.uuid',
'kibana.alert.rule.rule_id',
'kibana.alert.rule.type',
'kibana.alert.suppression.docs_count',
'kibana.alert.original_event.kind',
'kibana.alert.original_event.module',
'file.path',
'file.Ext.code_signature.subject_name',
'file.Ext.code_signature.trusted',
'file.hash.sha256',
'host.os.family',
'event.code',
'process.entry_leader.entity_id',
],
language: 'eql',
pagination: {
activePage: 0,
querySize: 25,
},
runtimeMappings: {},
size: 100,
sort: [
{
direction: 'asc',
esTypes: ['date'],
field: '@timestamp',
type: 'date',
},
],
timerange: {
from: '2018-02-12T20:39:22.229Z',
interval: '12h',
to: '2024-02-13T20:39:22.229Z',
},
timestampField: '@timestamp',
};

describe('timelineEqlRequestOptionsSchema', () => {
it('should correctly parse the last eql request object without unknown fields', () => {
expect(timelineEqlRequestOptionsSchema.parse(mockEqlRequestOptions)).toEqual(
mockEqlRequestOptions
);
});

it('should correctly parse the last eql request object and remove unknown fields', () => {
const invalidEqlRequest = {
...mockEqlRequestOptions,
unknownField: 'should-be-removed',
};
expect(timelineEqlRequestOptionsSchema.parse(invalidEqlRequest)).toEqual(mockEqlRequestOptions);
});

it('should correctly error if an incorrect field type is provided for a schema key', () => {
const invalidEqlRequest = {
...mockEqlRequestOptions,
fieldRequested: 123,
};

expect(() => {
timelineEqlRequestOptionsSchema.parse(invalidEqlRequest);
}).toThrowErrorMatchingInlineSnapshot(`
"[
{
\\"code\\": \\"invalid_type\\",
\\"expected\\": \\"array\\",
\\"received\\": \\"number\\",
\\"path\\": [
\\"fieldRequested\\"
],
\\"message\\": \\"Expected array, received number\\"
}
]"
`);
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { timelineEventsAllSchema } from './events_all';
import { mockBaseTimelineRequest } from './mocks/base_timeline_request';

const mockEventsAllRequest = {
...mockBaseTimelineRequest,
factoryQueryType: 'eventsAll',
excludeEcsData: false,
pagination: { activePage: 0, querySize: 25 },
fieldRequested: [
'@timestamp',
'_index',
'message',
'host.name',
'event.module',
'agent.type',
'event.dataset',
'event.action',
'user.name',
'source.ip',
'destination.ip',
],
sort: [
{
field: '@timestamp',
type: 'date',
direction: 'desc',
esTypes: [],
},
],
fields: [],
language: 'kuery',
};

describe('timelineEventsAllSchema', () => {
it('should correctly parse the events request object', () => {
expect(timelineEventsAllSchema.parse(mockEventsAllRequest)).toEqual(mockEventsAllRequest);
});

it('should correctly parse the events request object and remove unknown fields', () => {
const invalidEventsRequest = {
...mockEventsAllRequest,
unknownField: 'shouldBeRemoved',
};
expect(timelineEventsAllSchema.parse(invalidEventsRequest)).toEqual(mockEventsAllRequest);
});

it('should correctly error if an incorrect field type is provided for a schema key', () => {
const invalidEventsRequest = {
...mockEventsAllRequest,
excludeEcsData: 'notABoolean',
};

expect(() => {
timelineEventsAllSchema.parse(invalidEventsRequest);
}).toThrowErrorMatchingInlineSnapshot(`
"[
{
\\"code\\": \\"invalid_type\\",
\\"expected\\": \\"boolean\\",
\\"received\\": \\"string\\",
\\"path\\": [
\\"excludeEcsData\\"
],
\\"message\\": \\"Expected boolean, received string\\"
}
]"
`);
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { timelineEventsDetailsSchema } from './events_details';

const mockEventsDetails = {
entityType: 'events',
indexName: 'test-large-index',
eventId: 'enfXnY0Byt9Ce9tO1aWh',
factoryQueryType: 'eventsDetails',
runtimeMappings: {},
};

describe('timelineEventsDetailsSchema', () => {
it('should correctly parse the event details request schema', () => {
expect(timelineEventsDetailsSchema.parse(mockEventsDetails)).toEqual(mockEventsDetails);
});

it('should correctly parse the event details request schema and remove unknown fields', () => {
const invalidEventsDetailsRequest = {
...mockEventsDetails,
unknownField: 'should-be-removed',
};
expect(timelineEventsDetailsSchema.parse(invalidEventsDetailsRequest)).toEqual(
mockEventsDetails
);
});

it('should correctly error if an incorrect field type is provided for a schema key', () => {
const invalidEventsDetailsRequest = {
...mockEventsDetails,
indexName: 123,
};

expect(() => {
timelineEventsDetailsSchema.parse(invalidEventsDetailsRequest);
}).toThrowErrorMatchingInlineSnapshot(`
"[
{
\\"code\\": \\"invalid_type\\",
\\"expected\\": \\"string\\",
\\"received\\": \\"number\\",
\\"path\\": [
\\"indexName\\"
],
\\"message\\": \\"Expected string, received number\\"
}
]"
`);
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { timelineEventsLastEventTimeRequestSchema } from './events_last_event_time';
import { mockBaseTimelineRequest } from './mocks/base_timeline_request';

const mockEventsLastEventTimeRequest = {
...mockBaseTimelineRequest,
// Remove fields that are omitted in the schema
runtimeMappings: undefined,
filterQuery: undefined,
timerange: undefined,
// Add eventsLastEventTime specific fields
factoryQueryType: 'eventsLastEventTime',
indexKey: 'hosts',
details: {},
};

describe('timelineEventsLastEventTimeRequestSchema', () => {
it('should correctly parse the last event time request object without unknown fields', () => {
expect(timelineEventsLastEventTimeRequestSchema.parse(mockEventsLastEventTimeRequest)).toEqual(
mockEventsLastEventTimeRequest
);
});

it('should correctly parse the last event time request object and remove unknown fields', () => {
const invalidEventsDetailsRequest = {
...mockEventsLastEventTimeRequest,
unknownField: 'should-be-removed',
};
expect(timelineEventsLastEventTimeRequestSchema.parse(invalidEventsDetailsRequest)).toEqual(
mockEventsLastEventTimeRequest
);
});

it('should correctly error if an incorrect field type is provided for a schema key', () => {
const invalidEventsDetailsRequest = {
...mockEventsLastEventTimeRequest,
indexKey: 'unknown-key',
};

expect(() => {
timelineEventsLastEventTimeRequestSchema.parse(invalidEventsDetailsRequest);
}).toThrowErrorMatchingInlineSnapshot(`
"[
{
\\"received\\": \\"unknown-key\\",
\\"code\\": \\"invalid_enum_value\\",
\\"options\\": [
\\"hostDetails\\",
\\"hosts\\",
\\"users\\",
\\"userDetails\\",
\\"ipDetails\\",
\\"network\\"
],
\\"path\\": [
\\"indexKey\\"
],
\\"message\\": \\"Invalid enum value. Expected 'hostDetails' | 'hosts' | 'users' | 'userDetails' | 'ipDetails' | 'network', received 'unknown-key'\\"
}
]"
`);
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { timelineKpiRequestOptionsSchema } from './kpi';
import { mockBaseTimelineRequest } from './mocks/base_timeline_request';

const mockKpiRequest = {
...mockBaseTimelineRequest,
factoryQueryType: 'eventsKpi',
};

describe('timelineKpiRequestOptionsSchema', () => {
it('should correctly parse the events kpi request object', () => {
expect(timelineKpiRequestOptionsSchema.parse(mockKpiRequest)).toEqual(mockKpiRequest);
});

it('should correctly parse the events kpi request object and remove unknown fields', () => {
const invalidKpiRequest = {
...mockKpiRequest,
unknownField: 'shouldBeRemoved',
};
expect(timelineKpiRequestOptionsSchema.parse(invalidKpiRequest)).toEqual(mockKpiRequest);
});

it('should correctly error if an incorrect field type is provided for a schema key', () => {
const invalidKpiRequest = {
...mockKpiRequest,
factoryQueryType: 'someOtherType',
};

expect(() => {
timelineKpiRequestOptionsSchema.parse(invalidKpiRequest);
}).toThrowErrorMatchingInlineSnapshot(`
"[
{
\\"received\\": \\"someOtherType\\",
\\"code\\": \\"invalid_literal\\",
\\"expected\\": \\"eventsKpi\\",
\\"path\\": [
\\"factoryQueryType\\"
],
\\"message\\": \\"Invalid literal value, expected \\\\\\"eventsKpi\\\\\\"\\"
}
]"
`);
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export const mockBaseTimelineRequest = {
id: 'Fnh1dVQ4SDRTUldtRXpUcDEwZXliWHcdZXdlWVBFWkVSWHVIdzY4a19JbFRvUTozMzgzNzk=',
defaultIndex: ['*-large-index'],
filterQuery:
'{"bool":{"must":[],"filter":[{"bool":{"should":[{"exists":{"field":"host.name"}}],"minimum_should_match":1}},{"range":{"@timestamp":{"gte":"2019-02-13T15:39:10.392Z","lt":"2024-02-14T04:59:59.999Z","format":"strict_date_optional_time"}}}],"should":[],"must_not":[]}}',
runtimeMappings: {},
timerange: {
interval: '12h',
from: '2019-02-13T15:39:10.392Z',
to: '2024-02-14T04:59:59.999Z',
},
entityType: 'events',
};
Loading

0 comments on commit f5bd489

Please sign in to comment.