Skip to content

Commit

Permalink
[SIEM][Detection Engine] Fixes bug with timeline templates not working (
Browse files Browse the repository at this point in the history 
#60538)

### Summary

Fixes a bug with the timeline templates not working when specifying filters.

* Creates a type safe mechanism for getting StringArrays or regular strings
* AddsType Script function returns to functions in the helpers file
* Adds unit tests for the effected areas of code and corner cases

Before this fix you would get these toaster errors if you tried to use a template name such as `host.name` in the timeline filters:

<img width="677" alt="Screen Shot 2020-03-18 at 12 58 01 AM" src="https://user-images.githubusercontent.com/1151048/76934058-0bd2fc80-68b4-11ea-8dad-7c257bb81a1d.png">

After this fix it will work for you.

Testing:

1) Create a timeline template that has a host.name as both a query and a filter such as this. You can give the value of the host.name any value such as placeholder.

<img width="1125" alt="Screen Shot 2020-03-18 at 12 56 04 AM" src="https://user-images.githubusercontent.com/1151048/76934108-20af9000-68b4-11ea-8a11-4ba9c935506f.png">

2) Create a signal that uses it and produces a lot of signals off of something such as all host names
<img width="1054" alt="Screen Shot 2020-03-18 at 12 50 47 AM" src="https://user-images.githubusercontent.com/1151048/76934198-4f2d6b00-68b4-11ea-8ae3-6de76154cbb7.png">

3) Ensure you select your **Timeline template** you saved by using the drop down
<img width="1071" alt="Screen Shot 2020-03-18 at 12 51 21 AM" src="https://user-images.githubusercontent.com/1151048/76934281-73894780-68b4-11ea-9a2a-a0a9176f28ce.png">

4) Once your signals have run, go to the signals page and send one of the signals for your newly crated rule which has a host name to the timeline from "View in timeline"
<img width="568" alt="Screen Shot 2020-03-18 at 12 52 10 AM" src="https://user-images.githubusercontent.com/1151048/76934365-a4697c80-68b4-11ea-91a5-e0dea7e3e18f.png">

You should notice that your timeline has both the query and the filter set correctly such as this
<img width="1114" alt="Screen Shot 2020-03-18 at 12 56 23 AM" src="https://user-images.githubusercontent.com/1151048/76934432-c105b480-68b4-11ea-9a82-3e8a2da19376.png">


### Other notes

All the different fields you can choose from for templates are:
```
  'host.name',
  'host.hostname',
  'host.domain',
  'host.id',
  'host.ip',
  'client.ip',
  'destination.ip',
  'server.ip',
  'source.ip',
  'network.community_id',
  'user.name',
  'process.name',
```

And it should not work with anything outside of those. You should be able to mix and match them into different filters and queries to have a multiples of them.

### Checklist

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
  • Loading branch information
@FrankHassanabad
FrankHassanabad authored Mar 19, 2020
1 parent 9cb2477 commit 4ee34e6
Show file tree
Hide file tree
Showing 2 changed files with 328 additions and 18 deletions.
273 changes: 273 additions & 0 deletions x-pack/legacy/plugins/siem/public/pages/detection_engine/components/signals/helpers.test.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,273 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

import {
getStringArray,
replaceTemplateFieldFromQuery,
replaceTemplateFieldFromMatchFilters,
reformatDataProviderWithNewValue,
} from './helpers';
import { mockEcsData } from '../../../../mock/mock_ecs';
import { Filter } from '../../../../../../../../../src/plugins/data/public';
import { DataProvider } from '../../../../components/timeline/data_providers/data_provider';
import { mockDataProviders } from '../../../../components/timeline/data_providers/mock/mock_data_providers';
import { cloneDeep } from 'lodash/fp';

describe('helpers', () => {
let mockEcsDataClone = cloneDeep(mockEcsData);
beforeEach(() => {
mockEcsDataClone = cloneDeep(mockEcsData);
});
describe('getStringOrStringArray', () => {
test('it should correctly return a string array', () => {
const value = getStringArray('x', {
x: 'The nickname of the developer we all :heart:',
});
expect(value).toEqual(['The nickname of the developer we all :heart:']);
});

test('it should correctly return a string array with a single element', () => {
const value = getStringArray('x', {
x: ['The nickname of the developer we all :heart:'],
});
expect(value).toEqual(['The nickname of the developer we all :heart:']);
});

test('it should correctly return a string array with two elements of strings', () => {
const value = getStringArray('x', {
x: ['The nickname of the developer we all :heart:', 'We are all made of stars'],
});
expect(value).toEqual([
'The nickname of the developer we all :heart:',
'We are all made of stars',
]);
});

test('it should correctly return a string array with deep elements', () => {
const value = getStringArray('x.y.z', {
x: { y: { z: 'zed' } },
});
expect(value).toEqual(['zed']);
});

test('it should correctly return a string array with a non-existent value', () => {
const value = getStringArray('non.existent', {
x: { y: { z: 'zed' } },
});
expect(value).toEqual([]);
});

test('it should trace an error if the value is not a string', () => {
const mockConsole: Console = ({ trace: jest.fn() } as unknown) as Console;
const value = getStringArray('a', { a: 5 }, mockConsole);
expect(value).toEqual([]);
expect(
mockConsole.trace
).toHaveBeenCalledWith(
'Data type that is not a string or string array detected:',
5,
'when trying to access field:',
'a',
'from data object of:',
{ a: 5 }
);
});

test('it should trace an error if the value is an array of mixed values', () => {
const mockConsole: Console = ({ trace: jest.fn() } as unknown) as Console;
const value = getStringArray('a', { a: ['hi', 5] }, mockConsole);
expect(value).toEqual([]);
expect(
mockConsole.trace
).toHaveBeenCalledWith(
'Data type that is not a string or string array detected:',
['hi', 5],
'when trying to access field:',
'a',
'from data object of:',
{ a: ['hi', 5] }
);
});
});

describe('replaceTemplateFieldFromQuery', () => {
test('given an empty query string this returns an empty query string', () => {
const replacement = replaceTemplateFieldFromQuery('', mockEcsDataClone[0]);
expect(replacement).toEqual('');
});

test('given a query string with spaces this returns an empty query string', () => {
const replacement = replaceTemplateFieldFromQuery(' ', mockEcsDataClone[0]);
expect(replacement).toEqual('');
});

test('it should replace a query with a template value such as apache from a mock template', () => {
const replacement = replaceTemplateFieldFromQuery(
'host.name: placeholdertext',
mockEcsDataClone[0]
);
expect(replacement).toEqual('host.name: apache');
});

test('it should replace a template field with an ECS value that is not an array', () => {
mockEcsDataClone[0].host!.name = ('apache' as unknown) as string[]; // very unsafe cast for this test case
const replacement = replaceTemplateFieldFromQuery('host.name: *', mockEcsDataClone[0]);
expect(replacement).toEqual('host.name: *');
});

test('it should NOT replace a query with a template value that is not part of the template fields array', () => {
const replacement = replaceTemplateFieldFromQuery(
'user.id: placeholdertext',
mockEcsDataClone[0]
);
expect(replacement).toEqual('user.id: placeholdertext');
});
});

describe('replaceTemplateFieldFromMatchFilters', () => {
test('given an empty query filter this will return an empty filter', () => {
const replacement = replaceTemplateFieldFromMatchFilters([], mockEcsDataClone[0]);
expect(replacement).toEqual([]);
});

test('given a query filter this will return that filter with the placeholder replaced', () => {
const filters: Filter[] = [
{
meta: {
type: 'phrase',
key: 'host.name',
alias: 'alias',
disabled: false,
negate: false,
params: { query: 'Braden' },
},
query: { match_phrase: { 'host.name': 'Braden' } },
},
];
const replacement = replaceTemplateFieldFromMatchFilters(filters, mockEcsDataClone[0]);
const expected: Filter[] = [
{
meta: {
type: 'phrase',
key: 'host.name',
alias: 'alias',
disabled: false,
negate: false,
params: { query: 'apache' },
},
query: { match_phrase: { 'host.name': 'apache' } },
},
];
expect(replacement).toEqual(expected);
});

test('given a query filter with a value not in the templateFields, this will NOT replace the placeholder value', () => {
const filters: Filter[] = [
{
meta: {
type: 'phrase',
key: 'user.id',
alias: 'alias',
disabled: false,
negate: false,
params: { query: 'Evan' },
},
query: { match_phrase: { 'user.id': 'Evan' } },
},
];
const replacement = replaceTemplateFieldFromMatchFilters(filters, mockEcsDataClone[0]);
const expected: Filter[] = [
{
meta: {
type: 'phrase',
key: 'user.id',
alias: 'alias',
disabled: false,
negate: false,
params: { query: 'Evan' },
},
query: { match_phrase: { 'user.id': 'Evan' } },
},
];
expect(replacement).toEqual(expected);
});
});

describe('reformatDataProviderWithNewValue', () => {
test('it should replace a query with a template value such as apache from a mock data provider', () => {
const mockDataProvider: DataProvider = mockDataProviders[0];
mockDataProvider.queryMatch.field = 'host.name';
mockDataProvider.id = 'Braden';
mockDataProvider.name = 'Braden';
mockDataProvider.queryMatch.value = 'Braden';
const replacement = reformatDataProviderWithNewValue(mockDataProvider, mockEcsDataClone[0]);
expect(replacement).toEqual({
id: 'apache',
name: 'apache',
enabled: true,
excluded: false,
kqlQuery: '',
queryMatch: {
field: 'host.name',
value: 'apache',
operator: ':',
displayField: undefined,
displayValue: undefined,
},
and: [],
});
});

test('it should replace a query with a template value such as apache from a mock data provider using a string in the data provider', () => {
mockEcsDataClone[0].host!.name = ('apache' as unknown) as string[]; // very unsafe cast for this test case
const mockDataProvider: DataProvider = mockDataProviders[0];
mockDataProvider.queryMatch.field = 'host.name';
mockDataProvider.id = 'Braden';
mockDataProvider.name = 'Braden';
mockDataProvider.queryMatch.value = 'Braden';
const replacement = reformatDataProviderWithNewValue(mockDataProvider, mockEcsDataClone[0]);
expect(replacement).toEqual({
id: 'apache',
name: 'apache',
enabled: true,
excluded: false,
kqlQuery: '',
queryMatch: {
field: 'host.name',
value: 'apache',
operator: ':',
displayField: undefined,
displayValue: undefined,
},
and: [],
});
});

test('it should NOT replace a query with a template value that is not part of a template such as user.id', () => {
const mockDataProvider: DataProvider = mockDataProviders[0];
mockDataProvider.queryMatch.field = 'user.id';
mockDataProvider.id = 'my-id';
mockDataProvider.name = 'Rebecca';
mockDataProvider.queryMatch.value = 'Rebecca';
const replacement = reformatDataProviderWithNewValue(mockDataProvider, mockEcsDataClone[0]);
expect(replacement).toEqual({
id: 'my-id',
name: 'Rebecca',
enabled: true,
excluded: false,
kqlQuery: '',
queryMatch: {
field: 'user.id',
value: 'Rebecca',
operator: ':',
displayField: undefined,
displayValue: undefined,
},
and: [],
});
});
});
});
73 changes: 55 additions & 18 deletions x-pack/legacy/plugins/siem/public/pages/detection_engine/components/signals/helpers.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ interface FindValueToChangeInQuery {
valueToChange: string;
}

/**
* Fields that will be replaced with the template strings from a a saved timeline template.
* This is used for the signals detection engine feature when you save a timeline template
* and are the fields you can replace when creating a template.
*/
const templateFields = [
'host.name',
'host.hostname',
Expand All @@ -32,6 +37,36 @@ const templateFields = [
'process.name',
];

/**
* This will return an unknown as a string array if it exists from an unknown data type and a string
* that represents the path within the data object the same as lodash's "get". If the value is non-existent
* we will return an empty array. If it is a non string value then this will log a trace to the console
* that it encountered an error and return an empty array.
* @param field string of the field to access
* @param data The unknown data that is typically a ECS value to get the value
* @param localConsole The local console which can be sent in to make this pure (for tests) or use the default console
*/
export const getStringArray = (field: string, data: unknown, localConsole = console): string[] => {
const value: unknown | undefined = get(field, data);
if (value == null) {
return [];
} else if (typeof value === 'string') {
return [value];
} else if (Array.isArray(value) && value.every(element => typeof element === 'string')) {
return value;
} else {
localConsole.trace(
'Data type that is not a string or string array detected:',
value,
'when trying to access field:',
field,
'from data object of:',
data
);
return [];
}
};

export const findValueToChangeInQuery = (
keuryNode: KueryNode,
valueToChange: FindValueToChangeInQuery[] = []
Expand Down Expand Up @@ -66,31 +101,33 @@ export const findValueToChangeInQuery = (
);
};

export const replaceTemplateFieldFromQuery = (query: string, ecsData: Ecs) => {
export const replaceTemplateFieldFromQuery = (query: string, ecsData: Ecs): string => {
if (query.trim() !== '') {
const valueToChange = findValueToChangeInQuery(esKuery.fromKueryExpression(query));
return valueToChange.reduce((newQuery, vtc) => {
const newValue = get(vtc.field, ecsData);
if (newValue != null) {
return newQuery.replace(vtc.valueToChange, newValue);
const newValue = getStringArray(vtc.field, ecsData);
if (newValue.length) {
return newQuery.replace(vtc.valueToChange, newValue[0]);
} else {
return newQuery;
}
return newQuery;
}, query);
} else {
return '';
}
return '';
};

export const replaceTemplateFieldFromMatchFilters = (filters: Filter[], ecsData: Ecs) =>
export const replaceTemplateFieldFromMatchFilters = (filters: Filter[], ecsData: Ecs): Filter[] =>
filters.map(filter => {
if (
filter.meta.type === 'phrase' &&
filter.meta.key != null &&
templateFields.includes(filter.meta.key)
) {
const newValue = get(filter.meta.key, ecsData);
if (newValue != null) {
filter.meta.params = { query: newValue };
filter.query = { match_phrase: { [filter.meta.key]: newValue } };
const newValue = getStringArray(filter.meta.key, ecsData);
if (newValue.length) {
filter.meta.params = { query: newValue[0] };
filter.query = { match_phrase: { [filter.meta.key]: newValue[0] } };
}
}
return filter;
Expand All @@ -101,11 +138,11 @@ export const reformatDataProviderWithNewValue = <T extends DataProvider | DataPr
ecsData: Ecs
): T => {
if (templateFields.includes(dataProvider.queryMatch.field)) {
const newValue = get(dataProvider.queryMatch.field, ecsData);
if (newValue != null) {
dataProvider.id = dataProvider.id.replace(dataProvider.name, newValue);
dataProvider.name = newValue;
dataProvider.queryMatch.value = newValue;
const newValue = getStringArray(dataProvider.queryMatch.field, ecsData);
if (newValue.length) {
dataProvider.id = dataProvider.id.replace(dataProvider.name, newValue[0]);
dataProvider.name = newValue[0];
dataProvider.queryMatch.value = newValue[0];
dataProvider.queryMatch.displayField = undefined;
dataProvider.queryMatch.displayValue = undefined;
}
Expand All @@ -116,8 +153,8 @@ export const reformatDataProviderWithNewValue = <T extends DataProvider | DataPr
export const replaceTemplateFieldFromDataProviders = (
dataProviders: DataProvider[],
ecsData: Ecs
) =>
dataProviders.map((dataProvider: DataProvider) => {
): DataProvider[] =>
dataProviders.map(dataProvider => {
const newDataProvider = reformatDataProviderWithNewValue(dataProvider, ecsData);
if (newDataProvider.and != null && !isEmpty(newDataProvider.and)) {
newDataProvider.and = newDataProvider.and.map(andDataProvider =>
Expand Down

0 comments on commit 4ee34e6

Please sign in to comment.