Skip to content

Commit

Permalink
[Security Solutions][Detection Engine] Adds ability to ignore fields …
Browse files Browse the repository at this point in the history
…during alert indexing and a workaround for an EQL bug (elastic#110927)

## Summary

Adds a workaround for EQL bug: elastic/elasticsearch#77152
Adds the safety feature mentioned here: elastic#110802

Adds the ability to ignore particular [fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) when the field is merged with [_source](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering). Also fixes an EQL bug where EQL is introducing the meta field of `_ignored` within the fields and causing documents to not be indexable when we merge with the fields from EQL.

Alerting document creation uses the fields API to get [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html),  [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type), etc... that are only available within the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) and then merges the field values not found within the `_source` document with the `_source` document and then finally indexes this merged document as an alert document.

This fix/ability is a "safety feature" in that if a problematic [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type) is discovered or another bug along the stack we can set a `kibana.yml` key/value pair to ignore the problematic field.

This _WILL NOT_ remove problematic fields from the `_source` document. This will only ignore problematic constant keyword, runtime fields, aliases, or anything else found in the fields API that is causing merge issues.

This PR:
  * Adds a `alertIgnoreFields` `kibana.yml` array key with a default of an empty array if not specified.
  * Plumbs the `alertIgnoreFields` through the stack and into the fields/_source merge strategies of `missingFields` and `allFields`
  * Adds a temporary `isEqlBug77152` where it hard codes an ignore of `_ignored` until the EQL problem is fixed and then we will remove the workaround
  * Adds unit tests
  * Adds e2e tests which covers the described use cases above.

The `alertIgnoreFields` key/value within `kibana.yml` if set should be an array of strings of each field you want to ignore. This can also contain regular expressions as long as they are of the form, `"/regex/"` in the array.

Example if you want to ignore fields that are problematic called "host.name" and then one in which you want to ignore all fields that start with "user." using a regular expression:

```yml
xpack.securitySolution.alertIgnoreFields: ['host.name', '/user\..*/']
```

Although there are e2e tests which exercise the use cases...

If you want to manual test the EQL bug fix you would add these documents in dev tools:

```json
# Delete and add a mapping with a small ignore_above.
DELETE eql-issue-ignore-fields-delme
PUT eql-issue-ignore-fields-delme
{
  "mappings" : {
    "dynamic": "strict",
    "properties" : {
      "@timestamp": {
        "type": "date"
      },
      "some_keyword" : {
        "ignore_above": 5,
        "type" : "keyword"
      },
      "other_keyword" : {
        "ignore_above": 10,
        "type" : "keyword"
      }
    }
  }
}

# Add a single document with one field that will be truncated and a second that will not.
PUT eql-issue-ignore-fields-delme/_doc/1
{
  "@timestamp": "2021-09-02T04:13:05.626Z",
  "some_keyword": "longer than normal",
  "other_keyword": "normal"
}
```

Then create an alert which queries everything from it:
<img width="1155" alt="Screen Shot 2021-09-01 at 10 15 06 PM" src="https://user-images.githubusercontent.com/1151048/131781042-faa424cf-65a5-4ebb-b801-3f188940c81d.png">

and ensure signals are created:
<img width="2214" alt="Screen Shot 2021-09-01 at 10 30 18 PM" src="https://user-images.githubusercontent.com/1151048/131782069-b9ab959c-f22d-44d5-baf0-561fe349c037.png">

To test the manual exclusions of any other problematic fields, create any index which has runtime fields or `constant keywords` but does not have anything within the `_source` document using dev tools. For example you can use `constant keyword` like so

```json
PUT constant-keywords-deleme
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "testing_ignored": {
        "properties": {
          "constant": {
            "type": "constant_keyword",
            "value": "constant_value"
          }
        }
      },
      "testing_regex": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "normal_constant": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "small_field": {
        "type": "keyword",
        "ignore_above": 10
      }
    }
  }
}

PUT constant-keywords-deleme/_doc/1
{
  "@timestamp": "2021-09-02T04:20:01.760Z"
}
```

Set in your `kibana.yml` the key/value of:

```yml
xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/']
```

Setup a rule to run:
<img width="1083" alt="Screen Shot 2021-09-01 at 10 23 23 PM" src="https://user-images.githubusercontent.com/1151048/131781696-fea0d421-836f-465c-9be6-5289fbb622a4.png">

Once it runs you should notice that the constant values for testing are not on the signals table since it only typically exists in the fields API:
<img width="1166" alt="Screen Shot 2021-09-01 at 10 26 16 PM" src="https://user-images.githubusercontent.com/1151048/131781782-1684fb1d-bed9-4cf0-be9a-0abe1f0f34d1.png">

But the normal one still exists:
<img width="1136" alt="Screen Shot 2021-09-01 at 10 26 31 PM" src="https://user-images.githubusercontent.com/1151048/131781827-5450c693-de9e-4285-b082-9f7a2cbd5d07.png">

If you change the `xpack.securitySolution.alertIgnoreFields` by removing it and re-generate the signals you will see these values added back.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)

# Conflicts:
#	src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/create_security_rule_type_factory.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/utils/build_bulk_body.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/wrap_hits_factory.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/create_indicator_match_alert_type.test.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/create_indicator_match_alert_type.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.test.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.test.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/query/create_query_alert_type.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/types.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_bulk_body.test.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_bulk_body.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/signals/signal_rule_alert_type.test.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/signals/signal_rule_alert_type.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/signals/wrap_hits_factory.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/signals/wrap_sequences_factory.ts
#	x-pack/plugins/security_solution/server/plugin.ts
#	x-pack/test/detection_engine_api_integration/security_and_spaces/tests/index.ts
  • Loading branch information
FrankHassanabad committed Sep 3, 2021
1 parent 9ac2d2f commit d36f1f3
Show file tree
Hide file tree
Showing 30 changed files with 1,015 additions and 234 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ kibana_vars=(
xpack.security.sessionTimeout
xpack.securitySolution.alertMergeStrategy
xpack.securitySolution.alertResultListDefaultDateRange
xpack.securitySolution.alertIgnoreFields
xpack.securitySolution.endpointResultListDefaultFirstPageIndex
xpack.securitySolution.endpointResultListDefaultPageSize
xpack.securitySolution.maxRuleImportExportSize
Expand Down
80 changes: 80 additions & 0 deletions x-pack/plugins/security_solution/server/config.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { configSchema } from './config';

describe('config', () => {
describe('alertIgnoreFields', () => {
test('should default to an empty array', () => {
expect(configSchema.validate({}).alertIgnoreFields).toEqual([]);
});

test('should accept an array of strings', () => {
expect(
configSchema.validate({ alertIgnoreFields: ['foo.bar', 'mars.bar'] }).alertIgnoreFields
).toEqual(['foo.bar', 'mars.bar']);
});

test('should throw if a non string is being sent in', () => {
expect(
() =>
configSchema.validate({
alertIgnoreFields: 5,
}).alertIgnoreFields
).toThrow('[alertIgnoreFields]: expected value of type [array] but got [number]');
});

test('should throw if we send in an invalid regular expression as a string', () => {
expect(
() =>
configSchema.validate({
alertIgnoreFields: ['/(/'],
}).alertIgnoreFields
).toThrow(
'[alertIgnoreFields]: "Invalid regular expression: /(/: Unterminated group" at array position 0'
);
});

test('should throw with two errors if we send two invalid regular expressions', () => {
expect(
() =>
configSchema.validate({
alertIgnoreFields: ['/(/', '/(invalid/'],
}).alertIgnoreFields
).toThrow(
'[alertIgnoreFields]: "Invalid regular expression: /(/: Unterminated group" at array position 0. "Invalid regular expression: /(invalid/: Unterminated group" at array position 1'
);
});

test('should throw with two errors with a valid string mixed in if we send two invalid regular expressions', () => {
expect(
() =>
configSchema.validate({
alertIgnoreFields: ['/(/', 'valid.string', '/(invalid/'],
}).alertIgnoreFields
).toThrow(
'[alertIgnoreFields]: "Invalid regular expression: /(/: Unterminated group" at array position 0. "Invalid regular expression: /(invalid/: Unterminated group" at array position 2'
);
});

test('should accept a valid regular expression within the string', () => {
expect(
configSchema.validate({
alertIgnoreFields: ['/(.*)/'],
}).alertIgnoreFields
).toEqual(['/(.*)/']);
});

test('should accept two valid regular expressions', () => {
expect(
configSchema.validate({
alertIgnoreFields: ['/(.*)/', '/(.valid*)/'],
}).alertIgnoreFields
).toEqual(['/(.*)/', '/(.valid*)/']);
});
});
});
49 changes: 49 additions & 0 deletions x-pack/plugins/security_solution/server/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,61 @@ export const configSchema = schema.object({
maxRuleImportPayloadBytes: schema.number({ defaultValue: 10485760 }),
maxTimelineImportExportSize: schema.number({ defaultValue: 10000 }),
maxTimelineImportPayloadBytes: schema.number({ defaultValue: 10485760 }),

/**
* This is used within the merge strategies:
* server/lib/detection_engine/signals/source_fields_merging
*
* For determining which strategy for merging "fields" and "_source" together to get
* runtime fields, constant keywords, etc...
*
* "missingFields" (default) This will only merge fields that are missing from the _source and exist in the fields.
* "noFields" This will turn off all merging of runtime fields, constant keywords from fields.
* "allFields" This will merge and overwrite anything found within "fields" into "_source" before indexing the data.
*/
alertMergeStrategy: schema.oneOf(
[schema.literal('allFields'), schema.literal('missingFields'), schema.literal('noFields')],
{
defaultValue: 'missingFields',
}
),

/**
* This is used within the merge strategies:
* server/lib/detection_engine/signals/source_fields_merging
*
* For determining if we need to ignore particular "fields" and not merge them with "_source" such as
* runtime fields, constant keywords, etc...
*
* This feature and functionality is mostly as "safety feature" meaning that we have had bugs in the past
* where something down the stack unexpectedly ends up in the fields API which causes documents to not
* be indexable. Rather than changing alertMergeStrategy to be "noFields", you can use this array to add
* any problematic values.
*
* You can use plain dotted notation strings such as "host.name" or a regular expression such as "/host\..+/"
*/
alertIgnoreFields: schema.arrayOf(schema.string(), {
defaultValue: [],
validate(ignoreFields) {
const errors = ignoreFields.flatMap((ignoreField, index) => {
if (ignoreField.startsWith('/') && ignoreField.endsWith('/')) {
try {
new RegExp(ignoreField.slice(1, -1));
return [];
} catch (error) {
return [`"${error.message}" at array position ${index}`];
}
} else {
return [];
}
});
if (errors.length !== 0) {
return errors.join('. ');
} else {
return undefined;
}
},
}),
[SIGNALS_INDEX_KEY]: schema.string({ defaultValue: DEFAULT_SIGNALS_INDEX }),

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ export const createMockConfig = (): ConfigType => ({
packagerTaskInterval: '60s',
validateArtifactDownloads: true,
alertMergeStrategy: 'missingFields',
alertIgnoreFields: [],
prebuiltRulesFromFileSystem: true,
prebuiltRulesFromSavedObjects: false,
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ describe('buildBulkBody', () => {
const fakeSignalSourceHit: SignalHitOptionalTimestamp = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
// Timestamp will potentially always be different so remove it for the test
delete fakeSignalSourceHit['@timestamp'];
Expand Down Expand Up @@ -109,7 +110,8 @@ describe('buildBulkBody', () => {
const fakeSignalSourceHit: SignalHitOptionalTimestamp = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
// Timestamp will potentially always be different so remove it for the test
delete fakeSignalSourceHit['@timestamp'];
Expand Down Expand Up @@ -191,7 +193,8 @@ describe('buildBulkBody', () => {
const fakeSignalSourceHit: SignalHitOptionalTimestamp = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
// Timestamp will potentially always be different so remove it for the test
delete fakeSignalSourceHit['@timestamp'];
Expand Down Expand Up @@ -259,7 +262,8 @@ describe('buildBulkBody', () => {
const fakeSignalSourceHit: SignalHitOptionalTimestamp = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
// Timestamp will potentially always be different so remove it for the test
delete fakeSignalSourceHit['@timestamp'];
Expand Down Expand Up @@ -324,7 +328,8 @@ describe('buildBulkBody', () => {
const fakeSignalSourceHit: SignalHitOptionalTimestamp = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
// Timestamp will potentially always be different so remove it for the test
delete fakeSignalSourceHit['@timestamp'];
Expand Down Expand Up @@ -388,7 +393,8 @@ describe('buildBulkBody', () => {
const { '@timestamp': timestamp, ...fakeSignalSourceHit } = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
const expected: Omit<SignalHit, '@timestamp'> & { someKey: string } = {
someKey: 'someValue',
Expand Down Expand Up @@ -448,7 +454,8 @@ describe('buildBulkBody', () => {
const { '@timestamp': timestamp, ...fakeSignalSourceHit } = buildBulkBody(
ruleSO,
doc,
'missingFields'
'missingFields',
[]
);
const expected: Omit<SignalHit, '@timestamp'> & { someKey: string } = {
someKey: 'someValue',
Expand Down Expand Up @@ -677,7 +684,8 @@ describe('buildSignalFromEvent', () => {
ancestor,
ruleSO,
true,
'missingFields'
'missingFields',
[]
);

// Timestamp will potentially always be different so remove it for the test
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@ import type { ConfigType } from '../../../config';
export const buildBulkBody = (
ruleSO: SavedObject<AlertAttributes>,
doc: SignalSourceHit,
mergeStrategy: ConfigType['alertMergeStrategy']
mergeStrategy: ConfigType['alertMergeStrategy'],
ignoreFields: ConfigType['alertIgnoreFields']
): SignalHit => {
const mergedDoc = getMergeStrategy(mergeStrategy)({ doc });
const mergedDoc = getMergeStrategy(mergeStrategy)({ doc, ignoreFields });
const rule = buildRuleWithOverrides(ruleSO, mergedDoc._source ?? {});
const signal: Signal = {
...buildSignal([mergedDoc], rule),
Expand Down Expand Up @@ -68,11 +69,12 @@ export const buildSignalGroupFromSequence = (
sequence: EqlSequence<SignalSource>,
ruleSO: SavedObject<AlertAttributes>,
outputIndex: string,
mergeStrategy: ConfigType['alertMergeStrategy']
mergeStrategy: ConfigType['alertMergeStrategy'],
ignoreFields: ConfigType['alertIgnoreFields']
): WrappedSignalHit[] => {
const wrappedBuildingBlocks = wrapBuildingBlocks(
sequence.events.map((event) => {
const signal = buildSignalFromEvent(event, ruleSO, false, mergeStrategy);
const signal = buildSignalFromEvent(event, ruleSO, false, mergeStrategy, ignoreFields);
signal.signal.rule.building_block_type = 'default';
return signal;
}),
Expand Down Expand Up @@ -134,9 +136,10 @@ export const buildSignalFromEvent = (
event: BaseSignalHit,
ruleSO: SavedObject<AlertAttributes>,
applyOverrides: boolean,
mergeStrategy: ConfigType['alertMergeStrategy']
mergeStrategy: ConfigType['alertMergeStrategy'],
ignoreFields: ConfigType['alertIgnoreFields']
): SignalHit => {
const mergedEvent = getMergeStrategy(mergeStrategy)({ doc: event });
const mergedEvent = getMergeStrategy(mergeStrategy)({ doc: event, ignoreFields });
const rule = applyOverrides
? buildRuleWithOverrides(ruleSO, mergedEvent._source ?? {})
: buildRuleWithoutOverrides(ruleSO);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ describe('searchAfterAndBulkCreate', () => {
ruleSO,
signalsIndex: DEFAULT_SIGNALS_INDEX,
mergeStrategy: 'missingFields',
ignoreFields: [],
});
});

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ describe('signal_rule_alert_type', () => {
ml: mlMock,
lists: listMock.createSetup(),
mergeStrategy: 'missingFields',
ignoreFields: [],
});
});

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,15 @@ export const signalRulesAlertType = ({
ml,
lists,
mergeStrategy,
ignoreFields,
}: {
logger: Logger;
eventsTelemetry: TelemetryEventsSender | undefined;
version: string;
ml: SetupPlugins['ml'];
lists: SetupPlugins['lists'] | undefined;
mergeStrategy: ConfigType['alertMergeStrategy'];
ignoreFields: ConfigType['alertIgnoreFields'];
}): SignalRuleAlertTypeDefinition => {
return {
id: SIGNALS_ID,
Expand Down Expand Up @@ -237,12 +239,14 @@ export const signalRulesAlertType = ({
ruleSO: savedObject,
signalsIndex: params.outputIndex,
mergeStrategy,
ignoreFields,
});

const wrapSequences = wrapSequencesFactory({
ruleSO: savedObject,
signalsIndex: params.outputIndex,
mergeStrategy,
ignoreFields,
});

if (isMlRule(type)) {
Expand Down
Loading

0 comments on commit d36f1f3

Please sign in to comment.