[SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more #51190

FrankHassanabad · 2019-11-20T17:00:07Z

Summary

risk_score now required on a POST to the rules
output_index now required on a POST to the rules
Enabled a mechanism to deploy using environment variables a way to turn signals on for testing
Removed SIGNALS_REINDEX algorithm now
Added an optional meta object for misc storage of UI information on a POST
Added status field for the signal document for the signals data grid viewer
Added default signals output index to ui settings of siem:defaultSignalsIndex
Removed revision from signals as we are not doing revisioning
Updated schema to utilize newer rules with slightly different structure
Updated the copying of rule meta data into signals to have latest fields
Added ability for saved searches to save state so if a saved search is deleted you can have a fallback
Updated README.md with new instructions on how to use the system

Screen shot of the advanced setting for the siem signals output index.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, including a check against IE11~~

~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~

~~- [ ] Documentation was added for features that require explanation or tutorials~~

Unit or functional tests were updated or added to match the most common scenarios

~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

~~- [ ] This was checked for breaking API changes and was labeled appropriately~~

This includes a feature addition or change that requires a release note and was labeled appropriately

… alerting and actions are behind a feature flag

…ts and types and refactoring

elasticmachine · 2019-11-20T23:25:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ca5b198

elasticmachine · 2019-11-20T23:36:20Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2019-11-21T01:02:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 174f589

x-pack/legacy/plugins/siem/index.ts

XavierM · 2019-11-21T01:49:23Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/__mocks__/es_results.ts

  ruleId: 'rule-1',
  description: 'Detecting root and admin users',
  falsePositives: [],
  immutable: false,
  index: ['auditbeat-*', 'filebeat-*', 'packetbeat-*', 'winlogbeat-*'],
-  interval: '5m',
-  name: 'Detect Root/Admin Users',


I thought that the name and interval were required.

@XavierM it's omitted https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/types.ts#L127

Those two are omitted because they are part of the main base object of alerting and are no longer alert parameters. This fixes some typing issues and bugs we had around our TypeScript types to keep things accurate.

XavierM · 2019-11-21T01:56:26Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/__mocks__/request_responses.ts

@@ -121,12 +124,15 @@ export const getResult = (): SignalAlertType => ({
    index: ['auditbeat-*', 'filebeat-*', 'packetbeat-*', 'winlogbeat-*'],
    falsePositives: [],
    from: 'now-6m',
-    filter: undefined,
+    filter: null,


what is the difference between filter and filters

The difference is that filter does not require any kql at all. A user can just push a regular filter they happen to have not based on any query. It's a pure filter only.

However, per our conversations, before shipping I will more than likely remove it and only keep the query + filter to keep the API from having use cases no one is asking for.

x-pack/legacy/plugins/siem/server/lib/detection_engine/README.md

dhurley14 · 2019-11-21T16:58:09Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/README.md

@@ -37,50 +76,36 @@ source your .zhsrc/.bashrc or open a new terminal to ensure you get the new valu
 Optional env var when set to true will utilize `reindex` api for reindexing
 instead of the scroll and bulk index combination.

-```
+```sh
 export USE_REINDEX_API=true
 ```


I don't think this code exists anymore so probably don't need the env var in the doc.

dhurley14 · 2019-11-21T17:24:00Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/signals_alert_type.ts

@@ -30,10 +28,13 @@ export const signalsAlertType = ({ logger }: { logger: Logger }): SignalAlertTyp
        immutable: schema.boolean({ defaultValue: false }),
        index: schema.arrayOf(schema.string()),
        language: schema.nullable(schema.string()),
+        outputIndex: schema.string(),


the less env vars the better 👍 This is nice to have.

dhurley14 · 2019-11-21T17:25:04Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/signals_alert_type.ts

-            alertId
-          );
+        const bulkIndexResult = await searchAfterAndBulkIndex({
+          someResult: noReIndexResult,


I should rename this parameter to be more descriptive.. I'll add that in my open PR.

dhurley14 · 2019-11-21T17:39:34Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/utils.ts

-        sortId,
-        params,
-        service,
-        logger


funny my linter didn't catch the missing dangling comma..

dhurley14

Ran and tested locally 👍 looks great! Thanks for these updates.

elasticmachine · 2019-11-21T19:22:58Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 97abff6

…ng, and more (elastic#51190) ## Summary - `risk_score` now required on a POST to the rules - `output_index` now required on a POST to the rules - Enabled a mechanism to deploy using environment variables a way to turn signals on for testing - Removed `SIGNALS_REINDEX` algorithm now - Added an optional `meta` object for misc storage of UI information on a POST - Added `status` field for the signal document for the signals data grid viewer - Added default signals output index to ui settings of `siem:defaultSignalsIndex` - Removed revision from signals as we are not doing revisioning - Updated schema to utilize newer rules with slightly different structure - Updated the copying of rule meta data into signals to have latest fields - Added ability for saved searches to save state so if a saved search is deleted you can have a fallback - Updated `README.md` with new instructions on how to use the system Screen shot of the advanced setting for the siem signals output index. <img width="677" alt="Screen Shot 2019-11-19 at 9 08 40 PM" src="https://user-images.githubusercontent.com/1151048/69287461-9b40fb00-0bb3-11ea-9761-9e0c6df69bb9.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ - [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)

…ng, and more (#51190) (#51367) ## Summary - `risk_score` now required on a POST to the rules - `output_index` now required on a POST to the rules - Enabled a mechanism to deploy using environment variables a way to turn signals on for testing - Removed `SIGNALS_REINDEX` algorithm now - Added an optional `meta` object for misc storage of UI information on a POST - Added `status` field for the signal document for the signals data grid viewer - Added default signals output index to ui settings of `siem:defaultSignalsIndex` - Removed revision from signals as we are not doing revisioning - Updated schema to utilize newer rules with slightly different structure - Updated the copying of rule meta data into signals to have latest fields - Added ability for saved searches to save state so if a saved search is deleted you can have a fallback - Updated `README.md` with new instructions on how to use the system Screen shot of the advanced setting for the siem signals output index. <img width="677" alt="Screen Shot 2019-11-19 at 9 08 40 PM" src="https://user-images.githubusercontent.com/1151048/69287461-9b40fb00-0bb3-11ea-9761-9e0c6df69bb9.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ - [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)

FrankHassanabad added 5 commits November 19, 2019 16:02

Added risk_score to the schema

d5c38f7

Merge branch 'master' into risk-score

edac450

Added output index as a required field and defaulted it for developers

b6f1c9e

Forgot to checkin the in siem/index

30d6647

Added ability to turn on siem signals for server side use even though…

cab25f9

… alerting and actions are behind a feature flag

This comment has been minimized.

Sign in to view

FrankHassanabad added 2 commits November 20, 2019 14:26

Fixed the signal rule copying over to be more complete along with tes…

43fcd16

…ts and types and refactoring

Added meta object for the UI to utilize

ca5b198

This comment has been minimized.

Sign in to view

Updated saved id to work with fall back parameters

174f589

FrankHassanabad changed the title ~~Risk score~~ [SIEM][Detection Engine] Addition of risk score, output index, and rule copying Nov 20, 2019

FrankHassanabad self-assigned this Nov 20, 2019

FrankHassanabad changed the title ~~[SIEM][Detection Engine] Addition of risk score, output index, and rule copying~~ [SIEM][Detection Engine] Addition of risk score, output index, rule copying, and more Nov 20, 2019

FrankHassanabad changed the title ~~[SIEM][Detection Engine] Addition of risk score, output index, rule copying, and more~~ [SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more Nov 20, 2019

FrankHassanabad marked this pull request as ready for review November 20, 2019 23:36

FrankHassanabad added the Team:SIEM label Nov 20, 2019

FrankHassanabad added release_note:enhancement v8.0.0 v7.6.0 labels Nov 20, 2019

XavierM reviewed Nov 21, 2019

View reviewed changes

x-pack/legacy/plugins/siem/index.ts Outdated Show resolved Hide resolved

XavierM reviewed Nov 21, 2019

View reviewed changes

Merge branch 'master' into risk-score

acd752a

dhurley14 reviewed Nov 21, 2019

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/README.md Show resolved Hide resolved

dhurley14 reviewed Nov 21, 2019

View reviewed changes

dhurley14 approved these changes Nov 21, 2019

View reviewed changes

FrankHassanabad added 2 commits November 21, 2019 11:01

Updated PR from feedback and merge from master

f109cfe

Updated per feedback

97abff6

FrankHassanabad merged commit acac80c into elastic:master Nov 21, 2019

FrankHassanabad deleted the risk-score branch November 21, 2019 20:00

FrankHassanabad mentioned this pull request Nov 21, 2019

[7.x] [SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more (#51190) #51367

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more #51190

[SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more #51190

FrankHassanabad commented Nov 20, 2019 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

elasticmachine commented Nov 20, 2019

elasticmachine commented Nov 20, 2019

elasticmachine commented Nov 21, 2019

XavierM Nov 21, 2019 •

edited

Loading

dhurley14 Nov 21, 2019

FrankHassanabad Nov 21, 2019

XavierM Nov 21, 2019

FrankHassanabad Nov 21, 2019

dhurley14 Nov 21, 2019 •

edited

Loading

dhurley14 Nov 21, 2019

dhurley14 Nov 21, 2019

dhurley14 Nov 21, 2019

dhurley14 left a comment

elasticmachine commented Nov 21, 2019

[SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more #51190

[SIEM][Detection Engine] Adds of risk score, output index, rule copying, and more #51190

Conversation

FrankHassanabad commented Nov 20, 2019 • edited Loading

Summary

Checklist

For maintainers

This comment has been minimized.

This comment has been minimized.

elasticmachine commented Nov 20, 2019

💚 Build Succeeded

elasticmachine commented Nov 20, 2019

elasticmachine commented Nov 21, 2019

💚 Build Succeeded

XavierM Nov 21, 2019 • edited Loading

Choose a reason for hiding this comment

dhurley14 Nov 21, 2019

Choose a reason for hiding this comment

FrankHassanabad Nov 21, 2019

Choose a reason for hiding this comment

XavierM Nov 21, 2019

Choose a reason for hiding this comment

FrankHassanabad Nov 21, 2019

Choose a reason for hiding this comment

dhurley14 Nov 21, 2019 • edited Loading

Choose a reason for hiding this comment

dhurley14 Nov 21, 2019

Choose a reason for hiding this comment

dhurley14 Nov 21, 2019

Choose a reason for hiding this comment

dhurley14 Nov 21, 2019

Choose a reason for hiding this comment

dhurley14 left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 21, 2019

💚 Build Succeeded

FrankHassanabad commented Nov 20, 2019 •

edited

Loading

XavierM Nov 21, 2019 •

edited

Loading

dhurley14 Nov 21, 2019 •

edited

Loading