Skip to content

Commit

Permalink
[SIEM][Detection Engine] Adds of risk score, output index, rule copyi…
Browse files Browse the repository at this point in the history 
…ng, and more (#51190)

## Summary

- `risk_score` now required on a POST to the rules 
- `output_index` now required on a POST to the rules
- Enabled a mechanism to deploy using environment variables a way to turn signals on for testing
- Removed `SIGNALS_REINDEX` algorithm now
- Added an optional `meta` object for misc storage of UI information on a POST
- Added `status` field for the signal document for the signals data grid viewer
- Added default signals output index to ui settings of `siem:defaultSignalsIndex`
- Removed revision from signals as we are not doing revisioning
- Updated schema to utilize newer rules with slightly different structure
- Updated the copying of rule meta data into signals to have latest fields
- Added ability for saved searches to save state so if a saved search is deleted you can have a fallback
- Updated `README.md` with new instructions on how to use the system

Screen shot of the advanced setting for the siem signals output index.
<img width="677" alt="Screen Shot 2019-11-19 at 9 08 40 PM" src="https://user-images.githubusercontent.com/1151048/69287461-9b40fb00-0bb3-11ea-9761-9e0c6df69bb9.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~

~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~

~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~

### For maintainers

~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

- [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
  • Loading branch information
@FrankHassanabad
FrankHassanabad authored Nov 21, 2019
1 parent 5f2a326 commit acac80c
Show file tree
Hide file tree
Showing 45 changed files with 1,342 additions and 539 deletions.
1 change: 1 addition & 0 deletions x-pack/legacy/plugins/siem/common/constants.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export const DEFAULT_TIME_RANGE = 'timepicker:timeDefaults';
export const DEFAULT_REFRESH_RATE_INTERVAL = 'timepicker:refreshIntervalDefaults';
export const DEFAULT_SIEM_TIME_RANGE = 'siem:timeDefaults';
export const DEFAULT_SIEM_REFRESH_INTERVAL = 'siem:refreshIntervalDefaults';
export const DEFAULT_SIGNALS_INDEX_KEY = 'siem:defaultSignalsIndex';
export const DEFAULT_SIGNALS_INDEX = '.siem-signals';
export const DEFAULT_ANOMALY_SCORE = 'siem:defaultAnomalyScore';
export const DEFAULT_MAX_TABLE_QUERY_SIZE = 10000;
Expand Down
91 changes: 71 additions & 20 deletions x-pack/legacy/plugins/siem/index.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,78 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { siem } from '.';
import { getRequiredPlugins } from '.';

// This test is a temporary test which is so we do not accidentally check-in
// feature flags turned on from "alerting" and "actions". If those get
// turned on during a check-in it will cause everyone's Kibana to not start.
// Once alerting and actions are part of the plugins by default this test
// should be removed.
describe('siem plugin tests', () => {
// This test is a temporary test which is so we do not accidentally check-in
// feature flags turned on from "alerting" and "actions". If those get
// turned on during a check-in it will cause everyone's Kibana to not start.
// Once alerting and actions are part of the plugins by default this test
// should be removed.
test(`
You have accidentally tried to check-in a feature flag with alerting located
here: x-pack/legacy/plugins/siem/index.ts, please change the plugin require to
NOT have these two inside of the require array."
`, () => {
class MockPlugin {
require: string[];
constructor({ require }: { require: string[] }) {
this.require = require;
}
}
const plugin = siem({ Plugin: MockPlugin });
expect(plugin.require.includes('alerting')).toBe(false);
expect(plugin.require.includes('actions')).toBe(false);
describe('getRequiredPlugins', () => {
test('null settings returns regular kibana and elasticsearch plugins', () => {
expect(getRequiredPlugins(null, null)).toEqual(['kibana', 'elasticsearch']);
});

test('undefined settings returns regular kibana and elasticsearch plugins', () => {
expect(getRequiredPlugins(undefined, undefined)).toEqual(['kibana', 'elasticsearch']);
});

test('alertingFeatureEnabled being false returns regular kibana and elasticsearch plugins', () => {
expect(getRequiredPlugins('false', undefined)).toEqual(['kibana', 'elasticsearch']);
});

test('alertingFeatureEnabled being true returns action and alerts', () => {
expect(getRequiredPlugins('true', undefined)).toEqual([
'kibana',
'elasticsearch',
'alerting',
'actions',
]);
});

test('alertingFeatureEnabled being false but a string for siemIndex returns alerting and actions', () => {
expect(getRequiredPlugins('false', '.siem-signals-frank')).toEqual([
'kibana',
'elasticsearch',
'alerting',
'actions',
]);
});

test('alertingFeatureEnabled being true and a string for siemIndex returns alerting and actions', () => {
expect(getRequiredPlugins('true', '.siem-signals-frank')).toEqual([
'kibana',
'elasticsearch',
'alerting',
'actions',
]);
});

test('alertingFeatureEnabled being true and an empty string for siemIndex returns regular kibana and elasticsearch plugins', () => {
expect(getRequiredPlugins(undefined, '')).toEqual(['kibana', 'elasticsearch']);
});

test('alertingFeatureEnabled being true and a string of spaces for siemIndex returns regular kibana and elasticsearch plugins', () => {
expect(getRequiredPlugins(undefined, ' ')).toEqual(['kibana', 'elasticsearch']);
});

test('alertingFeatureEnabled being null and a string for siemIndex returns alerting and actions', () => {
expect(getRequiredPlugins(null, '.siem-signals-frank')).toEqual([
'kibana',
'elasticsearch',
'alerting',
'actions',
]);
});

test('alertingFeatureEnabled being undefined and a string for siemIndex returns alerting and actions', () => {
expect(getRequiredPlugins(undefined, '.siem-signals-frank')).toEqual([
'kibana',
'elasticsearch',
'alerting',
'actions',
]);
});
});
});
44 changes: 38 additions & 6 deletions x-pack/legacy/plugins/siem/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,41 @@ import {
DEFAULT_INTERVAL_VALUE,
DEFAULT_FROM,
DEFAULT_TO,
DEFAULT_SIGNALS_INDEX,
DEFAULT_SIGNALS_INDEX_KEY,
} from './common/constants';
import { defaultIndexPattern } from './default_index_pattern';

// This is VERY TEMPORARY as we need a way to turn on alerting and actions
// for the server without having to manually edit this file. Once alerting
// and actions has their enabled true by default this can be removed.
// 'alerting', 'actions' are hidden behind feature flags at the moment so if you turn
// these on without the feature flags turned on then Kibana will crash since we are a legacy plugin
// and legacy plugins cannot have optional requirements.
// This returns ['alerting', 'actions', 'kibana', 'elasticsearch'] iff alertingFeatureEnabled is true
// or if the developer signalsIndex is setup. Otherwise this returns ['kibana', 'elasticsearch']
export const getRequiredPlugins = (
alertingFeatureEnabled: string | null | undefined,
signalsIndex: string | null | undefined
) => {
const baseRequire = ['kibana', 'elasticsearch'];
if (
(signalsIndex != null && signalsIndex.trim() !== '') ||
(alertingFeatureEnabled && alertingFeatureEnabled.toLowerCase() === 'true')
) {
return [...baseRequire, 'alerting', 'actions'];
} else {
return baseRequire;
}
};

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export const siem = (kibana: any) => {
return new kibana.Plugin({
id: APP_ID,
configPrefix: 'xpack.siem',
publicDir: resolve(__dirname, 'public'),
require: ['kibana', 'elasticsearch'],
// Uncomment these lines to turn on alerting and action for detection engine and comment the other
// require statement out. These are hidden behind feature flags at the moment so if you turn
// these on without the feature flags turned on then Kibana will crash since we are a legacy plugin
// and legacy plugins cannot have optional requirements.
// require: ['kibana', 'elasticsearch', 'alerting', 'actions'],
require: getRequiredPlugins(process.env.ALERTING_FEATURE_ENABLED, process.env.SIGNALS_INDEX),
uiExports: {
app: {
description: i18n.translate('xpack.siem.securityDescription', {
Expand Down Expand Up @@ -106,6 +126,18 @@ export const siem = (kibana: any) => {
category: ['siem'],
requiresPageReload: true,
},
[DEFAULT_SIGNALS_INDEX_KEY]: {
name: i18n.translate('xpack.siem.uiSettings.defaultSignalsIndexLabel', {
defaultMessage: 'Elasticsearch signals index',
}),
value: DEFAULT_SIGNALS_INDEX,
description: i18n.translate('xpack.siem.uiSettings.defaultSignalsIndexDescription', {
defaultMessage:
'<p>Elasticsearch signals index from which outputted signals will appear by default</p>',
}),
category: ['siem'],
requiresPageReload: true,
},
[DEFAULT_ANOMALY_SCORE]: {
name: i18n.translate('xpack.siem.uiSettings.defaultAnomalyScoreLabel', {
defaultMessage: 'Anomaly threshold',
Expand Down
4 changes: 4 additions & 0 deletions x-pack/legacy/plugins/siem/scripts/convert_saved_search_to_signals.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ const FROM = 'now-6m';
const TO = 'now';
const IMMUTABLE = true;
const INDEX = ['auditbeat-*', 'filebeat-*', 'packetbeat-*', 'winlogbeat-*'];
const OUTPUT_INDEX = process.env.SIGNALS_INDEX || '.siem-signals';
const RISK_SCORE = 50;

const walk = dir => {
const list = fs.readdirSync(dir);
Expand Down Expand Up @@ -119,6 +121,7 @@ async function main() {
if (query != null && query.trim() !== '') {
const outputMessage = {
rule_id: fileToWrite,
risk_score: RISK_SCORE,
description: description || title,
immutable: IMMUTABLE,
index: INDEX,
Expand All @@ -131,6 +134,7 @@ async function main() {
query,
language,
filters: filter,
output_index: OUTPUT_INDEX,
};

fs.writeFileSync(
Expand Down
135 changes: 94 additions & 41 deletions x-pack/legacy/plugins/siem/server/lib/detection_engine/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,45 @@
Temporary README.md for developers working on the backend detection engine
Temporary README.md for users and developers working on the backend detection engine
for how to get started.

# Setup for Users

If you're just a user and want to enable the REST interfaces and UI screens do the following.
NOTE: this is very temporary and once alerting and actions is enabled by default you will no
longer have to do these steps

Set the environment variable ALERTING_FEATURE_ENABLED to be true in your .profile or your windows
global environment variable.

```sh
export ALERTING_FEATURE_ENABLED=true
```

In your `kibana.yml` file enable alerting and actions like so:

```sh
# Feature flag to turn on alerting
xpack.alerting.enabled: true

# Feature flag to turn on actions which goes with alerting
xpack.actions.enabled: true
```

Start Kibana and you will see these messages indicating signals is activated like so:

```sh
server log [11:39:05.561] [info][siem] Detected feature flags for actions and alerting and enabling signals API endpoints
```

If you see crashes like this:

```ts
FATAL Error: Unmet requirement "alerting" for plugin "siem"
```

It is because Kibana is not picking up your changes from `kibana.yml` and not seeing that alerting and actions is enabled.

# For Developers

See these two other pages for references:
https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/alerting/README.md
https://github.com/elastic/kibana/tree/master/x-pack/legacy/plugins/actions
Expand All @@ -19,7 +58,7 @@ brew install jq

Open up your .zshrc/.bashrc and add these lines with the variables filled in:

```
```sh
export ELASTICSEARCH_USERNAME=${user}
export ELASTICSEARCH_PASSWORD=${password}
export ELASTICSEARCH_URL=https://${ip}:9200
Expand All @@ -37,50 +76,37 @@ source your .zhsrc/.bashrc or open a new terminal to ensure you get the new valu
Optional env var when set to true will utilize `reindex` api for reindexing
instead of the scroll and bulk index combination.

```
```sh
export USE_REINDEX_API=true
```

Add these lines to your `kibana.dev.yml` to turn on the feature toggles of alerting and actions:

```
```sh
# Feature flag to turn on alerting
xpack.alerting.enabled: true

# Feature flag to turn on actions which goes with alerting
xpack.actions.enabled: true
# White list everything for ease of development (do not do in production)
xpack.actions.whitelistedHosts: ['*']
```

Open `x-pack/legacy/plugins/siem/index.ts` and find these lines and add the require statement
while commenting out the other require statement:

```
// Uncomment these lines to turn on alerting and action for detection engine and comment the other
// require statement out. These are hidden behind feature flags at the moment so if you turn
// these on without the feature flags turned on then Kibana will crash since we are a legacy plugin
// and legacy plugins cannot have optional requirements.
// require: ['kibana', 'elasticsearch', 'alerting', 'actions'],
```
Restart Kibana and ensure that you are using `--no-base-path` as changing the base path is a feature but will
get in the way of the CURL scripts written as is. You should see alerting and actions starting up like so afterwards

Restart Kibana and you should see alerting and actions starting up

```
```sh
server log [22:05:22.277] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
server log [22:05:22.270] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
```

You should also see the SIEM detect the feature flags and start the API endpoints for signals

```
```sh
server log [11:39:05.561] [info][siem] Detected feature flags for actions and alerting and enabling signals API endpoints
```

Open a terminal and go into the scripts folder `cd kibana/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts` and run:

```
```sh
./hard_reset.sh
./post_signal.sh
```
Expand All @@ -91,37 +117,64 @@ which will:
- Delete any existing alerts you have
- Delete any existing alert tasks you have
- Delete any existing signal mapping you might have had.
- Add the latest signal index and its mappings
- Posts a sample signal which checks for root or admin every 5 minutes
- Add the latest signal index and its mappings using your settings from `SIGNALS_INDEX` environment variable.
- Posts the sample signal from `signals/root_or_admin_1.json` by replacing its `output_index` with your `SIGNALS_INDEX` environment variable
- The sample signal checks for root or admin every 5 minutes and reports that as a signal if it is a positive hit

Now you can run

```sh
./get_alert_instances.sh
./find_signals.sh
```

You should see the new alert instance created like so:
You should see the new signals created like so:

```ts
```sh
{
"id": "908a6af1-ac63-4d52-a856-fc635a00db0f",
"alertTypeId": "siem.signals",
"interval": "5m",
"actions": [ ],
"alertTypeParams": {},
"enabled": true,
"throttle": null,
"createdBy": "elastic",
"updatedBy": "elastic",
"apiKeyOwner": "elastic",
"scheduledTaskId": "4f401ca0-e402-11e9-94ed-051d758a6c79"
"page": 1,
"perPage": 20,
"total": 1,
"data": [
{
"created_by": "elastic",
"description": "Detecting root and admin users",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "a556065c-0656-4ba1-ad64-a77ca9d2013b",
"immutable": false,
"index": [
"auditbeat-*",
"filebeat-*",
"packetbeat-*",
"winlogbeat-*"
],
"interval": "5m",
"rule_id": "rule-1",
"language": "kuery",
"output_index": ".siem-signals-frank-hassanabad",
"max_signals": 100,
"risk_score": 1,
"name": "Detect Root/Admin Users",
"query": "user.name: root or user.name: admin",
"references": [
"http://www.example.com",
"https://ww.example.com"
],
"severity": "high",
"updated_by": "elastic",
"tags": [],
"to": "now",
"type": "query"
}
]
}
```

Every 5 minutes you should see this message in your terminal now:
Every 5 minutes if you get positive hits you will see messages on info like so:

```
server log [22:17:33.945] [info][alerting] SIEM Alert Fired
```sh
server log [09:54:59.013] [info][plugins][siem] Total signals found from signal rule "id: a556065c-0656-4ba1-ad64-a77ca9d2013b", "ruleId: rule-1": 10000
```

See the scripts folder and the tools for more command line fun.
Expand Down
Loading

0 comments on commit acac80c

Please sign in to comment.