From b5831b8884b19fe08168feb6e99ea1722a4b823c Mon Sep 17 00:00:00 2001 From: Ben Skelker Date: Sun, 8 Dec 2019 15:19:12 +0200 Subject: [PATCH 1/3] clarifies siem indices and kibana index patterns requirements --- docs/en/siem/installation.asciidoc | 22 +++++++++++-------- docs/en/siem/siem-ui.asciidoc | 34 ++++++++++++++++++++---------- 2 files changed, 36 insertions(+), 20 deletions(-) diff --git a/docs/en/siem/installation.asciidoc b/docs/en/siem/installation.asciidoc index afa0560f7..04fbc0879 100644 --- a/docs/en/siem/installation.asciidoc +++ b/docs/en/siem/installation.asciidoc @@ -18,11 +18,20 @@ https://www.elastic.co/cloud/elasticsearch-service/signup[Try the {es} Service for free]. ============== -* *{beats}* shippers (version 7.x or later) installed for each system you want to -monitor +* *{beats}* (version 7.x or later) or other data shippers installed for each +system you want to monitor. -You might need to modify UI settings in {kib} to change default behaviors, -such as the index pattern used to query the data. For more information, see {kib}. +[IMPORTANT] +============== +If you use a third-party collector to ship data to the {siem-app}, you must +map its fields to the {ecs-ref}[Elastic Common Schema (ECS)]. Additionally, you +must add its index to the {siem-soln} {es} indices (*{kib}* -> +*Management* -> *Advanced Settings* -> *`siem:defaultIndex`*). + +It is also important to note that {siem-soln} uses the +{ecs-ref}/ecs-host.html[`host.name`] ECS field as the primary key for +identifying hosts. +============== [float] [[install-beats]] @@ -39,11 +48,6 @@ Windows event logs * https://www.elastic.co/products/beats/packetbeat[{packetbeat}] for analyzing network activity -NOTE: The {ecs-ref}[Elastic Common Schema (ECS)] enables {siem-soln} to work -with custom and third-party data sources in addition to -those supported by Beats. It is important to note that {siem-soln} uses the -{ecs-ref}/ecs-host.html[`host.name`] ECS field as the primary key for identifying hosts. - You can install {beats} using a {kib}-based guide or directly from the command line. [float] diff --git a/docs/en/siem/siem-ui.asciidoc b/docs/en/siem/siem-ui.asciidoc index e6a668aba..86ddfc988 100644 --- a/docs/en/siem/siem-ui.asciidoc +++ b/docs/en/siem/siem-ui.asciidoc @@ -16,9 +16,10 @@ collapsible, resizable, moveable, and so forth. The *{kibana-ref}/kuery-query.html[{kib} Query Language (KQL)]* bar is available throughout the {siem-app} for searching and filtering. -NOTE: The default index patterns for {siem-soln} events are `endgame-*`, -`auditbeat-*`, `winlogbeat-*`, `filebeat-*`, and `packetbeat-*`. You can change -the default index patterns in {kib} -> Management -> Advanced Settings -> `siem:defaultIndex`. +NOTE: The default index glob patterns defined for {siem-soln} events are +`endgame-*`, `auditbeat-*`, `winlogbeat-*`, `filebeat-*`, and `packetbeat-*`. +You can change the default glob patterns in {kib} -> Management -> Advanced +Settings -> `siem:defaultIndex`. [float] @@ -205,24 +206,35 @@ that aren't present? Let us know. We welcome your input. Depending on your {kib} setup, to display and interact with data on the map you might need to: +* <> * <> * <> NOTE: To see source and destination connections lines on the map, you must configure `source.geo` and `destination.geo` ECS fields for your indices. +[float] +[[kibana-index-pattern]] +==== Create {kib} index patterns + +To display data on the map, you must define {kib} index patterns +(Management -> Index Patterns) that match all the {siem-soln} {es} indices you +want to visualize. For more information, see +{kibana-ref}/tutorial-define-index.html[Define your index patterns]. + +NOTE: The {siem-soln} {es} indices are defined in the `siem:defaultIndex` field +(*{kib}* -> *Management* -> *Advanced Settings* -> *`siem:defaultIndex`*). + [float] [[geoip-data]] ==== Add geoIP data -If you are not using Beats to ship your data, add the relevant index patterns to -Kibana (Management -> Index patterns) and the SIEM app (Management -> Advanced -settings -> SIEM default index). -When the ECS {ecs-ref}/ecs-geo.html[source.geo.location -and destination.geo.location] fields are mapped, network data is displayed on +When the ECS {ecs-ref}/ecs-geo.html[source.geo.location and +destination.geo.location] fields are mapped, network data is displayed on the map. -If you use Beats, configure a geoIP processor to add data to the relevant fields: +If you use Beats, configure a geoIP processor to add data to the relevant +fields: [[geo-pipeleine]] . Define an ingest node pipeline that uses one or more `geoIP` processors to add @@ -283,8 +295,8 @@ that contains the IP address to use for the geographical lookup, and `"ignore_missing": true` configures the pipeline to continue processing when it encounters an event that doesn't have the specified field. -. In your Beats configuration files, add the pipeline to the `output.elasticsearch` -tag: +. In your Beats configuration files, add the pipeline to the +`output.elasticsearch`tag: + [source,yml] ---------------------------------- From 839531f4159e4d19f949978e093d06e764d866a8 Mon Sep 17 00:00:00 2001 From: Ben Skelker Date: Tue, 10 Dec 2019 11:14:30 +0200 Subject: [PATCH 2/3] corrections after feedback --- docs/en/siem/siem-ui.asciidoc | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/en/siem/siem-ui.asciidoc b/docs/en/siem/siem-ui.asciidoc index 86ddfc988..2d5e21706 100644 --- a/docs/en/siem/siem-ui.asciidoc +++ b/docs/en/siem/siem-ui.asciidoc @@ -217,14 +217,27 @@ configure `source.geo` and `destination.geo` ECS fields for your indices. [[kibana-index-pattern]] ==== Create {kib} index patterns -To display data on the map, you must define {kib} index patterns -(Management -> Index Patterns) that match all the {siem-soln} {es} indices you -want to visualize. For more information, see -{kibana-ref}/tutorial-define-index.html[Define your index patterns]. +To display map data, you must define +{kib} {kibana-ref}/tutorial-define-index.html[index patterns] (*Management* -> +*Index Patterns*) with exactly the same names or glob patterns used to define +the {siem-soln} {es} indices. NOTE: The {siem-soln} {es} indices are defined in the `siem:defaultIndex` field (*{kib}* -> *Management* -> *Advanced Settings* -> *`siem:defaultIndex`*). +For example, if you define a {siem-soln} {es} `servers-europe-*` glob pattern, +to display map data for the matching indices you must also define a {kib} index +pattern named `servers-europe-*`. If you use a different {kib} index pattern, +such as `servers-*`, map data for the indices is *not* displayed. + +// ===================== +// Keeping this commented out because it will be true in the future... +// To display data on the map, you must define {kib} index patterns +// (Management -> Index Patterns) that match all the {siem-soln} {es} indices you +// want to visualize. For more information, see +// {kibana-ref}/tutorial-define-index.html[Define your index patterns]. +// ===================== + [float] [[geoip-data]] ==== Add geoIP data From f6024fcc92d0e1f56bd6bf5c8c2836e4957d7926 Mon Sep 17 00:00:00 2001 From: Ben Skelker Date: Wed, 11 Dec 2019 09:35:01 +0200 Subject: [PATCH 3/3] corrections after review --- docs/en/siem/installation.asciidoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/en/siem/installation.asciidoc b/docs/en/siem/installation.asciidoc index 04fbc0879..d891725af 100644 --- a/docs/en/siem/installation.asciidoc +++ b/docs/en/siem/installation.asciidoc @@ -28,9 +28,8 @@ map its fields to the {ecs-ref}[Elastic Common Schema (ECS)]. Additionally, you must add its index to the {siem-soln} {es} indices (*{kib}* -> *Management* -> *Advanced Settings* -> *`siem:defaultIndex`*). -It is also important to note that {siem-soln} uses the -{ecs-ref}/ecs-host.html[`host.name`] ECS field as the primary key for -identifying hosts. +{siem-soln} uses the {ecs-ref}/ecs-host.html[`host.name`] ECS field as the +primary key for identifying hosts. ============== [float]