Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM][Docs]Clarifies SIEM index and Kibana index pattern requirements #739

Merged
merged 3 commits into from
Dec 11, 2019
Merged

[SIEM][Docs]Clarifies SIEM index and Kibana index pattern requirements #739

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7c2d03a
clarifies siem indices and kibana index patterns requirements
Dec 8, 2019
cb55302
corrections after feedback
Dec 10, 2019
4e7be00
corrections after review
Dec 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions docs/en/siem/installation.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,19 @@ https://www.elastic.co/cloud/elasticsearch-service/signup[Try the {es}
Service for free].
==============

* *{beats}* shippers (version 7.x or later) installed for each system you want to
monitor
* *{beats}* (version 7.x or later) or other data shippers installed for each
system you want to monitor.

You might need to modify UI settings in {kib} to change default behaviors,
such as the index pattern used to query the data. For more information, see {kib}.
[IMPORTANT]
==============
If you use a third-party collector to ship data to the {siem-app}, you must
map its fields to the {ecs-ref}[Elastic Common Schema (ECS)]. Additionally, you
must add its index to the {siem-soln} {es} indices (*{kib}* ->
*Management* -> *Advanced Settings* -> *`siem:defaultIndex`*).

{siem-soln} uses the {ecs-ref}/ecs-host.html[`host.name`] ECS field as the
primary key for identifying hosts.
==============

[float]
[[install-beats]]
Expand All @@ -39,11 +47,6 @@ Windows event logs
* https://www.elastic.co/products/beats/packetbeat[{packetbeat}] for analyzing
network activity

NOTE: The {ecs-ref}[Elastic Common Schema (ECS)] enables {siem-soln} to work
with custom and third-party data sources in addition to
those supported by Beats. It is important to note that {siem-soln} uses the
{ecs-ref}/ecs-host.html[`host.name`] ECS field as the primary key for identifying hosts.

You can install {beats} using a {kib}-based guide or directly from the command line.

[float]
Expand Down
47 changes: 36 additions & 11 deletions docs/en/siem/siem-ui.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ collapsible, resizable, moveable, and so forth.
The *{kibana-ref}/kuery-query.html[{kib} Query Language (KQL)]* bar is available
throughout the {siem-app} for searching and filtering.

NOTE: The default index patterns for {siem-soln} events are `endgame-*`,
`auditbeat-*`, `winlogbeat-*`, `filebeat-*`, and `packetbeat-*`. You can change
the default index patterns in {kib} -> Management -> Advanced Settings -> `siem:defaultIndex`.
NOTE: The default index glob patterns defined for {siem-soln} events are
`endgame-*`, `auditbeat-*`, `winlogbeat-*`, `filebeat-*`, and `packetbeat-*`.
You can change the default glob patterns in {kib} -> Management -> Advanced
Settings -> `siem:defaultIndex`.
Comment on lines +19 to +22
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a heads up that this will be changing to include apm-*-transcation* as part of elastic/kibana#52297 (comment). As noted in that comment, there is going to be specific logic for matching apm-*-transcation* to the apm-* Kibana Index Pattern.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @spong

We can update the documentation anytime - so we'll add this when it's implemented.



[float]
Expand Down Expand Up @@ -204,24 +205,48 @@ that aren't present? Let us know. We welcome your input.
Depending on your {kib} setup, to display and interact with data on the map you
might need to:

* <<kibana-index-pattern>>
* <<geoip-data, Add geographical IP data to events>>
* <<private-network>>

NOTE: To see source and destination connections lines on the map, you must
configure `source.geo` and `destination.geo` ECS fields for your indices.

[float]
[[kibana-index-pattern]]
==== Create {kib} index patterns

To display map data, you must define
{kib} {kibana-ref}/tutorial-define-index.html[index patterns] (*Management* ->
*Index Patterns*) with exactly the same names or glob patterns used to define
the {siem-soln} {es} indices.

NOTE: The {siem-soln} {es} indices are defined in the `siem:defaultIndex` field
(*{kib}* -> *Management* -> *Advanced Settings* -> *`siem:defaultIndex`*).

For example, if you define a {siem-soln} {es} `servers-europe-*` glob pattern,
to display map data for the matching indices you must also define a {kib} index
pattern named `servers-europe-*`. If you use a different {kib} index pattern,
such as `servers-*`, map data for the indices is *not* displayed.

// =====================
// Keeping this commented out because it will be true in the future...
// To display data on the map, you must define {kib} index patterns
// (Management -> Index Patterns) that match all the {siem-soln} {es} indices you
// want to visualize. For more information, see
// {kibana-ref}/tutorial-define-index.html[Define your index patterns].
// =====================

[float]
[[geoip-data]]
==== Add geoIP data

If you are not using Beats to ship your data, add the relevant index patterns to
Kibana (Management -> Index patterns) and the SIEM app (Management -> Advanced
settings -> SIEM default index).
When the ECS {ecs-ref}/ecs-geo.html[source.geo.location
and destination.geo.location] fields are mapped, network data is displayed on
When the ECS {ecs-ref}/ecs-geo.html[source.geo.location and
destination.geo.location] fields are mapped, network data is displayed on
the map.

If you use Beats, configure a geoIP processor to add data to the relevant fields:
If you use Beats, configure a geoIP processor to add data to the relevant
fields:

[[geo-pipeleine]]
. Define an ingest node pipeline that uses one or more `geoIP` processors to add
Expand Down Expand Up @@ -282,8 +307,8 @@ that contains the IP address to use for the geographical lookup, and
`"ignore_missing": true` configures the pipeline to continue processing when
it encounters an event that doesn't have the specified field.

. In your Beats configuration files, add the pipeline to the `output.elasticsearch`
tag:
. In your Beats configuration files, add the pipeline to the
`output.elasticsearch`tag:
+
[source,yml]
----------------------------------
Expand Down