Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add getting started doc #888

Merged
merged 16 commits into from
Jul 30, 2020
Binary file added docs/images/ecs-getting-started-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions docs/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ ECS has been scoped to accommodate a wide variety of events, spanning:
Logstash, Elasticsearch ingest node, all of the above, or none of the above.
- *Consumers*: whether consumed by API, Kibana queries, dashboards, apps, or other means.

[float]
=== New to ECS?

If you're new to ECS and looking for an introduction to its benefits and examples of the
core concepts, <<ecs-getting-started>> is a great place to start.

[float]
=== My events don't map with ECS
Expand Down
284 changes: 284 additions & 0 deletions docs/using-getting-started.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,284 @@
[[ecs-getting-started]]
=== Getting Started
webmat marked this conversation as resolved.
Show resolved Hide resolved

ECS enables and encourages users to normalize event data in order
to better analyze, visualize, and correlate their events. Collected events
can be normalized at ingest time, consistently searched across indices,
and visualized predictably.

Note that when adopting an Elastic solution, such as https://www.elastic.co/observability[Observability]
or https://www.elastic.co/security[Security], all events will map to ECS out of the box. Elastic
provides an extensive set of https://www.elastic.co/integrations[integrations] to simplify
ingesting your data sources.

If you rely on custom data pipelines and/or building content around specific needs,
ECS can still help to alleviate the challenges of searching, analyzing, and visualizing
across your data. Let's see how using a common schema can simplify the search experience,
and then take a look at how an event's contents can be mapped to ECS field sets.

[float]
==== Simplified Search

With ECS defining a normalized schema across all of your data sources, querying
against those sources is simplified. Consider searching for a particular source IP
address prior to adopting ECS. All the various data sources and their field mappings
would need to be considered in your query:

[source,sh]
-----------
src:10.42.42.42 OR client_ip:10.42.42.42 OR apache.access.remote_ip:10.42.42.42 OR
context.user.ip:10.42.42.42 OR src_ip:10.42.42.42
-----------

With all sources mapped to ECS, the query becomes much simpler:

[source,sh]
-----------
source.ip:10.42.42.42
-----------

Not only does this simplify writing queries, but saved queries shared with other
users become much more obvious. To gain familiarity with ECS fields, you can also
take a look at the <<ecs-field-reference>> section.

[float]
==== Unified Visualizations

With normalized data from different data sources, building insightful visualizations
across sources is simple. From a single, centralized dashboard, events from web servers,
IDS/IPS devices, and firewalls can be aggregated and visualized, and enhanced with drill-downs,
and pivoting for delving into deeper investigations. Centralized monitoring of diverse data
sources is straightforward with normalized ECS data.

[role="screenshot"]
image:images/ecs-getting-started-dashboard.png[Simplify visualization using ECS]


[float]
==== Translating Data Sources

To align events to ECS, some sort of parsing will usually be necessary
to transform the contents of the original event into the relevant ECS fields. Depending on
how you've designed your Elastic Stack data ingestion pipelines, the amount of work to parse
your events will vary.

For example, an Apache web server log event:

[source,sh]
-----------
10.42.42.42 - - [15/Jul/2020:20:48:32 +0000] "GET /content HTTP/1.1" 200 2571 "-"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/83.0.4103.106 Safari/537.36"
-----------

In order to map this event to ECS, the contents of the event is associated with the
appropriate ECS fields.

[options="header"]
|=====
| Field Name | Value

// ===============================================================

| @timestamp
| `2020-07-15T20:20:48.000Z`

// ==============================================================

| event.original
| 10.42.42.42 - - [15/Jul/2020:20:48:32 +0000] "GET /content HTTP/1.1" 200 2571 "-"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/83.0.4103.106 Safari/537.36

// ==============================================================

| http.request.method
| GET

// ==============================================================

| http.response.body.bytes
| 2571

// ==============================================================

| http.response.status_code
| 200

// ==============================================================

| http.version
| 1.1

// ==============================================================

| message
| GET /content HTTP/1.1" 200 2571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36

// ==============================================================

| source.address
| 10.42.42.42

// ==============================================================

| source.ip
| 10.42.42.42

// ==============================================================

| url.original
| `/content`

// ==============================================================

| user_agent.original
| `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36`

|=====

Beyond extracting values that are present in the original event, we also populate
other fields to provide additional context about the event itself.

* `ecs.version`: States which version of ECS the ingest pipeline was developed against.
* `event.dataset` and `event.module`: Answers "where is this event from" and are expected to have a
hardcoded value per pipeline, per source.
* `event.kind`, `event.category`, `event.type`, and `event.outcome`: The https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html[categorization fields]
should also be hardcoded using knowledge of each type of event the source emits. The contents of
these fields are limited to the specifically allowed values detailed in the ECS documentation.

[options="header"]
|=====
| Field Name | Value

// ===============================================================

| ecs.version
| 1.5.0

// ==============================================================

| event.module
| apache

// ===============================================================

| event.dataset
| apache.access

// ==============================================================

| event.kind
| event

// ===============================================================

| event.category
| [ "web", "access" ]

// ==============================================================

| event.outcome
| success

|=====

Lastly, existing field values can be interpreted or enriched using a processor, with the results
populating additional fields in the final event.

* The `user_agent` processor extracts details from the original user agent string, `user_agent.original`.
* IP fields like `source.ip` can provide enrichment using the `geopip` processor to add information about the
location and autonomous system number (ASN) associated with an IP address.
* The `registered domain` processor reads a field containing a hostname and writes the registered domain to
another field
* Event collectors, such as https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html[Beats], can enrich
each event with metadata from the machine's hosting provider (cloud) and/or from the host machine (host).

ebeahan marked this conversation as resolved.
Show resolved Hide resolved
Here are some examples of additional fields processed by metadata or parser processors.

[options="header"]
|=====
| Field Name | Value | Processor

// ==============================================================

| host.architecture
| x86_64
| `add_host_metadata`

// ==============================================================

| host.hostname
| mbp.example.com
| `add_host_metadata`

// ==============================================================

| host.ip
| [ "192.168.1.100" ]
| `add_host_metadata`

// ==============================================================

| host.os.family
| darwin
| `add_host_metadata`

// ==============================================================

| host.os.kernel
| 19.4.0
| `add_host_metadata`

// ==============================================================

| host.os.name
| Mac OS X
| `add_host_metadata`

// ==============================================================

| host.os.version
| 10.15.4
| `add_host_metadata`

// ==============================================================

| user_agent.name
| Chrome
| `user_agent`

// ==============================================================

| user_agent.os.full
| Mac OS X 10.15.4
| `user_agent`

// ==============================================================

| user_agent.os.name
| Mac OS X
| `user_agent`

// ==============================================================

| user_agent.os.version
| 10.15.4
| `user_agent`

// ==============================================================

| user_agent.version
| 83.0.4103.106
| `user_agent`

|=====

[float]
==== Field Mapping Reference Guides

We've covered at a high level how to map your events to ECS. Now if you'd like your events to render well in the Elastic
solutions, check out the reference guides below to learn more about each:

* https://www.elastic.co/guide/en/logs/guide/current/logs-fields-reference.html[Logs UI fields reference]
* https://www.elastic.co/guide/en/security/master/siem-field-reference.html[Elastic Security fields reference]
6 changes: 5 additions & 1 deletion docs/using.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,17 @@
ECS fields follow a series of guidelines, to ensure a consistent and predictable
feel, across various use cases.

If you're new to ECS and would like an introduction on implementing and using
the schema, check out the <<ecs-getting-started>> guide.

Whether you're trying to recall a field name, implementing a solution that
follows ECS, or proposing a change to the schema, the <<ecs-guidelines>> and
<<ecs-conventions>> will help get you there.

If you're wondering how to best capture event details that don't map to existing
ECS fields, head over to <<ecs-custom-fields-in-ecs>>
ECS fields, head over to <<ecs-custom-fields-in-ecs>>.

include::using-getting-started.asciidoc[][]
include::using-guidelines.asciidoc[]
include::using-conventions.asciidoc[]
include::using-custom-fields.asciidoc[]