diff --git a/docs/images/ecs-getting-started-dashboard.png b/docs/images/ecs-getting-started-dashboard.png new file mode 100644 index 0000000000..5d10c1db52 Binary files /dev/null and b/docs/images/ecs-getting-started-dashboard.png differ diff --git a/docs/index.asciidoc b/docs/index.asciidoc index c4054d9ed9..2296f08408 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -34,6 +34,11 @@ ECS has been scoped to accommodate a wide variety of events, spanning: Logstash, Elasticsearch ingest node, all of the above, or none of the above. - *Consumers*: whether consumed by API, Kibana queries, dashboards, apps, or other means. +[float] +=== New to ECS? + +If you're new to ECS and looking for an introduction to its benefits and examples of the +core concepts, <> is a great place to start. [float] === My events don't map with ECS diff --git a/docs/using-getting-started.asciidoc b/docs/using-getting-started.asciidoc new file mode 100644 index 0000000000..6a1d45645f --- /dev/null +++ b/docs/using-getting-started.asciidoc @@ -0,0 +1,284 @@ +[[ecs-getting-started]] +=== Getting Started + +ECS enables and encourages users to normalize event data in order +to better analyze, visualize, and correlate their events. Collected events +can be normalized at ingest time, consistently searched across indices, +and visualized predictably. + +Note that when adopting an Elastic solution, such as https://www.elastic.co/observability[Observability] +or https://www.elastic.co/security[Security], all events will map to ECS out of the box. Elastic +provides an extensive set of https://www.elastic.co/integrations[integrations] to simplify +ingesting your data sources. + +If you rely on custom data pipelines and/or building content around specific needs, +ECS can still help to alleviate the challenges of searching, analyzing, and visualizing +across your data. Let's see how using a common schema can simplify the search experience, +and then take a look at how an event's contents can be mapped to ECS field sets. + +[float] +==== Simplified Search + +With ECS defining a normalized schema across all of your data sources, querying +against those sources is simplified. Consider searching for a particular source IP +address prior to adopting ECS. All the various data sources and their field mappings +would need to be considered in your query: + +[source,sh] +----------- +src:10.42.42.42 OR client_ip:10.42.42.42 OR apache.access.remote_ip:10.42.42.42 OR +context.user.ip:10.42.42.42 OR src_ip:10.42.42.42 +----------- + +With all sources mapped to ECS, the query becomes much simpler: + +[source,sh] +----------- +source.ip:10.42.42.42 +----------- + +Not only does this simplify writing queries, but saved queries shared with other +users become much more obvious. To gain familiarity with ECS fields, you can also +take a look at the <> section. + +[float] +==== Unified Visualizations + +With normalized data from different data sources, building insightful visualizations +across sources is simple. From a single, centralized dashboard, events from web servers, +IDS/IPS devices, and firewalls can be aggregated and visualized, and enhanced with drill-downs, +and pivoting for delving into deeper investigations. Centralized monitoring of diverse data +sources is straightforward with normalized ECS data. + +[role="screenshot"] +image:images/ecs-getting-started-dashboard.png[Simplify visualization using ECS] + + +[float] +==== Translating Data Sources + +To align events to ECS, some sort of parsing will usually be necessary +to transform the contents of the original event into the relevant ECS fields. Depending on +how you've designed your Elastic Stack data ingestion pipelines, the amount of work to parse +your events will vary. + +For example, an Apache web server log event: + +[source,sh] +----------- +10.42.42.42 - - [15/Jul/2020:20:48:32 +0000] "GET /content HTTP/1.1" 200 2571 "-" +"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) +Chrome/83.0.4103.106 Safari/537.36" +----------- + +In order to map this event to ECS, the contents of the event is associated with the +appropriate ECS fields. + +[options="header"] +|===== +| Field Name | Value + +// =============================================================== + +| @timestamp +| `2020-07-15T20:20:48.000Z` + +// ============================================================== + +| event.original +| 10.42.42.42 - - [15/Jul/2020:20:48:32 +0000] "GET /content HTTP/1.1" 200 2571 "-" +"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) +Chrome/83.0.4103.106 Safari/537.36 + +// ============================================================== + +| http.request.method +| GET + +// ============================================================== + +| http.response.body.bytes +| 2571 + +// ============================================================== + +| http.response.status_code +| 200 + +// ============================================================== + +| http.version +| 1.1 + +// ============================================================== + +| message +| GET /content HTTP/1.1" 200 2571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36 + +// ============================================================== + +| source.address +| 10.42.42.42 + +// ============================================================== + +| source.ip +| 10.42.42.42 + +// ============================================================== + +| url.original +| `/content` + +// ============================================================== + +| user_agent.original +| `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36` + +|===== + +Beyond extracting values that are present in the original event, we also populate +other fields to provide additional context about the event itself. + +* `ecs.version`: States which version of ECS the ingest pipeline was developed against. +* `event.dataset` and `event.module`: Answers "where is this event from" and are expected to have a + hardcoded value per pipeline, per source. +* `event.kind`, `event.category`, `event.type`, and `event.outcome`: The https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html[categorization fields] + should also be hardcoded using knowledge of each type of event the source emits. The contents of + these fields are limited to the specifically allowed values detailed in the ECS documentation. + +[options="header"] +|===== +| Field Name | Value + +// =============================================================== + +| ecs.version +| 1.5.0 + +// ============================================================== + +| event.module +| apache + +// =============================================================== + +| event.dataset +| apache.access + +// ============================================================== + +| event.kind +| event + +// =============================================================== + +| event.category +| [ "web", "access" ] + +// ============================================================== + +| event.outcome +| success + +|===== + +Lastly, existing field values can be interpreted or enriched using a processor, with the results +populating additional fields in the final event. + +* The `user_agent` processor extracts details from the original user agent string, `user_agent.original`. +* IP fields like `source.ip` can provide enrichment using the `geopip` processor to add information about the + location and autonomous system number (ASN) associated with an IP address. +* The `registered domain` processor reads a field containing a hostname and writes the registered domain to + another field +* Event collectors, such as https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html[Beats], can enrich + each event with metadata from the machine's hosting provider (cloud) and/or from the host machine (host). + +Here are some examples of additional fields processed by metadata or parser processors. + +[options="header"] +|===== +| Field Name | Value | Processor + +// ============================================================== + +| host.architecture +| x86_64 +| `add_host_metadata` + +// ============================================================== + +| host.hostname +| mbp.example.com +| `add_host_metadata` + +// ============================================================== + +| host.ip +| [ "192.168.1.100" ] +| `add_host_metadata` + +// ============================================================== + +| host.os.family +| darwin +| `add_host_metadata` + +// ============================================================== + +| host.os.kernel +| 19.4.0 +| `add_host_metadata` + +// ============================================================== + +| host.os.name +| Mac OS X +| `add_host_metadata` + +// ============================================================== + +| host.os.version +| 10.15.4 +| `add_host_metadata` + +// ============================================================== + +| user_agent.name +| Chrome +| `user_agent` + +// ============================================================== + +| user_agent.os.full +| Mac OS X 10.15.4 +| `user_agent` + +// ============================================================== + +| user_agent.os.name +| Mac OS X +| `user_agent` + +// ============================================================== + +| user_agent.os.version +| 10.15.4 +| `user_agent` + +// ============================================================== + +| user_agent.version +| 83.0.4103.106 +| `user_agent` + +|===== + +[float] +==== Field Mapping Reference Guides + +We've covered at a high level how to map your events to ECS. Now if you'd like your events to render well in the Elastic +solutions, check out the reference guides below to learn more about each: + +* https://www.elastic.co/guide/en/logs/guide/current/logs-fields-reference.html[Logs UI fields reference] +* https://www.elastic.co/guide/en/security/master/siem-field-reference.html[Elastic Security fields reference] diff --git a/docs/using.asciidoc b/docs/using.asciidoc index 4cb25f7a9e..fbce72bb27 100644 --- a/docs/using.asciidoc +++ b/docs/using.asciidoc @@ -4,13 +4,17 @@ ECS fields follow a series of guidelines, to ensure a consistent and predictable feel, across various use cases. +If you're new to ECS and would like an introduction on implementing and using +the schema, check out the <> guide. + Whether you're trying to recall a field name, implementing a solution that follows ECS, or proposing a change to the schema, the <> and <> will help get you there. If you're wondering how to best capture event details that don't map to existing -ECS fields, head over to <> +ECS fields, head over to <>. +include::using-getting-started.asciidoc[][] include::using-guidelines.asciidoc[] include::using-conventions.asciidoc[] include::using-custom-fields.asciidoc[]