From a47c8db84531cd8ea929c552e4caa69a36a4563e Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Wed, 30 Sep 2020 15:30:55 -0500 Subject: [PATCH 1/3] add discrete attribute for section headers --- scripts/templates/field_details.j2 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/scripts/templates/field_details.j2 b/scripts/templates/field_details.j2 index 0b1bb6e224..1ceedf55e0 100644 --- a/scripts/templates/field_details.j2 +++ b/scripts/templates/field_details.j2 @@ -5,6 +5,7 @@ {{ fieldset['description']|replace("\n", "\n\n") }} {# Field Details Table Header -#} +[discrete] ==== {{ fieldset['title'] }} Field Details [options="header"] @@ -67,6 +68,7 @@ example: `{{ field['example'] }}` {# do we have `nestings` or `reusable` sections to worry about? -#} {% if 'nestings' in fieldset or 'reusable' in fieldset -%} +[discrete] ==== Field Reuse {% if 'reusable' in fieldset -%} @@ -88,6 +90,7 @@ Note also that the `{{ fieldset['name'] }}` fields are not expected to be used d {% if 'nestings' in fieldset -%} [[ecs-{{ fieldset['name'] }}-nestings]] +[discrete] ===== Field sets that can be nested under {{ fieldset['title'] }} [options="header"] From f9f12b5a61e6c7321ef49fcc436a09c33abc4e8d Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Wed, 30 Sep 2020 15:32:35 -0500 Subject: [PATCH 2/3] update generated field details --- docs/field-details.asciidoc | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/docs/field-details.asciidoc b/docs/field-details.asciidoc index 773c61cce0..d13a5896c5 100644 --- a/docs/field-details.asciidoc +++ b/docs/field-details.asciidoc @@ -3,6 +3,7 @@ The `base` field set contains all fields which are at the root of the events. These fields are common across all types of events. +[discrete] ==== Base Field Details [options="header"] @@ -89,6 +90,7 @@ The agent fields contain the data about the software entity, if any, that collec Examples include Beats. Agents may also run on observers. ECS agent.* fields shall be populated with details of the agent running on the host or observer where the event happened or the measurement was taken. +[discrete] ==== Agent Field Details [options="header"] @@ -194,6 +196,7 @@ example: `6.0.0-rc2` An autonomous system (AS) is a collection of connected Internet Protocol (IP) routing prefixes under the control of one or more network operators on behalf of a single administrative entity or domain that presents a common, clearly defined routing policy to the internet. +[discrete] ==== Autonomous System Field Details [options="header"] @@ -236,6 +239,7 @@ example: `Google LLC` |===== +[discrete] ==== Field Reuse The `as` fields are expected to be nested at: `client.as`, `destination.as`, `server.as`, `source.as`. @@ -254,6 +258,7 @@ For TCP events, the client is the initiator of the TCP connection that sends the Client / server representations can add semantic context to an exchange, which is helpful to visualize the data in certain situations. If your context falls in that category, you should still ensure that source and destination are filled appropriately. +[discrete] ==== Client Field Details [options="header"] @@ -419,12 +424,14 @@ example: `co.uk` |===== +[discrete] ==== Field Reuse [[ecs-client-nestings]] +[discrete] ===== Field sets that can be nested under Client [options="header"] @@ -459,6 +466,7 @@ example: `co.uk` Fields related to the cloud or infrastructure the events are coming from. +[discrete] ==== Cloud Field Details [options="header"] @@ -612,6 +620,7 @@ example: `us-east-1` These fields contain information about binary code signatures. +[discrete] ==== Code Signature Field Details [options="header"] @@ -693,6 +702,7 @@ example: `true` |===== +[discrete] ==== Field Reuse The `code_signature` fields are expected to be nested at: `dll.code_signature`, `file.code_signature`, `process.code_signature`. @@ -709,6 +719,7 @@ Container fields are used for meta information about the specific container that These fields help correlate data based containers from any runtime. +[discrete] ==== Container Field Details [options="header"] @@ -807,6 +818,7 @@ Destination fields capture details about the receiver of a network exchange/pack Destination fields are usually populated in conjunction with source fields. The source and destination fields are considered the baseline and should always be filled if an event contains source and destination details from a network transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated. +[discrete] ==== Destination Field Details [options="header"] @@ -972,12 +984,14 @@ example: `co.uk` |===== +[discrete] ==== Field Reuse [[ecs-destination-nestings]] +[discrete] ===== Field sets that can be nested under Destination [options="header"] @@ -1022,6 +1036,7 @@ Many operating systems refer to "shared code libraries" with different names, bu * Dynamic library (`.dylib`) commonly used on macOS +[discrete] ==== DLL Field Details [options="header"] @@ -1060,12 +1075,14 @@ example: `C:\Windows\System32\kernel32.dll` |===== +[discrete] ==== Field Reuse [[ecs-dll-nestings]] +[discrete] ===== Field sets that can be nested under DLL [options="header"] @@ -1102,6 +1119,7 @@ Fields describing DNS queries and answers. DNS events should either represent a single DNS query prior to getting answers (`dns.type:query`) or they should represent a full exchange and contain the query details as well as all of the answers that were provided for this query (`dns.type:answer`). +[discrete] ==== DNS Field Details [options="header"] @@ -1386,6 +1404,7 @@ example: `answer` Meta-information specific to ECS. +[discrete] ==== ECS Field Details [options="header"] @@ -1418,6 +1437,7 @@ These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. +[discrete] ==== Error Field Details [options="header"] @@ -1506,6 +1526,7 @@ The event fields are used for context information about the log or metric event A log is defined as an event containing details of something that happened. Log events must include the time at which the thing happened. Examples of log events include a process starting on a host, a network packet being sent from a source to a destination, or a network connection between a client and a server being initiated or closed. A metric is defined as an event containing one or more numerical measurements and the time at which the measurement was taken. Examples of metric events include memory pressure measured on a host and device temperature. See the `event.kind` definition in this section for additional details about metric and state events. +[discrete] ==== Event Field Details [options="header"] @@ -1944,6 +1965,7 @@ A file is defined as a set of information that has been created on, or has exist File objects can be associated with host events, network events, and/or file events (e.g., those produced by File Integrity Monitoring [FIM] products or services). File fields provide details about the affected file associated with the event or metric. +[discrete] ==== File Field Details [options="header"] @@ -2254,12 +2276,14 @@ example: `1001` |===== +[discrete] ==== Field Reuse [[ecs-file-nestings]] +[discrete] ===== Field sets that can be nested under File [options="header"] @@ -2302,6 +2326,7 @@ Geo fields can carry data about a specific location related to an event. This geolocation information can be derived from techniques such as Geo IP, or be user-supplied. +[discrete] ==== Geo Field Details [options="header"] @@ -2420,6 +2445,7 @@ example: `Quebec` |===== +[discrete] ==== Field Reuse The `geo` fields are expected to be nested at: `client.geo`, `destination.geo`, `host.geo`, `observer.geo`, `server.geo`, `source.geo`. @@ -2434,6 +2460,7 @@ Note also that the `geo` fields are not expected to be used directly at the root The group fields are meant to represent groups that are relevant to the event. +[discrete] ==== Group Field Details [options="header"] @@ -2485,6 +2512,7 @@ type: keyword |===== +[discrete] ==== Field Reuse The `group` fields are expected to be nested at: `user.group`. @@ -2501,6 +2529,7 @@ The hash fields represent different hash algorithms and their values. Field names for common hashes (e.g. MD5, SHA1) are predefined. Add fields for other hashes by lowercasing the hash algorithm name and using underscore separators as appropriate (snake case, e.g. sha3_512). +[discrete] ==== Hash Field Details [options="header"] @@ -2563,6 +2592,7 @@ type: keyword |===== +[discrete] ==== Field Reuse The `hash` fields are expected to be nested at: `dll.hash`, `file.hash`, `process.hash`. @@ -2579,6 +2609,7 @@ A host is defined as a general computing instance. ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes. +[discrete] ==== Host Field Details [options="header"] @@ -2724,12 +2755,14 @@ example: `1325` |===== +[discrete] ==== Field Reuse [[ecs-host-nestings]] +[discrete] ===== Field sets that can be nested under Host [options="header"] @@ -2764,6 +2797,7 @@ example: `1325` Fields related to HTTP activity. Use the `url` field set to store the url of the request. +[discrete] ==== HTTP Field Details [options="header"] @@ -2957,6 +2991,7 @@ example: `1.1` The interface fields are used to record ingress and egress interface information when reported by an observer (e.g. firewall, router, load balancer) in the context of the observer handling a network connection. In the case of a single observer interface (e.g. network sensor on a span port) only the observer.ingress information should be populated. +[discrete] ==== Interface Field Details [options="header"] @@ -3006,6 +3041,7 @@ example: `eth0` |===== +[discrete] ==== Field Reuse The `interface` fields are expected to be nested at: `observer.egress.interface`, `observer.ingress.interface`. @@ -3024,6 +3060,7 @@ The log.* fields are typically populated with details about the logging mechanis The details specific to your event source are typically not logged under `log.*`, but rather in `event.*` or in other ECS fields. +[discrete] ==== Log Field Details [options="header"] @@ -3230,6 +3267,7 @@ The network is defined as the communication path over which a host or network ev The network.* fields should be populated with details about the network activity associated with an event. +[discrete] ==== Network Field Details [options="header"] @@ -3428,12 +3466,14 @@ example: `ipv4` |===== +[discrete] ==== Field Reuse [[ecs-network-nestings]] +[discrete] ===== Field sets that can be nested under Network [options="header"] @@ -3464,6 +3504,7 @@ An observer is defined as a special network, security, or application device use This could be a custom hardware appliance or a server that has been configured to run special network, security, or application software. Examples include firewalls, web proxies, intrusion detection/prevention systems, network monitoring sensors, web application firewalls, data loss prevention systems, and APM servers. The observer.* fields shall be populated with details of the system, if any, that detects, observes and/or creates a network, security, or application event or metric. Message queues and ETL components used in processing events or metrics are not considered observers in ECS. +[discrete] ==== Observer Field Details [options="header"] @@ -3655,12 +3696,14 @@ type: keyword |===== +[discrete] ==== Field Reuse [[ecs-observer-nestings]] +[discrete] ===== Field sets that can be nested under Observer [options="header"] @@ -3715,6 +3758,7 @@ The organization fields enrich data with information about the company or entity These fields help you arrange or filter data stored in an index by one or multiple organizations. +[discrete] ==== Organization Field Details [options="header"] @@ -3762,6 +3806,7 @@ Multi-fields: The OS fields contain information about the operating system. +[discrete] ==== Operating System Field Details [options="header"] @@ -3862,6 +3907,7 @@ example: `10.14.1` |===== +[discrete] ==== Field Reuse The `os` fields are expected to be nested at: `host.os`, `observer.os`, `user_agent.os`. @@ -3876,6 +3922,7 @@ Note also that the `os` fields are not expected to be used directly at the root These fields contain information about an installed software package. It contains general information about a package, such as name, version or size. It also contains installation details, such as time or location. +[discrete] ==== Package Field Details [options="header"] @@ -4066,6 +4113,7 @@ example: `1.12.9` These fields contain Windows Portable Executable (PE) metadata. +[discrete] ==== PE Header Field Details [options="header"] @@ -4169,6 +4217,7 @@ example: `Microsoft® Windows® Operating System` |===== +[discrete] ==== Field Reuse The `pe` fields are expected to be nested at: `dll.pe`, `file.pe`, `process.pe`. @@ -4185,6 +4234,7 @@ These fields contain information about a process. These fields can help you correlate metrics information with a process id/name from a log message. The `process.pid` often stays in the metric itself and is copied to the global field for correlation. +[discrete] ==== Process Field Details [options="header"] @@ -4452,6 +4502,7 @@ example: `/home/alice` |===== +[discrete] ==== Field Reuse The `process` fields are expected to be nested at: `process.parent`. @@ -4462,6 +4513,7 @@ Note also that the `process` fields may be used directly at the root of the even [[ecs-process-nestings]] +[discrete] ===== Field sets that can be nested under Process [options="header"] @@ -4502,6 +4554,7 @@ Note also that the `process` fields may be used directly at the root of the even Fields related to Windows Registry operations. +[discrete] ==== Registry Field Details [options="header"] @@ -4619,6 +4672,7 @@ Some pieces of information can be seen in many places in an ECS event. To facili A concrete example is IP addresses, which can be under host, observer, source, destination, client, server, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:192.0.2.15`. +[discrete] ==== Related Field Details [options="header"] @@ -4700,6 +4754,7 @@ Rule fields are used to capture the specifics of any observer or agent rules tha Examples of data sources that would populate the rule fields include: network admission control platforms, network or host IDS/IPS, network firewalls, web application firewalls, url filters, endpoint detection and response (EDR) systems, etc. +[discrete] ==== Rule Field Details [options="header"] @@ -4854,6 +4909,7 @@ For TCP events, the server is the receiver of the initial SYN packet(s) of the T Client / server representations can add semantic context to an exchange, which is helpful to visualize the data in certain situations. If your context falls in that category, you should still ensure that source and destination are filled appropriately. +[discrete] ==== Server Field Details [options="header"] @@ -5019,12 +5075,14 @@ example: `co.uk` |===== +[discrete] ==== Field Reuse [[ecs-server-nestings]] +[discrete] ===== Field sets that can be nested under Server [options="header"] @@ -5061,6 +5119,7 @@ The service fields describe the service for or from which the data was collected These fields help you find and correlate logs for a specific service and version. +[discrete] ==== Service Field Details [options="header"] @@ -5189,6 +5248,7 @@ Source fields capture details about the sender of a network exchange/packet. The Source fields are usually populated in conjunction with destination fields. The source and destination fields are considered the baseline and should always be filled if an event contains source and destination details from a network transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated. +[discrete] ==== Source Field Details [options="header"] @@ -5354,12 +5414,14 @@ example: `co.uk` |===== +[discrete] ==== Field Reuse [[ecs-source-nestings]] +[discrete] ===== Field sets that can be nested under Source [options="header"] @@ -5396,6 +5458,7 @@ Fields to classify events and alerts according to a threat taxonomy such as the These fields are for users to classify alerts from all of their sources (e.g. IDS, NGFW, etc.) within a common taxonomy. The threat.tactic.* are meant to capture the high level category of the threat (e.g. "impact"). The threat.technique.* fields are meant to capture which kind of approach is used by this detected threat, to accomplish the goal (e.g. "endpoint denial of service"). +[discrete] ==== Threat Field Details [options="header"] @@ -5580,6 +5643,7 @@ example: `https://attack.mitre.org/techniques/T1059/001/` Fields related to a TLS connection. These fields focus on the TLS protocol itself and intentionally avoids in-depth analysis of the related x.509 certificate files. +[discrete] ==== TLS Field Details [options="header"] @@ -5976,12 +6040,14 @@ example: `tls` |===== +[discrete] ==== Field Reuse [[ecs-tls-nestings]] +[discrete] ===== Field sets that can be nested under TLS [options="header"] @@ -6010,6 +6076,7 @@ example: `tls` Distributed tracing makes it possible to analyze performance throughout a microservice architecture all in one view. This is accomplished by tracing all of the requests - from the initial web request in the front-end service - to queries made through multiple back-end services. +[discrete] ==== Tracing Field Details [options="header"] @@ -6070,6 +6137,7 @@ example: `00f067aa0ba902b7` URL fields provide support for complete or partial URLs, and supports the breaking down into scheme, domain, path, and so on. +[discrete] ==== URL Field Details [options="header"] @@ -6290,6 +6358,7 @@ The user fields describe information about the user that is relevant to the even Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them. +[discrete] ==== User Field Details [options="header"] @@ -6410,6 +6479,7 @@ example: `["kibana_admin", "reporting_user"]` |===== +[discrete] ==== Field Reuse The `user` fields are expected to be nested at: `client.user`, `destination.user`, `host.user`, `server.user`, `source.user`. @@ -6420,6 +6490,7 @@ Note also that the `user` fields may be used directly at the root of the events. [[ecs-user-nestings]] +[discrete] ===== Field sets that can be nested under User [options="header"] @@ -6444,6 +6515,7 @@ The user_agent fields normally come from a browser request. They often show up in web service logs coming from the parsed user agent string. +[discrete] ==== User agent Field Details [options="header"] @@ -6512,12 +6584,14 @@ example: `12.0` |===== +[discrete] ==== Field Reuse [[ecs-user_agent-nestings]] +[discrete] ===== Field sets that can be nested under User agent [options="header"] @@ -6546,6 +6620,7 @@ Network.inner VLAN fields are used to report inner q-in-q 802.1q tags (multiple Observer.ingress and observer.egress VLAN values are used to record observer specific information when observer events contain discrete ingress and egress VLAN information, typically provided by firewalls, routers, or load balancers. +[discrete] ==== VLAN Field Details [options="header"] @@ -6582,6 +6657,7 @@ example: `outside` |===== +[discrete] ==== Field Reuse The `vlan` fields are expected to be nested at: `network.inner.vlan`, `network.vlan`, `observer.egress.vlan`, `observer.ingress.vlan`. @@ -6596,6 +6672,7 @@ Note also that the `vlan` fields are not expected to be used directly at the roo The vulnerability fields describe information about a vulnerability that is relevant to an event. +[discrete] ==== Vulnerability Field Details [options="header"] @@ -6799,6 +6876,7 @@ example: `Critical` This implements the common core fields for x509 certificates. This information is likely logged with TLS sessions, digital signatures found in executable binaries, S/MIME information in email bodies, or analysis of files on disk. When only a single certificate is logged in an event, it should be nested under `file`. When hashes of the DER-encoded certificate are available, the `hash` data set should be populated as well (e.g. `file.hash.sha256`). For events that contain certificate information for both sides of the connection, the x509 object could be nested under the respective side of the connection information (e.g. `tls.server.x509`). +[discrete] ==== x509 Certificate Field Details [options="header"] @@ -7160,6 +7238,7 @@ example: `3` |===== +[discrete] ==== Field Reuse The `x509` fields are expected to be nested at: `file.x509`, `tls.client.x509`, `tls.server.x509`. From e3c668afa25d9d8217f9d74b759aa4174beb53f5 Mon Sep 17 00:00:00 2001 From: Eric Beahan Date: Wed, 30 Sep 2020 15:39:07 -0500 Subject: [PATCH 3/3] changelog --- CHANGELOG.next.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.next.md b/CHANGELOG.next.md index b3b73b8bda..54ae430335 100644 --- a/CHANGELOG.next.md +++ b/CHANGELOG.next.md @@ -46,6 +46,7 @@ Thanks, you're awesome :-) --> #### Improvements * Field details Jinja2 template components have been consolidated into one template #897 +* Add `[discrete]` marker before each section header in field details. #989 #### Deprecated