From f5903085002e34f762920ecbdc479ec92584f004 Mon Sep 17 00:00:00 2001 From: ruflin Date: Tue, 17 Jul 2018 14:24:35 +0200 Subject: [PATCH] Proposal: Introduce connection prefix, move source / destination There have been recently several discussions around source, destination and connection recently, especially in https://github.com/elastic/ecs/issues/9. The conclusion from my side is that source and destination normally belongs to a connection and we actually miss a connection prefix. Also some information from network like `forward_ip` more belong to a connection then network. An additional change I made to source and destination is that they both contain now a host prefix. All the fields in source and destination also exist in `host`. The host prefix can be reused here too. This makes ECS very predictable that every time `host.*` shows up it will contain the same fields. Also source and destination could contain additional data like the location, see https://github.com/elastic/ecs/issues/50 for more details. The connection fields now look as following: | Field | Description | Type | |---|---|---|---|---| | `connection.destination.host.ip` | IP address of the destination.
Can be one or multiple IPv4 or IPv6 addresses. | ip | | `connection.destination.host.name` | Hostname of the destination. | keyword | | `connection.destination.host.port` | Port of the destination. | long | | `connection.destination.host.mac` | MAC address of the destination. | keyword | | `connection.destination.host.domain` | Destination domain. | keyword | | `connection.destination.host.subdomain` | Destination subdomain. | keyword | | `connection.source.host.ip` | IP address of the source.
Can be one or multiple IPv4 or IPv6 addresses. | ip | | `connection.source.host.name` | Hostname of the source. | keyword | | `connection.source.host.port` | Port of the source. | long | | `connection.source.host.mac` | MAC address of the source. | keyword | | `connection.source.host.domain` | Source domain. | keyword | | `connection.source.host.subdomain` | Source subdomain. | keyword | | `connection.direction` | Direction of the network traffic.
Recommended values are:
* inbound
* outbound
* unknown | keyword | | `connection.forwarded_ip` | Host IP address when the source IP address is the proxy. | ip | I opened a PR to discuss this instead of an issue as it will allow us to discuss the high level parts as comment but also details directly in the code. --- README.md | 58 ++++++++---------- schema.csv | 28 ++++----- schemas/connection.yml | 83 +++++++++++++++++++++++++ schemas/destination.yml | 36 ----------- schemas/network.yml | 16 ----- schemas/source.yml | 37 ------------ template.json | 130 ++++++++++++++++++++++------------------ 7 files changed, 192 insertions(+), 196 deletions(-) create mode 100644 schemas/connection.yml delete mode 100644 schemas/destination.yml delete mode 100644 schemas/source.yml diff --git a/README.md b/README.md index 1335a57bc2..72f962ebb6 100644 --- a/README.md +++ b/README.md @@ -26,8 +26,8 @@ ECS defines these fields. * [Base fields](#base) * [Agent fields](#agent) * [Cloud fields](#cloud) + * [Connection fields](#connection) * [Container fields](#container) - * [Destination fields](#destination) * [Device fields](#device) * [Error fields](#error) * [Event fields](#event) @@ -42,7 +42,6 @@ ECS defines these fields. * [Operating System fields](#os) * [Process fields](#process) * [Service fields](#service) - * [Source fields](#source) * [TLS fields](#tls) * [URL fields](#url) * [User fields](#user) @@ -96,6 +95,29 @@ Fields related to the cloud or infrastructure the events are coming from. Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on. +## Connection fields + +Connection related fields. + + +| Field | Description | Type | Multi Field | Example | +|---|---|---|---|---| +| `connection.destination.host.ip` | IP address of the destination.
Can be one or multiple IPv4 or IPv6 addresses. | ip | | | +| `connection.destination.host.name` | Hostname of the destination. | keyword | | | +| `connection.destination.host.port` | Port of the destination. | long | | | +| `connection.destination.host.mac` | MAC address of the destination. | keyword | | | +| `connection.destination.host.domain` | Destination domain. | keyword | | | +| `connection.destination.host.subdomain` | Destination subdomain. | keyword | | | +| `connection.source.host.ip` | IP address of the source.
Can be one or multiple IPv4 or IPv6 addresses. | ip | | | +| `connection.source.host.name` | Hostname of the source. | keyword | | | +| `connection.source.host.port` | Port of the source. | long | | | +| `connection.source.host.mac` | MAC address of the source. | keyword | | | +| `connection.source.host.domain` | Source domain. | keyword | | | +| `connection.source.host.subdomain` | Source subdomain. | keyword | | | +| `connection.direction` | Direction of the network traffic.
Recommended values are:
* inbound
* outbound
* unknown | keyword | | `inbound` | +| `connection.forwarded_ip` | Host IP address when the source IP address is the proxy. | ip | | `192.1.1.2` | + + ## Container fields Container fields are used for meta information about the specific container that is the source of information. These fields help correlate data based containers from any runtime. @@ -111,21 +133,6 @@ Container fields are used for meta information about the specific container that | `container.labels` | Image labels. | object | | | -## Destination fields - -Destination fields describe details about the destination of a packet/event. - - -| Field | Description | Type | Multi Field | Example | -|---|---|---|---|---| -| `destination.ip` | IP address of the destination.
Can be one or multiple IPv4 or IPv6 addresses. | ip | | | -| `destination.hostname` | Hostname of the destination. | keyword | | | -| `destination.port` | Port of the destination. | long | | | -| `destination.mac` | MAC address of the destination. | keyword | | | -| `destination.domain` | Destination domain. | keyword | | | -| `destination.subdomain` | Destination subdomain. | keyword | | | - - ## Device fields Device fields are used to provide additional information about the device that is the source of the information. This could be a firewall, network device, etc. @@ -286,8 +293,6 @@ Fields related to network data. | Field | Description | Type | Multi Field | Example | |---|---|---|---|---| | `network.protocol` | Network protocol name. | keyword | | `http` | -| `network.direction` | Direction of the network traffic.
Recommended values are:
* inbound
* outbound
* unknown | keyword | | `inbound` | -| `network.forwarded_ip` | Host IP address when the source IP address is the proxy. | ip | | `192.1.1.2` | | `network.inbound.bytes` | Network inbound bytes. | long | | `184` | | `network.inbound.packets` | Network inbound packets. | long | | `12` | | `network.outbound.bytes` | Network outbound bytes. | long | | `184` | @@ -349,21 +354,6 @@ The service fields describe the service for or from which the data was collected | `service.ephemeral_id` | Ephemeral identifier of this service (if one exists).
This id normally changes across restarts, but `service.id` does not. | keyword | | `8a4f500f` | -## Source fields - -Source fields describe details about the source of the event. - - -| Field | Description | Type | Multi Field | Example | -|---|---|---|---|---| -| `source.ip` | IP address of the source.
Can be one or multiple IPv4 or IPv6 addresses. | ip | | | -| `source.hostname` | Hostname of the source. | keyword | | | -| `source.port` | Port of the source. | long | | | -| `source.mac` | MAC address of the source. | keyword | | | -| `source.domain` | Source domain. | keyword | | | -| `source.subdomain` | Source subdomain. | keyword | | | - - ## TLS fields The tls fields contain the TLS related data about a specific connection. diff --git a/schema.csv b/schema.csv index 3a028d6323..be5dcb3690 100644 --- a/schema.csv +++ b/schema.csv @@ -14,18 +14,26 @@ cloud.instance.name,keyword,0, cloud.machine.type,keyword,0,t2.medium cloud.provider,keyword,0,ec2 cloud.region,keyword,0,us-east-1 +connection.destination.host.domain,keyword,1, +connection.destination.host.ip,ip,0, +connection.destination.host.mac,keyword,0, +connection.destination.host.name,keyword,0, +connection.destination.host.port,long,0, +connection.destination.host.subdomain,keyword,1, +connection.direction,keyword,0,inbound +connection.forwarded_ip,ip,0,192.1.1.2 +connection.source.host.domain,keyword,1, +connection.source.host.ip,ip,0, +connection.source.host.mac,keyword,1, +connection.source.host.name,keyword,0, +connection.source.host.port,long,1, +connection.source.host.subdomain,keyword,1, container.id,keyword,0, container.image.name,keyword,0, container.image.tag,keyword,0, container.labels,object,0, container.name,keyword,0, container.runtime,keyword,0,docker -destination.domain,keyword,1, -destination.hostname,keyword,0, -destination.ip,ip,0, -destination.mac,keyword,0, -destination.port,long,0, -destination.subdomain,keyword,1, device.hostname,keyword,0, device.ip,ip,0, device.mac,keyword,0, @@ -93,8 +101,6 @@ log.level,keyword,0,ERR log.line,long,0,18 log.message,keyword,1,Sep 19 08:26:10 localhost My log log.offset,long,0,12 -network.direction,keyword,0,inbound -network.forwarded_ip,ip,0,192.1.1.2 network.inbound.bytes,long,0,184 network.inbound.packets,long,0,12 network.outbound.bytes,long,0,184 @@ -119,12 +125,6 @@ service.name,keyword,1,elasticsearch service.state,keyword,1, service.type,keyword,1, service.version,keyword,1,3.2.4 -source.domain,keyword,1, -source.hostname,keyword,0, -source.ip,ip,0, -source.mac,keyword,1, -source.port,long,1, -source.subdomain,keyword,1, tls.certificates,keyword,0, tls.ciphersuite,keyword,0,ECDHE-ECDSA-AES-128-CBC-SHA tls.servername,keyword,0,localhost diff --git a/schemas/connection.yml b/schemas/connection.yml new file mode 100644 index 0000000000..e1a7076aae --- /dev/null +++ b/schemas/connection.yml @@ -0,0 +1,83 @@ +--- +- name: connection + title: Connection + group: 2 + description: > + Connection related fields. + fields: + - name: destination.host.ip + type: ip + description: > + IP address of the destination. + + Can be one or multiple IPv4 or IPv6 addresses. + - name: destination.host.name + type: keyword + description: > + Hostname of the destination. + - name: destination.host.port + type: long + description: > + Port of the destination. + - name: destination.host.mac + type: keyword + description: > + MAC address of the destination. + - name: destination.host.domain + type: keyword + phase: 1 + description: > + Destination domain. + - name: destination.host.subdomain + type: keyword + phase: 1 + description: > + Destination subdomain. + + - name: source.host.ip + type: ip + description: > + IP address of the source. + + Can be one or multiple IPv4 or IPv6 addresses. + - name: source.host.name + type: keyword + description: > + Hostname of the source. + - name: source.host.port + type: long + phase: 1 + description: > + Port of the source. + - name: source.host.mac + type: keyword + phase: 1 + description: > + MAC address of the source. + - name: source.host.domain + type: keyword + phase: 1 + description: > + Source domain. + - name: source.host.subdomain + type: keyword + phase: 1 + description: > + Source subdomain. + + - name: direction + type: keyword + description: > + Direction of the network traffic. + + Recommended values are: + * inbound + * outbound + * unknown + example: inbound + + - name: forwarded_ip + type: ip + description: > + Host IP address when the source IP address is the proxy. + example: 192.1.1.2 diff --git a/schemas/destination.yml b/schemas/destination.yml deleted file mode 100644 index 01446229c2..0000000000 --- a/schemas/destination.yml +++ /dev/null @@ -1,36 +0,0 @@ ---- -- name: destination - title: Destination - group: 2 - description: > - Destination fields describe details about the destination of a - packet/event. - fields: - - name: ip - type: ip - description: > - IP address of the destination. - - Can be one or multiple IPv4 or IPv6 addresses. - - name: hostname - type: keyword - description: > - Hostname of the destination. - - name: port - type: long - description: > - Port of the destination. - - name: mac - type: keyword - description: > - MAC address of the destination. - - name: domain - type: keyword - phase: 1 - description: > - Destination domain. - - name: subdomain - type: keyword - phase: 1 - description: > - Destination subdomain. diff --git a/schemas/network.yml b/schemas/network.yml index 9e85393757..db1e14effa 100644 --- a/schemas/network.yml +++ b/schemas/network.yml @@ -10,22 +10,6 @@ description: > Network protocol name. example: http - - name: direction - type: keyword - description: > - Direction of the network traffic. - - Recommended values are: - * inbound - * outbound - * unknown - example: inbound - - - name: forwarded_ip - type: ip - description: > - Host IP address when the source IP address is the proxy. - example: 192.1.1.2 # Metrics - name: inbound.bytes diff --git a/schemas/source.yml b/schemas/source.yml deleted file mode 100644 index d9afe3dbce..0000000000 --- a/schemas/source.yml +++ /dev/null @@ -1,37 +0,0 @@ ---- -- name: source - title: Source - group: 2 - description: > - Source fields describe details about the source of the event. - fields: - - name: ip - type: ip - description: > - IP address of the source. - - Can be one or multiple IPv4 or IPv6 addresses. - - name: hostname - type: keyword - description: > - Hostname of the source. - - name: port - type: long - phase: 1 - description: > - Port of the source. - - name: mac - type: keyword - phase: 1 - description: > - MAC address of the source. - - name: domain - type: keyword - phase: 1 - description: > - Source domain. - - name: subdomain - type: keyword - phase: 1 - description: > - Source subdomain. diff --git a/template.json b/template.json index dd6d6339c3..95230e1ba0 100644 --- a/template.json +++ b/template.json @@ -87,6 +87,77 @@ } } }, + "connection": { + "properties": { + "destination": { + "properties": { + "host": { + "properties": { + "domain": { + "ignore_above": 1024, + "type": "keyword" + }, + "ip": { + "type": "ip" + }, + "mac": { + "ignore_above": 1024, + "type": "keyword" + }, + "name": { + "ignore_above": 1024, + "type": "keyword" + }, + "port": { + "type": "long" + }, + "subdomain": { + "ignore_above": 1024, + "type": "keyword" + } + } + } + } + }, + "direction": { + "ignore_above": 1024, + "type": "keyword" + }, + "forwarded_ip": { + "type": "ip" + }, + "source": { + "properties": { + "host": { + "properties": { + "domain": { + "ignore_above": 1024, + "type": "keyword" + }, + "ip": { + "type": "ip" + }, + "mac": { + "ignore_above": 1024, + "type": "keyword" + }, + "name": { + "ignore_above": 1024, + "type": "keyword" + }, + "port": { + "type": "long" + }, + "subdomain": { + "ignore_above": 1024, + "type": "keyword" + } + } + } + } + } + } + }, "container": { "properties": { "id": { @@ -118,32 +189,6 @@ } } }, - "destination": { - "properties": { - "domain": { - "ignore_above": 1024, - "type": "keyword" - }, - "hostname": { - "ignore_above": 1024, - "type": "keyword" - }, - "ip": { - "type": "ip" - }, - "mac": { - "ignore_above": 1024, - "type": "keyword" - }, - "port": { - "type": "long" - }, - "subdomain": { - "ignore_above": 1024, - "type": "keyword" - } - } - }, "device": { "properties": { "hostname": { @@ -492,13 +537,6 @@ }, "network": { "properties": { - "direction": { - "ignore_above": 1024, - "type": "keyword" - }, - "forwarded_ip": { - "type": "ip" - }, "inbound": { "properties": { "bytes": { @@ -617,32 +655,6 @@ } } }, - "source": { - "properties": { - "domain": { - "ignore_above": 1024, - "type": "keyword" - }, - "hostname": { - "ignore_above": 1024, - "type": "keyword" - }, - "ip": { - "type": "ip" - }, - "mac": { - "ignore_above": 1024, - "type": "keyword" - }, - "port": { - "type": "long" - }, - "subdomain": { - "ignore_above": 1024, - "type": "keyword" - } - } - }, "tags": { "ignore_above": 1024, "type": "keyword"