Skip to content

Commit

Permalink
Proposal: Introduce connection prefix, move source / destination
Browse files Browse the repository at this point in the history
There have been recently several discussions around source, destination and connection recently, especially in elastic#9. The conclusion from my side is that source and destination normally belongs to a connection and we actually miss a connection prefix. Also some information from network like `forward_ip` more belong to a connection then network.

An additional change I made to source and destination is that they both contain now a host prefix. All the fields in source and destination also exist in `host`. The host prefix can be reused here too. This makes ECS very predictable that every time `host.*` shows up it will contain the same fields. Also source and destination could contain additional data like the location, see elastic#50 for more details.

The connection fields now look as following:

| Field  | Description  | Type  |
|---|---|---|---|---|
| <a name="connection.destination.host.ip"></a>`connection.destination.host.ip`  | IP address of the destination.<br/>Can be one or multiple IPv4 or IPv6 addresses.  | ip  |
| <a name="connection.destination.host.name"></a>`connection.destination.host.name`  | Hostname of the destination.  | keyword  |
| <a name="connection.destination.host.port"></a>`connection.destination.host.port`  | Port of the destination.  | long  |
| <a name="connection.destination.host.mac"></a>`connection.destination.host.mac`  | MAC address of the destination.  | keyword  |
| <a name="connection.destination.host.domain"></a>`connection.destination.host.domain`  | Destination domain.  | keyword  |
| <a name="connection.destination.host.subdomain"></a>`connection.destination.host.subdomain`  | Destination subdomain.  | keyword  |
| <a name="connection.source.host.ip"></a>`connection.source.host.ip`  | IP address of the source.<br/>Can be one or multiple IPv4 or IPv6 addresses.  | ip  |
| <a name="connection.source.host.name"></a>`connection.source.host.name`  | Hostname of the source.  | keyword  |
| <a name="connection.source.host.port"></a>`connection.source.host.port`  | Port of the source.  | long  |
| <a name="connection.source.host.mac"></a>`connection.source.host.mac`  | MAC address of the source.  | keyword  |
| <a name="connection.source.host.domain"></a>`connection.source.host.domain`  | Source domain.  | keyword  |
| <a name="connection.source.host.subdomain"></a>`connection.source.host.subdomain`  | Source subdomain.  | keyword  |
| <a name="connection.direction"></a>`connection.direction`  | Direction of the network traffic.<br/>Recommended values are:<br/>  * inbound<br/>  * outbound<br/>  * unknown  | keyword  |
| <a name="connection.forwarded_ip"></a>`connection.forwarded_ip`  | Host IP address when the source IP address is the proxy.  | ip  |

I opened a PR to discuss this instead of an issue as it will allow us to discuss the high level parts as comment but also details directly in the code.
  • Loading branch information
ruflin committed Jul 17, 2018
1 parent a256a0c commit f590308
Show file tree
Hide file tree
Showing 7 changed files with 192 additions and 196 deletions.
58 changes: 24 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ ECS defines these fields.
* [Base fields](#base)
* [Agent fields](#agent)
* [Cloud fields](#cloud)
* [Connection fields](#connection)
* [Container fields](#container)
* [Destination fields](#destination)
* [Device fields](#device)
* [Error fields](#error)
* [Event fields](#event)
Expand All @@ -42,7 +42,6 @@ ECS defines these fields.
* [Operating System fields](#os)
* [Process fields](#process)
* [Service fields](#service)
* [Source fields](#source)
* [TLS fields](#tls)
* [URL fields](#url)
* [User fields](#user)
Expand Down Expand Up @@ -96,6 +95,29 @@ Fields related to the cloud or infrastructure the events are coming from.
Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.


## <a name="connection"></a> Connection fields

Connection related fields.


| Field | Description | Type | Multi Field | Example |
|---|---|---|---|---|
| <a name="connection.destination.host.ip"></a>`connection.destination.host.ip` | IP address of the destination.<br/>Can be one or multiple IPv4 or IPv6 addresses. | ip | | |
| <a name="connection.destination.host.name"></a>`connection.destination.host.name` | Hostname of the destination. | keyword | | |
| <a name="connection.destination.host.port"></a>`connection.destination.host.port` | Port of the destination. | long | | |
| <a name="connection.destination.host.mac"></a>`connection.destination.host.mac` | MAC address of the destination. | keyword | | |
| <a name="connection.destination.host.domain"></a>`connection.destination.host.domain` | Destination domain. | keyword | | |
| <a name="connection.destination.host.subdomain"></a>`connection.destination.host.subdomain` | Destination subdomain. | keyword | | |
| <a name="connection.source.host.ip"></a>`connection.source.host.ip` | IP address of the source.<br/>Can be one or multiple IPv4 or IPv6 addresses. | ip | | |
| <a name="connection.source.host.name"></a>`connection.source.host.name` | Hostname of the source. | keyword | | |
| <a name="connection.source.host.port"></a>`connection.source.host.port` | Port of the source. | long | | |
| <a name="connection.source.host.mac"></a>`connection.source.host.mac` | MAC address of the source. | keyword | | |
| <a name="connection.source.host.domain"></a>`connection.source.host.domain` | Source domain. | keyword | | |
| <a name="connection.source.host.subdomain"></a>`connection.source.host.subdomain` | Source subdomain. | keyword | | |
| <a name="connection.direction"></a>`connection.direction` | Direction of the network traffic.<br/>Recommended values are:<br/> * inbound<br/> * outbound<br/> * unknown | keyword | | `inbound` |
| <a name="connection.forwarded_ip"></a>`connection.forwarded_ip` | Host IP address when the source IP address is the proxy. | ip | | `192.1.1.2` |


## <a name="container"></a> Container fields

Container fields are used for meta information about the specific container that is the source of information. These fields help correlate data based containers from any runtime.
Expand All @@ -111,21 +133,6 @@ Container fields are used for meta information about the specific container that
| <a name="container.labels"></a>`container.labels` | Image labels. | object | | |


## <a name="destination"></a> Destination fields

Destination fields describe details about the destination of a packet/event.


| Field | Description | Type | Multi Field | Example |
|---|---|---|---|---|
| <a name="destination.ip"></a>`destination.ip` | IP address of the destination.<br/>Can be one or multiple IPv4 or IPv6 addresses. | ip | | |
| <a name="destination.hostname"></a>`destination.hostname` | Hostname of the destination. | keyword | | |
| <a name="destination.port"></a>`destination.port` | Port of the destination. | long | | |
| <a name="destination.mac"></a>`destination.mac` | MAC address of the destination. | keyword | | |
| <a name="destination.domain"></a>`destination.domain` | Destination domain. | keyword | | |
| <a name="destination.subdomain"></a>`destination.subdomain` | Destination subdomain. | keyword | | |


## <a name="device"></a> Device fields

Device fields are used to provide additional information about the device that is the source of the information. This could be a firewall, network device, etc.
Expand Down Expand Up @@ -286,8 +293,6 @@ Fields related to network data.
| Field | Description | Type | Multi Field | Example |
|---|---|---|---|---|
| <a name="network.protocol"></a>`network.protocol` | Network protocol name. | keyword | | `http` |
| <a name="network.direction"></a>`network.direction` | Direction of the network traffic.<br/>Recommended values are:<br/> * inbound<br/> * outbound<br/> * unknown | keyword | | `inbound` |
| <a name="network.forwarded_ip"></a>`network.forwarded_ip` | Host IP address when the source IP address is the proxy. | ip | | `192.1.1.2` |
| <a name="network.inbound.bytes"></a>`network.inbound.bytes` | Network inbound bytes. | long | | `184` |
| <a name="network.inbound.packets"></a>`network.inbound.packets` | Network inbound packets. | long | | `12` |
| <a name="network.outbound.bytes"></a>`network.outbound.bytes` | Network outbound bytes. | long | | `184` |
Expand Down Expand Up @@ -349,21 +354,6 @@ The service fields describe the service for or from which the data was collected
| <a name="service.ephemeral_id"></a>`service.ephemeral_id` | Ephemeral identifier of this service (if one exists).<br/>This id normally changes across restarts, but `service.id` does not. | keyword | | `8a4f500f` |


## <a name="source"></a> Source fields

Source fields describe details about the source of the event.


| Field | Description | Type | Multi Field | Example |
|---|---|---|---|---|
| <a name="source.ip"></a>`source.ip` | IP address of the source.<br/>Can be one or multiple IPv4 or IPv6 addresses. | ip | | |
| <a name="source.hostname"></a>`source.hostname` | Hostname of the source. | keyword | | |
| <a name="source.port"></a>`source.port` | Port of the source. | long | | |
| <a name="source.mac"></a>`source.mac` | MAC address of the source. | keyword | | |
| <a name="source.domain"></a>`source.domain` | Source domain. | keyword | | |
| <a name="source.subdomain"></a>`source.subdomain` | Source subdomain. | keyword | | |


## <a name="tls"></a> TLS fields

The tls fields contain the TLS related data about a specific connection.
Expand Down
28 changes: 14 additions & 14 deletions schema.csv
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,26 @@ cloud.instance.name,keyword,0,
cloud.machine.type,keyword,0,t2.medium
cloud.provider,keyword,0,ec2
cloud.region,keyword,0,us-east-1
connection.destination.host.domain,keyword,1,
connection.destination.host.ip,ip,0,
connection.destination.host.mac,keyword,0,
connection.destination.host.name,keyword,0,
connection.destination.host.port,long,0,
connection.destination.host.subdomain,keyword,1,
connection.direction,keyword,0,inbound
connection.forwarded_ip,ip,0,192.1.1.2
connection.source.host.domain,keyword,1,
connection.source.host.ip,ip,0,
connection.source.host.mac,keyword,1,
connection.source.host.name,keyword,0,
connection.source.host.port,long,1,
connection.source.host.subdomain,keyword,1,
container.id,keyword,0,
container.image.name,keyword,0,
container.image.tag,keyword,0,
container.labels,object,0,
container.name,keyword,0,
container.runtime,keyword,0,docker
destination.domain,keyword,1,
destination.hostname,keyword,0,
destination.ip,ip,0,
destination.mac,keyword,0,
destination.port,long,0,
destination.subdomain,keyword,1,
device.hostname,keyword,0,
device.ip,ip,0,
device.mac,keyword,0,
Expand Down Expand Up @@ -93,8 +101,6 @@ log.level,keyword,0,ERR
log.line,long,0,18
log.message,keyword,1,Sep 19 08:26:10 localhost My log
log.offset,long,0,12
network.direction,keyword,0,inbound
network.forwarded_ip,ip,0,192.1.1.2
network.inbound.bytes,long,0,184
network.inbound.packets,long,0,12
network.outbound.bytes,long,0,184
Expand All @@ -119,12 +125,6 @@ service.name,keyword,1,elasticsearch
service.state,keyword,1,
service.type,keyword,1,
service.version,keyword,1,3.2.4
source.domain,keyword,1,
source.hostname,keyword,0,
source.ip,ip,0,
source.mac,keyword,1,
source.port,long,1,
source.subdomain,keyword,1,
tls.certificates,keyword,0,
tls.ciphersuite,keyword,0,ECDHE-ECDSA-AES-128-CBC-SHA
tls.servername,keyword,0,localhost
Expand Down
83 changes: 83 additions & 0 deletions schemas/connection.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
- name: connection
title: Connection
group: 2
description: >
Connection related fields.
fields:
- name: destination.host.ip
type: ip
description: >
IP address of the destination.
Can be one or multiple IPv4 or IPv6 addresses.
- name: destination.host.name
type: keyword
description: >
Hostname of the destination.
- name: destination.host.port
type: long
description: >
Port of the destination.
- name: destination.host.mac
type: keyword
description: >
MAC address of the destination.
- name: destination.host.domain
type: keyword
phase: 1
description: >
Destination domain.
- name: destination.host.subdomain
type: keyword
phase: 1
description: >
Destination subdomain.
- name: source.host.ip
type: ip
description: >
IP address of the source.
Can be one or multiple IPv4 or IPv6 addresses.
- name: source.host.name
type: keyword
description: >
Hostname of the source.
- name: source.host.port
type: long
phase: 1
description: >
Port of the source.
- name: source.host.mac
type: keyword
phase: 1
description: >
MAC address of the source.
- name: source.host.domain
type: keyword
phase: 1
description: >
Source domain.
- name: source.host.subdomain
type: keyword
phase: 1
description: >
Source subdomain.
- name: direction
type: keyword
description: >
Direction of the network traffic.
Recommended values are:
* inbound
* outbound
* unknown
example: inbound

- name: forwarded_ip
type: ip
description: >
Host IP address when the source IP address is the proxy.
example: 192.1.1.2
36 changes: 0 additions & 36 deletions schemas/destination.yml

This file was deleted.

16 changes: 0 additions & 16 deletions schemas/network.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,6 @@
description: >
Network protocol name.
example: http
- name: direction
type: keyword
description: >
Direction of the network traffic.
Recommended values are:
* inbound
* outbound
* unknown
example: inbound

- name: forwarded_ip
type: ip
description: >
Host IP address when the source IP address is the proxy.
example: 192.1.1.2

# Metrics
- name: inbound.bytes
Expand Down
37 changes: 0 additions & 37 deletions schemas/source.yml

This file was deleted.

Loading

0 comments on commit f590308

Please sign in to comment.