diff --git a/CHANGELOG.md b/CHANGELOG.md index dd729b977f..00e0fac6aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,7 @@ All notable changes to this project will be documented in this file based on the * Redefine purpose of `agent.name` field to be user defined field. * Rename `url.href` to `url.original`. * Remove `source.subdomain` and `destination.subdomain` fields. +* Rename `event.version` to `ecs.version`. #169 ### Bugfixes diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c815f3556f..3d1b9416ac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,23 +21,23 @@ Here are the steps for contributing to ECS. - Clone your fork to your machine. * Create a local branch to hold your changes. - Run `git checkout -b branch-name`, where `branch-name` is the name you want to give your local branch -* Do your work. +* Do your work. - Make changes to the `.yml` files as needed -* Run `make` to update generated files. +* Run `make` to update generated files. - Note that the README.md file is generated, and should not be edited directly. Source files are in the /docs directory. -* Commit your changes locally. +* Commit your changes locally. - Run `git commit -a -m "your message"` * Push your changes to your own github.com fork. - - Run `git push --set-upstream origin branch-name` + - Run `git push --set-upstream origin branch-name` - In this command, `origin` is an alias that references your fork. -* Request feedback and permission to merge your changes. - - Create a [Pull Request](https://help.github.com/articles/creating-a-pull-request/) against the ECS repo. +* Request feedback and permission to merge your changes. + - Create a [Pull Request](https://help.github.com/articles/creating-a-pull-request/) against the ECS repo. - (Look for the `Compare & pull request` button on your branch in github.com.) * Next steps - Wait for reviews on your PR. - Incorporate review comments and push updates if needed. * Thank you for your contribution! - + **Important:** Be sure to push changes only to your own fork. Changes must be approved before they are merged into the main repository. ### Other ways to contribute @@ -74,9 +74,9 @@ The file structure is similar to this: Each prefix has its own file. The fields for each prefix are stored in the file. `title` and `description` describe the prefix. `footnote` adds more information following the field table. `level` is for sorting in the documentation output. -In the `fields` section, the `name` of the field is the first entry. -The `type` is the [Elasticsearch field type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). -`description` adds details about the field. +In the `fields` section, the `name` of the field is the first entry. +The `type` is the [Elasticsearch field type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). +`description` adds details about the field. `example` adds an sample value. The `phase` field indicates the development status of the field. If `phase` is left out, the default is 0. @@ -92,8 +92,9 @@ A field's `phase` indicates where it is in its development. Different phases exi ## Guidelines for implementing ECS * The document MUST have the `@timestamp` field. -* The [data type](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/mapping-types.html) defined for an ECS field MUST be used. -* It SHOULD have the field `event.version` to define which version of ECS it uses. +* The [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) + defined for an ECS field MUST be used. +* It SHOULD have the field `ecs.version` to define which version of ECS it uses. * As many fields as possible should be mapped to ECS. ### Writing fields diff --git a/README.md b/README.md index 5ff8891d7b..2f5d7efc16 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,7 @@ ECS defines these fields. * [Container fields](#container) * [Destination fields](#destination) * [Device fields](#device) + * [ECS fields](#ecs) * [Error fields](#error) * [Event fields](#event) * [File fields](#file) @@ -152,6 +153,16 @@ Device fields are used to provide additional information about the device that i | device.type | The type of the device the data is coming from.
There is no predefined list of device types. Some examples are `endpoint`, `firewall`, `ids`, `ips`, `proxy`. | core | keyword | `firewall` | +## ECS fields + +Meta-information specific to ECS. + + +| Field | Description | Level | Type | Example | +|---|---|---|---|---| +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events.
When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events.
The current version is 0.1.0. | core | keyword | `0.1.0` | + + ## Error fields These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. @@ -180,7 +191,6 @@ The event fields are used for context information about the data itself. | event.severity | Severity describes the severity of the event. What the different severity values mean can very different between use cases. It's up to the implementer to make sure severities are consistent across events. | core | long | `7` | | event.original | Raw text message of entire event. Used to demonstrate log integrity.
This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. | core | keyword | `Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232` | | event.hash | Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. | extended | keyword | `123456789012345678901234567890ABCD` | -| event.version | The version field contains the version an event for ECS adheres to.
This field should be provided as part of each event to make it possible to detect to which ECS version an event belongs.
event.version is a required field and must exist in all events. It describes which ECS version the event adheres to.
The current version is 0.1.0. | core | keyword | `0.1.0` | | event.duration | Duration of the event in nanoseconds. | core | long | | | event.created | event.created contains the date when the event was created.
This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.
In case the two timestamps are identical, @timestamp should be used. | core | date | | | event.risk_score | Risk score or priority of the event (e.g. security solutions). Use your system's original value here. | core | float | | @@ -451,8 +461,9 @@ Contributions of additional uses cases on top of ECS are welcome. ## Guidelines * The document MUST have the `@timestamp` field. -* The [data type](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/mapping-types.html) defined for an ECS field MUST be used. -* It SHOULD have the field `event.version` to define which version of ECS it uses. +* The [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) + defined for an ECS field MUST be used. +* It SHOULD have the field `ecs.version` to define which version of ECS it uses. * As many fields as possible should be mapped to ECS. **Writing fields** diff --git a/docs/implementing.md b/docs/implementing.md index 7e96be4489..b797666b04 100644 --- a/docs/implementing.md +++ b/docs/implementing.md @@ -3,8 +3,9 @@ ## Guidelines * The document MUST have the `@timestamp` field. -* The [data type](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/mapping-types.html) defined for an ECS field MUST be used. -* It SHOULD have the field `event.version` to define which version of ECS it uses. +* The [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) + defined for an ECS field MUST be used. +* It SHOULD have the field `ecs.version` to define which version of ECS it uses. * As many fields as possible should be mapped to ECS. **Writing fields** diff --git a/fields.yml b/fields.yml index 5d843c3076..66b819de9e 100644 --- a/fields.yml +++ b/fields.yml @@ -324,6 +324,28 @@ `endpoint`, `firewall`, `ids`, `ips`, `proxy`. example: firewall + - name: ecs + title: ECS + group: 2 + description: > + Meta-information specific to ECS. + type: group + fields: + + - name: version + level: core + type: keyword + required: true + description: > + ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. + + When querying across multiple indices -- which may conform to slightly + different ECS versions -- this field lets integrations adjust to the + schema version of the events. + + The current version is 0.1.0. + example: 0.1.0 + - name: error title: Error group: 2 @@ -452,22 +474,6 @@ Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. - - name: version - level: core - type: keyword - required: true - description: > - The version field contains the version an event for ECS adheres to. - - This field should be provided as part of each event to make it possible - to detect to which ECS version an event belongs. - - event.version is a required field and must exist in all events. It - describes which ECS version the event adheres to. - - The current version is 0.1.0. - example: 0.1.0 - - name: duration level: core type: long diff --git a/schema.csv b/schema.csv index 67ebd2f2ba..94ecf4d00e 100644 --- a/schema.csv +++ b/schema.csv @@ -33,6 +33,7 @@ device.serial_number,keyword,extended, device.type,keyword,core,firewall device.vendor,keyword,core, device.version,keyword,core, +ecs.version,keyword,core,0.1.0 error.code,keyword,core, error.id,keyword,core, error.message,text,core, @@ -49,7 +50,6 @@ event.risk_score,float,core, event.risk_score_norm,float,extended, event.severity,long,core,7 event.type,keyword,core,nginx-stats-metrics -event.version,keyword,core,0.1.0 file.ctime,date,extended, file.device,keyword,extended, file.extension,keyword,extended,png diff --git a/schemas/ecs.yml b/schemas/ecs.yml new file mode 100644 index 0000000000..17bf192e26 --- /dev/null +++ b/schemas/ecs.yml @@ -0,0 +1,22 @@ +--- +- name: ecs + title: ECS + group: 2 + description: > + Meta-information specific to ECS. + type: group + fields: + + - name: version + level: core + type: keyword + required: true + description: > + ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. + + When querying across multiple indices -- which may conform to slightly + different ECS versions -- this field lets integrations adjust to the + schema version of the events. + + The current version is 0.1.0. + example: 0.1.0 diff --git a/schemas/event.yml b/schemas/event.yml index 4d22a93306..ac8f725dc8 100644 --- a/schemas/event.yml +++ b/schemas/event.yml @@ -100,22 +100,6 @@ Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. - - name: version - level: core - type: keyword - required: true - description: > - The version field contains the version an event for ECS adheres to. - - This field should be provided as part of each event to make it possible - to detect to which ECS version an event belongs. - - event.version is a required field and must exist in all events. It - describes which ECS version the event adheres to. - - The current version is 0.1.0. - example: 0.1.0 - - name: duration level: core type: long diff --git a/template.json b/template.json index b381f130b4..d37b71485e 100644 --- a/template.json +++ b/template.json @@ -175,6 +175,14 @@ } } }, + "ecs": { + "properties": { + "version": { + "ignore_above": 1024, + "type": "keyword" + } + } + }, "error": { "properties": { "code": { @@ -241,10 +249,6 @@ "type": { "ignore_above": 1024, "type": "keyword" - }, - "version": { - "ignore_above": 1024, - "type": "keyword" } } },