More use cases cleanup (elastic#57)

* More use cases cleanup

* Add better description on how use cases are used
* Complete use cases table with ECS type, example, description if none is provided
* Adjust apm use case to use `@timestamp`
* Clean up generation script to not link `prefix.*` entries
* Refactor fields cleanup part

* update are in italic

README.md

@@ -439,7 +439,11 @@ The user_agent fields normally come from a browser request. They often show up i
 
 # <a name="use-cases"></a>Use cases
 
-ECS fields can be applied to common use cases.
+These are example on how ECS fields can be used in different use cases. Most use
+cases not only contain ECS fields but additional fields which are not in ECS to
+describe the full use case. The fields which are not in ECS are in italic.
+
+Contributions of additional uses cases on top of ECS are welcome.
 
  * [APM](https://github.com/elastic/ecs/blob/master/use-cases/apm.md)
  * [Auditbeat](https://github.com/elastic/ecs/blob/master/use-cases/auditbeat.md)

docs/use-cases-header.md

@@ -1,3 +1,7 @@
 # <a name="use-cases"></a>Use cases
 
-ECS fields can be applied to common use cases.
+These are example on how ECS fields can be used in different use cases. Most use
+cases not only contain ECS fields but additional fields which are not in ECS to
+describe the full use case. The fields which are not in ECS are in italic.
+
+Contributions of additional uses cases on top of ECS are welcome.

scripts/helper.py

@@ -9,7 +9,7 @@ def read_schema_file(path):
     with open(path) as f:
         fields = yaml.load(f.read())
 
-    clean_fields(fields)
+    clean_namespace_fields(fields)
     return fields
 
 
@@ -20,12 +20,12 @@ def read_use_case_file(path):
         use_case = yaml.load(f.read())
 
     fields = use_case["fields"]
-    clean_fields(fields)
+    clean_namespace_fields(fields)
     use_case["fields"] = fields
     return use_case
 
 
-def clean_fields(fields):
+def clean_namespace_fields(fields):
     """Cleans up all fields to set defaults
     """
     for namespace in fields:
@@ -34,39 +34,47 @@ def clean_fields(fields):
         if "group" not in namespace:
             namespace["group"] = 2
 
-        for field in namespace["fields"]:
-            clean_string_field(field, "description")
-            clean_string_field(field, "footnote")
-            clean_string_field(field, "example")
-            clean_string_field(field, "type")
+        prefix = ""
+        # Prefix if not base namespace
+        if namespace["name"] != "base":
+            prefix = namespace["name"]
 
-            # Prefix if not base namespace
-            if namespace["name"] != "base":
-                field["name"] = namespace["name"] + "." + field["name"]
+        clean_fields(namespace["fields"], prefix, namespace["group"])
 
-            if 'phase' not in field.keys():
-                field["phase"] = 0
 
-            if 'group' not in field.keys():
-                # If no group set, set parent group
-                field["group"] = namespace["group"]
+def clean_fields(fields, prefix, group):
+    for field in fields:
+        clean_string_field(field, "description")
+        clean_string_field(field, "footnote")
+        clean_string_field(field, "example")
+        clean_string_field(field, "type")
 
-            if "multi_fields" in field:
-                for f in field["multi_fields"]:
-                    clean_string_field(f, "description")
-                    clean_string_field(f, "example")
-                    clean_string_field(f, "type")
+        # Add prefix if needed
+        if prefix != "":
+            field["name"] = prefix + "." + field["name"]
 
-                    # Prefix if not base namespace
-                    if namespace["name"] != "base":
-                        f["name"] = field["name"] + "." + f["name"]
+        if 'phase' not in field.keys():
+            field["phase"] = 0
 
-                    if 'phase' not in f.keys():
-                        f["phase"] = 0
+        if 'group' not in field.keys():
+            # If no group set, set parent group
+            field["group"] = group
 
-                    if 'group' not in f.keys():
-                        # If no group set, set parent group
-                        f["group"] = namespace["group"]
+        if "multi_fields" in field:
+            for f in field["multi_fields"]:
+                clean_string_field(f, "description")
+                clean_string_field(f, "example")
+                clean_string_field(f, "type")
+
+                # multi fields always have a prefix
+                f["name"] = field["name"] + "." + f["name"]
+
+                if 'phase' not in f.keys():
+                    f["phase"] = 0
+
+                if 'group' not in f.keys():
+                    # If no group set, set parent group
+                    f["group"] = group
 
 
 def clean_string_field(field, key):
@@ -87,9 +95,13 @@ def get_markdown_row(field, link, multi_field):
     description = field["description"].replace("\n", "<br/>")
 
     show_name = field["name"]
-    non_ecs = 'ecs' in field.keys() and not field["ecs"]
-    # non ecs fields are italic
-    if non_ecs:
+
+    ecs = True
+    if 'ecs' in field.keys():
+        ecs = field["ecs"]
+
+    # non ecs fields are in italic
+    if not ecs:
         show_name = "*" + field["name"] + "*"
         description = "*" + description + "*"
 
@@ -104,7 +116,7 @@ def get_markdown_row(field, link, multi_field):
         multi_field = ""
 
     # If link is true, it link to the anchor is provided. This is used for the use-cases
-    if link and not non_ecs:
+    if link and ecs:
         return '| [{}]({}#{})  | {}  | {}  | {}  | {}  |\n'.format(show_name, link, field["name"], description, field["type"], multi_field, example)
 
     # By default a anchor is attached to the name

scripts/use-cases.py

@@ -33,15 +33,26 @@ def write_stdout():
             # In case a description exists for a prefix, add is as field with .*
             if "description" in f and f["description"] != "":
                 fields.append({
-                    "name": f["name"] + ".*",
+                    "name": f["name"] + ".*",
                     "description": f["description"],
                     "type": "",
                     "phase": "",
                     "example": "",
+                    "ecs": False,
                 })
 
             for f2 in f["fields"]:
-                f2["ecs"] = f2["name"] in flat_schema
+                # Complete ECS fields with ECS information if not set
+                if f2["name"] in flat_schema:
+                    f2["ecs"] = True
+                    f2["type"] = flat_schema[f2["name"]]["type"]
+                    if f2["description"] == "":
+                        f2["description"] = flat_schema[f2["name"]]["description"]
+                    if f2["example"] == "":
+                        f2["example"] = flat_schema[f2["name"]]["example"]
+                else:
+                    f2["ecs"] = False
+
                 fields.append(f2)
 
         global_fields = {"name": use_case["name"], "title": use_case["title"], "description": "", "fields": fields}

use-cases/apm.md

@@ -8,11 +8,11 @@ ECS usage for the APM data.
 | Field  | Description  | Type  | Multi Field  | Example  |
 |---|---|---|---|---|
 | <a name="id"></a>*id*  | *Unique id to describe the event.*  | keyword  |   | `8a4f500d`  |
-| <a name="timestamp"></a>*timestamp*  | *Timestamp when the event was created in the app / service.*  | date  |   | `2016-05-23T08:05:34.853Z`  |
-| [agent.*](https://github.com/elastic/ecs#agent.*)  | The agent fields are used to describe which agent did send the information.<br/>  |   |   |   |
+| [@timestamp](https://github.com/elastic/ecs#@timestamp)  | Timestamp when the event was created in the app / service.  | date  |   | `2016-05-23T08:05:34.853Z`  |
+| <a name="agent.&ast;"></a>*agent.&ast;*  | *The agent fields are used to describe which agent did send the information.<br/>*  |   |   |   |
 | [agent.version](https://github.com/elastic/ecs#agent.version)  | APM Agent version.  | keyword  |   | `3.14.0`  |
 | [agent.name](https://github.com/elastic/ecs#agent.name)  | APM agent name.  | keyword  |   | `elastic-node`  |
-| [service.*](https://github.com/elastic/ecs#service.*)  | The service fields describe the service inside which the APM agent is running.<br/>  |   |   |   |
+| <a name="service.&ast;"></a>*service.&ast;*  | *The service fields describe the service inside which the APM agent is running.<br/>*  |   |   |   |
 | [service.id](https://github.com/elastic/ecs#service.id)  | Unique identifier of the running service.  | keyword  |   | `d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6`  |
 | [service.name](https://github.com/elastic/ecs#service.name)  | Name of the service the agent is running in. This is normally a user defined name.  | keyword  |   | `user-service`  |
 | [service.version](https://github.com/elastic/ecs#service.version)  | Version of the service the agent is running in. This depends on if the service is given a version.  | keyword  |   | `3.2.4`  |

use-cases/apm.yml

@@ -10,7 +10,7 @@ fields:
       description: >
         Unique id to describe the event.
       example: 8a4f500d
-    - name: timestamp
+    - name: "@timestamp"
       type: date
       phase: 1
       example: "2016-05-23T08:05:34.853Z"

use-cases/auditbeat.md

@@ -7,11 +7,11 @@ ECS usage in Auditbeat.
 
 | Field  | Description  | Type  | Multi Field  | Example  |
 |---|---|---|---|---|
-| [event.module](https://github.com/elastic/ecs#event.module)  | Auditbeat module name.  |   |   |   |
-| [file.*](https://github.com/elastic/ecs#file.*)  | File attributes.<br/>  |   |   |   |
+| [event.module](https://github.com/elastic/ecs#event.module)  | Auditbeat module name.  | keyword  |   | `mysql`  |
+| <a name="file.&ast;"></a>*file.&ast;*  | *File attributes.<br/>*  |   |   |   |
 | [file.path](https://github.com/elastic/ecs#file.path)  | The path to the file.  | text  |   |   |
 | [file.path.raw](https://github.com/elastic/ecs#file.path.raw)  | The path to the file. This is a non-analyzed field that is useful for aggregations.  | keyword  | 1  |   |
-| [file.target_path](https://github.com/elastic/ecs#file.target_path)  | The target path for symlinks.  | keyword  |   |   |
+| [file.target_path](https://github.com/elastic/ecs#file.target_path)  | The target path for symlinks.  | text  |   |   |
 | [file.type](https://github.com/elastic/ecs#file.type)  | The file type (file, dir, or symlink).  | keyword  |   |   |
 | [file.device](https://github.com/elastic/ecs#file.device)  | The device.  | keyword  |   |   |
 | [file.inode](https://github.com/elastic/ecs#file.inode)  | The inode representing the file in the filesystem.  | keyword  |   |   |
@@ -23,7 +23,7 @@ ECS usage in Auditbeat.
 | [file.size](https://github.com/elastic/ecs#file.size)  | The file size in bytes (field is only added when `type` is `file`).  | long  |   |   |
 | [file.mtime](https://github.com/elastic/ecs#file.mtime)  | The last modified time of the file (time when content was modified).  | date  |   |   |
 | [file.ctime](https://github.com/elastic/ecs#file.ctime)  | The last change time of the file (time when metadata was changed).  | date  |   |   |
-| [hash.*](https://github.com/elastic/ecs#hash.*)  | Hash fields used in Auditbeat.<br/>The hash field contains cryptographic hashes of data associated with the event (such as a file). The keys are names of cryptographic algorithms. The values are encoded as hexidecimal (lower-case).<br/>All fields in user can have one or multiple entries.<br/>  |   |   |   |
+| <a name="hash.&ast;"></a>*hash.&ast;*  | *Hash fields used in Auditbeat.<br/>The hash field contains cryptographic hashes of data associated with the event (such as a file). The keys are names of cryptographic algorithms. The values are encoded as hexidecimal (lower-case).<br/>All fields in user can have one or multiple entries.<br/>*  |   |   |   |
 | <a name="hash.md5"></a>*hash.md5*  | *MD5 hash.*  | keyword  |   |   |
 | <a name="hash.sha1"></a>*hash.sha1*  | *SHA-1 hash.*  | keyword  |   |   |
 | <a name="hash.sha224"></a>*hash.sha224*  | *SHA-224 hash (SHA-2 family).*  | keyword  |   |   |

use-cases/beats.md

@@ -9,7 +9,7 @@ ECS fields used in Beats.
 |---|---|---|---|---|
 | <a name="id"></a>*id*  | *Unique id to describe the event.*  | keyword  |   | `8a4f500d`  |
 | <a name="timestamp"></a>*timestamp*  | *Timestamp when the event was created.*  | date  |   | `2016-05-23T08:05:34.853Z`  |
-| [agent.*](https://github.com/elastic/ecs#agent.*)  | The agent fields are used to describe by which beat the information was collected.<br/>  |   |   |   |
+| <a name="agent.&ast;"></a>*agent.&ast;*  | *The agent fields are used to describe by which beat the information was collected.<br/>*  |   |   |   |
 | [agent.version](https://github.com/elastic/ecs#agent.version)  | Beat version.  | keyword  |   | `6.0.0-rc2`  |
 | [agent.name](https://github.com/elastic/ecs#agent.name)  | Beat name.  | keyword  |   | `filebeat`  |
 | [agent.id](https://github.com/elastic/ecs#agent.id)  | Unique beat identifier.  | keyword  |   | `8a4f500d`  |

use-cases/filebeat-apache-access.md

@@ -9,7 +9,7 @@ ECS fields used in Filebeat for the apache module.
 |---|---|---|---|---|
 | <a name="id"></a>*id*  | *Unique id to describe the event.*  | keyword  |   | `8a4f500d`  |
 | [@timestamp](https://github.com/elastic/ecs#@timestamp)  | Timestamp of the log line after processing.  | date  |   | `2016-05-23T08:05:34.853Z`  |
-| [message](https://github.com/elastic/ecs#message)  | Log message of the event  | date  |   | `Hello World`  |
+| [message](https://github.com/elastic/ecs#message)  | Log message of the event  | text  |   | `Hello World`  |
 | [event.module](https://github.com/elastic/ecs#event.module)  | Currently fileset.module  | keyword  |   | `apache`  |
 | [event.dataset](https://github.com/elastic/ecs#event.dataset)  | Currenly fileset.name  | keyword  |   | `access`  |
 | [source.ip](https://github.com/elastic/ecs#source.ip)  | Source ip of the request. Currently apache.access.remote_ip  | ip  |   | `192.168.1.1`  |
@@ -20,9 +20,9 @@ ECS fields used in Filebeat for the apache module.
 | <a name="http.response.code"></a>*http.response.code*  | *Http response code, currently apache.access.response_code*  | keyword  |   | `404`  |
 | <a name="http.response.body_sent.bytes"></a>*http.response.body_sent.bytes*  | *Http response body bytes sent, currently apache.access.body_sent.bytes*  | long  |   | `117`  |
 | <a name="http.referer"></a>*http.referer*  | *Http referrer code, currently apache.access.referrer<br/>NOTE: In the RFC its misspell as referer and has become accepted standard*  | keyword  |   | `http://elastic.co/`  |
-| [user_agent.*](https://github.com/elastic/ecs#user_agent.*)  | User agent fields as in schema. Currently under apache.access.user_agent.*<br/>  |   |   |   |
+| <a name="user_agent.&ast;"></a>*user_agent.&ast;*  | *User agent fields as in schema. Currently under apache.access.user_agent.*<br/>*  |   |   |   |
 | [user_agent.raw](https://github.com/elastic/ecs#user_agent.raw)  | Raw user agent. Currently apache.access.agent  | text  |   | `http://elastic.co/`  |
-| [geoip.*](https://github.com/elastic/ecs#geoip.*)  | User agent fields as in schema. Currently under apache.access.geoip.*<br/>These are extracted from source.ip<br/>Should they be under source.geoip?<br/>  |   |   |   |
+| <a name="geoip.&ast;"></a>*geoip.&ast;*  | *User agent fields as in schema. Currently under apache.access.geoip.*<br/>These are extracted from source.ip<br/>Should they be under source.geoip?<br/>*  |   |   |   |
 | <a name="geoip...."></a>*geoip....*  | *All geoip fields.*  | text  |   |   |
 
 

use-cases/logging.md

@@ -15,7 +15,7 @@ ECS fields used in logging use cases.
 | [log.level](https://github.com/elastic/ecs#log.level)  | Log level field. Is expected to be `WARN`, `ERR`, `INFO` etc.  | keyword  |   | `ERR`  |
 | [log.line](https://github.com/elastic/ecs#log.line)  | Line number the log event was collected from.  | long  |   | `18`  |
 | [log.offset](https://github.com/elastic/ecs#log.offset)  | Offset of the log event.  | long  |   | `12`  |
-| [source.*](https://github.com/elastic/ecs#source.*)  | Describes from where the log entries come from.<br/>  |   |   |   |
+| <a name="source.&ast;"></a>*source.&ast;*  | *Describes from where the log entries come from.<br/>*  |   |   |   |
 | <a name="source.path"></a>*source.path*  | *File path of the file the data is harvested from.*  | keyword  |   | `/var/log/test.log`  |
 
 

use-cases/metricbeat.md

@@ -12,17 +12,17 @@ ECS fields used Metricbeat.
 | [agent.version](https://github.com/elastic/ecs#agent.version)  | Beat version.  | keyword  |   | `6.0.0-rc2`  |
 | [agent.name](https://github.com/elastic/ecs#agent.name)  | Beat name.  | keyword  |   | `filebeat`  |
 | [agent.id](https://github.com/elastic/ecs#agent.id)  | Unique beat identifier.  | keyword  |   | `8a4f500d`  |
-| [service.*](https://github.com/elastic/ecs#service.*)  | The service fields describe the service for / from which the data was collected.<br/>If logs or metrics are collected from Redis, `service.name` would be `redis`. This allows to find and correlate logs for a specicic service or even version with `service.version`.<br/>  |   |   |   |
+| <a name="service.&ast;"></a>*service.&ast;*  | *The service fields describe the service for / from which the data was collected.<br/>If logs or metrics are collected from Redis, `service.name` would be `redis`. This allows to find and correlate logs for a specicic service or even version with `service.version`.<br/>*  |   |   |   |
 | [service.id](https://github.com/elastic/ecs#service.id)  | Unique identifier of the running service.<br/>This id should uniquely identify this service. This makes it possible to correlate logs and metrics for one specific service. For example in case of issues with one redis instance, it's possible to filter on the id to see metrics and logs for this single instance.  | keyword  |   | `d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6`  |
 | [service.name](https://github.com/elastic/ecs#service.name)  | Name of the service data is collected from.<br/>The name is normally the same as the module name.  | keyword  |   | `elasticsearch`  |
 | [service.version](https://github.com/elastic/ecs#service.version)  | Version of the service the data was collected from.<br/>This allows to look at a data set only for a specific version of a service.  | keyword  |   | `3.2.4`  |
 | <a name="service.host"></a>*service.host*  | *Host address that is used to connect to the service.<br/>This normally contains hostname + port.<br/>REVIEW: Should this be service.uri instead, sometimes it's more then just the host? It could also include a path or the protocol.*  | keyword  |   | `elasticsearch:9200`  |
 | <a name="request.rtt"></a>*request.rtt*  | *Request round trip time.<br/>How long did the request take to fetch metrics from the service.<br/>REVIEW: THIS DOES NOT EXIST YET IN ECS.*  | long  |   | `115`  |
-| [error.*](https://github.com/elastic/ecs#error.*)  | Error namespace<br/>Use for errors which can happen during fetching information for a service.<br/>  |   |   |   |
+| <a name="error.&ast;"></a>*error.&ast;*  | *Error namespace<br/>Use for errors which can happen during fetching information for a service.<br/>*  |   |   |   |
 | [error.message](https://github.com/elastic/ecs#error.message)  | Error message returned by the service during fetching metrics.  | text  |   |   |
-| [error.code](https://github.com/elastic/ecs#error.code)  | Error code returned by the service during fetching metrics.  | long  |   |   |
-| [host.name](https://github.com/elastic/ecs#host.name)  | Hostname of the system metricbeat is running on or user defined name.  | text  |   |   |
-| [host.timezone.offset.sec](https://github.com/elastic/ecs#host.timezone.offset.sec)  | Timezone offset of the host in seconds.  | long  |   |   |
+| [error.code](https://github.com/elastic/ecs#error.code)  | Error code returned by the service during fetching metrics.  | keyword  |   |   |
+| [host.name](https://github.com/elastic/ecs#host.name)  | Hostname of the system metricbeat is running on or user defined name.  | keyword  |   |   |
+| [host.timezone.offset.sec](https://github.com/elastic/ecs#host.timezone.offset.sec)  | Timezone offset of the host in seconds.  | long  |   | `-5400`  |
 | [host.id](https://github.com/elastic/ecs#host.id)  | Unique host id.  | keyword  |   |   |
 | [event.module](https://github.com/elastic/ecs#event.module)  | Name of the module this data is coming from.  | keyword  |   | `mysql`  |
 | [event.dataset](https://github.com/elastic/ecs#event.dataset)  | Name of the dataset.<br/>This contains the information which is currently stored in metricset.name and metricset.module.  | keyword  |   | `stats`  |