Include structured event data with each event.

The benefit to this change is that Winlogbeat now provides the data from the event log message in a structured format so that most message no longer require groking to extract data. Changes - Add additional data to the events published by Winlogbeat. The new fields are activity_id, event_data, keywords, opcode, process_id, provider_guid, related_activity_id, task, thread_id, user_data, and version. - The message_inserts field was replaced with the event_data field - The category field was renamed to task to better align with the Windows Event Log API naming Closes #1053
elastic · Mar 15, 2016 · 074fa5d · 074fa5d
1 parent 10d77bf
commit 074fa5d
Show file tree

Hide file tree

Showing 26 changed files with 989 additions and 718 deletions.
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -31,6 +31,8 @@ https://github.com/elastic/beats/compare/v1.1.0...master[Check the HEAD diff]
 - Scalar values in used in the `fields` configuration setting are no longer automatically converted to strings. {pull}1092[1092]
 
 *Winlogbeat*
+- The `message_inserts` field was replaced with the `event_data` field {issue}1053[1053]
+- The `category` field was renamed to `task` to better align with the Windows Event Log API naming {issue}1053[1053]
 
 
 ==== Bugfixes
@@ -96,6 +98,9 @@ https://github.com/elastic/beats/compare/v1.1.0...master[Check the HEAD diff]
 - Add caching of event metadata handles and the system render context for the wineventlog API {pull}888[888]
 - Improve config validation by checking for unknown top-level YAML keys. {pull}1100[1100]
 - Add the ability to set tags, fields, and fields_under_root as options for each event log {pull}1092[1092]
+- Add additional data to the events published by Winlogbeat. The new fields are `activity_id`,
+`event_data`, `keywords`, `opcode`, `process_id`, `provider_guid`, `related_activity_id`,
+`task`, `thread_id`, `user_data`. and `version`. {issue}1053[1053]
 
 ==== Deprecated
 

diff --git a/winlogbeat/beater/winlogbeat.go b/winlogbeat/beater/winlogbeat.go
@@ -250,9 +250,9 @@ loop:
 			// of event if it is going to be filtered.
 			// TODO: Add a severity filter.
 			// TODO: Check the global IgnoreOlder filter.
-			if ignoreOlder != 0 && time.Since(lr.TimeGenerated) > ignoreOlder {
-				detailf("EventLog[%s] ignore_older filter dropping event: %s",
-					api.Name(), lr.String())
+			if ignoreOlder != 0 && time.Since(lr.TimeCreated.SystemTime) > ignoreOlder {
+				detailf("EventLog[%s] ignore_older filter dropping event: %+v",
+					api.Name(), lr)
 				ignoredEvents.Add("total", 1)
 				ignoredEvents.Add(api.Name(), 1)
 				continue
@@ -276,8 +276,8 @@ loop:
 		}
 
 		eb.checkpoint.Persist(api.Name(),
-			records[len(records)-1].RecordNumber,
-			records[len(records)-1].TimeGenerated.UTC())
+			records[len(records)-1].RecordID,
+			records[len(records)-1].TimeCreated.SystemTime.UTC())
 	}
 }
 

diff --git a/winlogbeat/config/config.go b/winlogbeat/config/config.go
@@ -41,7 +41,7 @@ func (s Settings) Validate() error {
 
 	// Check for invalid top-level keys.
 	var errs multierror.Errors
-	for k, _ := range s.All {
+	for k := range s.All {
 		k = strings.ToLower(k)
 		i := sort.SearchStrings(validKeys, k)
 		if i >= len(validKeys) || validKeys[i] != k {

diff --git a/winlogbeat/docs/fields.asciidoc b/winlogbeat/docs/fields.asciidoc
@@ -66,6 +66,15 @@ Contains data from a Windows event log record.
 
 
 
+==== activity_id
+
+type: string
+
+required: False
+
+A globally unique identifier that identifies the current activity. The events that are published with this identifier are part of the same activity.
+
+
 ==== computer_name
 
 type: string
@@ -75,13 +84,13 @@ required: True
 The name of the computer that generated the record. When using Windows event forwarding, this name can differ from the `beat.hostname`.
 
 
-==== category
+==== event_data
 
-type: string
+type: dictionary
 
 required: False
 
-The category for this event. The meaning of this value depends on the source of the event.
+The event specific data. This field is mutually exclusive with `user_data`.
 
 
 ==== event_id
@@ -93,6 +102,15 @@ required: True
 The event identifier. The value is specific to the source of the event.
 
 
+==== keywords
+
+type: string[]
+
+required: False
+
+The keywords are used to classify an event.
+
+
 ==== log_name
 
 type: string
@@ -106,7 +124,7 @@ The name of the event log from which this record was read. This value is one of
 
 type: string
 
-required: True
+required: False
 
 The level of the event. There are five levels of events that can be logged: Success, Information, Warning, Error, Audit Success, and Audit Failure.
 
@@ -126,25 +144,52 @@ type: string
 
 required: False
 
-The error that occurred while reading and formatting the message from the log. This field is mutually exclusive with `message`.
+The error that occurred while reading and formatting the message from the log.
 
 
-==== message_inserts
+==== record_number
 
-type: list
+type: string
+
+required: True
+
+The record number of the event log record. The first record written to an event log is record number 1, and other records are numbered sequentially. If the record number reaches the maximum value (2^32^ for the Event Logging API and 2^64^ for the Windows Event Log API), the next record number will be 0.
+
+
+==== related_activity_id
+
+type: string
 
 required: False
 
-The raw message data logged by an application. Normally this data is inserted into a parameterized string to create `message`, but in case of an error, Winlogbeat attempts to provide this raw data. This field is mutually exclusive with `message`.
+A globally unique identifier that identifies the activity to which control was transferred to. The related events would then have this identifier as their `activity_id` identifier.
 
 
-==== record_number
+==== opcode
 
 type: string
 
-required: True
+required: False
 
-The record number of the event log record. The first record written to an event log is record number 1, and other records are numbered sequentially. If the record number reaches the maximum value (2^32^ for the Event Logging API and 2^64^ for the Windows Event Log API), the next record number will be 0.
+The opcode defined in the event. Task and opcode are typically used to identify the location in the application from where the event was logged.
+
+
+==== provider_guid
+
+type: string
+
+required: False
+
+A globally unique identifier that identifies the provider that logged the event.
+
+
+==== process_id
+
+type: long
+
+required: False
+
+The process_id identifies the process that generated the event.
 
 
 ==== source_name
@@ -156,6 +201,33 @@ required: True
 The source of the event log record (the application or service that logged the record).
 
 
+==== task
+
+type: string
+
+required: False
+
+The task defined in the event. Task and opcode are typically used to identify the location in the application from where the event was logged. The category used by the Event Logging API (on pre Windows Vista operating systems) is written to this field.
+
+
+==== thread_id
+
+type: long
+
+required: False
+
+The thread_id identifies the thread that generated the event.
+
+
+==== user_data
+
+type: dictionary
+
+required: False
+
+The event specific data. This field is mutually exclusive with `event_data`.
+
+
 ==== user.identifier
 
 type: string
@@ -196,3 +268,11 @@ required: False
 The type of account associated with this event.
 
 
+==== version
+
+type: int
+
+required: False
+
+The version number of the event's definition.
+
diff --git a/winlogbeat/etc/fields.yml b/winlogbeat/etc/fields.yml
@@ -56,39 +56,54 @@ eventlog:
   description: >
     Contains data from a Windows event log record.
   fields:
+    - name: activity_id
+      type: string
+      required: false
+      description: >
+        A globally unique identifier that identifies the current activity. The
+        events that are published with this identifier are part of the same
+        activity.
+
     - name: computer_name
       type: string
       required: true
       description: >
         The name of the computer that generated the record. When using Windows
         event forwarding, this name can differ from the `beat.hostname`.
 
-    - name: category
-      type: string
+    - name: event_data
+      type: dictionary
       required: false
       description: >
-        The category for this event. The meaning of this value depends on the
-        source of the event.
+        The event specific data. This field is mutually exclusive with
+        `user_data`.
 
     - name: event_id
       type: long
       required: true
       description: >
         The event identifier. The value is specific to the source of the event.
 
+    - name: keywords
+      type: string[]
+      required: false
+      description: >
+        The keywords are used to classify an event.
+
     - name: log_name
       type: string
       required: true
       description: >
-        The name of the event log from which this record was read. This value is one
-        of the names from the `event_logs` collection in the configuration.
+        The name of the event log from which this record was read. This value is
+        one of the names from the `event_logs` collection in the configuration.
 
     - name: level
       type: string
-      required: true
+      required: false
       description: >
-        The level of the event. There are five levels of events that can be logged:
-        Success, Information, Warning, Error, Audit Success, and Audit Failure.
+        The level of the event. There are five levels of events that can be
+        logged: Success, Information, Warning, Error, Audit Success, and Audit
+        Failure.
 
     - name: message
       type: string
@@ -104,16 +119,7 @@ eventlog:
       required: false
       description: >
         The error that occurred while reading and formatting the message from
-        the log. This field is mutually exclusive with `message`.
-
-    - name: message_inserts
-      type: list
-      required: false
-      description: >
-        The raw message data logged by an application. Normally this data is
-        inserted into a parameterized string to create `message`, but in case of
-        an error, Winlogbeat attempts to provide this raw data. This field is
-        mutually exclusive with `message`.
+        the log.
 
     - name: record_number
       type: string
@@ -125,13 +131,64 @@ eventlog:
         for the Event Logging API and 2^64^ for the Windows Event Log API),
         the next record number will be 0.
 
+    - name: related_activity_id
+      type: string
+      required: false
+      description: >
+        A globally unique identifier that identifies the activity to which
+        control was transferred to. The related events would then have this
+        identifier as their `activity_id` identifier.
+
+    - name: opcode
+      type: string
+      required: false
+      description: >
+        The opcode defined in the event. Task and opcode are typically used to
+        identify the location in the application from where the event was
+        logged.
+
+    - name: provider_guid
+      type: string
+      required: false
+      description: >
+        A globally unique identifier that identifies the provider that logged
+        the event.
+
+    - name: process_id
+      type: long
+      required: false
+      description: >
+        The process_id identifies the process that generated the event.
+
     - name: source_name
       type: string
       required: true
       description: >
         The source of the event log record (the application or service that
         logged the record).
 
+    - name: task
+      type: string
+      required: false
+      description: >
+        The task defined in the event. Task and opcode are typically used to
+        identify the location in the application from where the event was
+        logged. The category used by the Event Logging API (on pre Windows Vista
+        operating systems) is written to this field.
+
+    - name: thread_id
+      type: long
+      required: false
+      description: >
+        The thread_id identifies the thread that generated the event.
+
+    - name: user_data
+      type: dictionary
+      required: false
+      description: >
+        The event specific data. This field is mutually exclusive with
+        `event_data`.
+
     - name: user.identifier
       type: string
       required: false
@@ -164,6 +221,11 @@ eventlog:
       description: >
         The type of account associated with this event.
 
+    - name: version
+      type: int
+      required: false
+      description: The version number of the event's definition.
+
 sections:
   - ["common", "Common Beat"]
   - ["eventlog", "Event Log Record"]
diff --git a/winlogbeat/etc/winlogbeat.template.json b/winlogbeat/etc/winlogbeat.template.json
@@ -34,6 +34,14 @@
             "enabled": false
           },
           "type": "string"
+        },
+        "process_id": {
+          "doc_values": "true",
+          "type": "long"
+        },
+        "thread_id": {
+          "doc_values": "true",
+          "type": "long"
         }
       }
     }