Filebeat: osquery module (#5971)

* Filebeat Osquery module

This adds a Filebeat module for centralising osquery logs.

osqueryd writes the results in JSON, which makes it fairly easy to ingest to the Elastic stack. The module uses the JSON decoding support in Filebeat, and then renames the fields to match the Beats naming conventions (most fields prefixed with osquery.result). There is an option (use_namespace: false) to leave the fields as they were in the original JSON, but changing that settings makes the sample dashboards unusable.

This module comes with two dashboards, one for the compliance pack in osquery, the other for the ossec-rootkit pack.

* Set read_timestamp

* Clarified the unix_timestamp column

* make update

* Rename Kafka module `beat.read_time` to `read_timestamp`, which is documented

filebeat/docs/fields.asciidoc

@@ -24,6 +24,7 @@ grouped in the following categories:
 * <<exported-fields-logstash>>
 * <<exported-fields-mysql>>
 * <<exported-fields-nginx>>
+* <<exported-fields-osquery>>
 * <<exported-fields-postgresql>>
 * <<exported-fields-redis>>
 * <<exported-fields-system>>
@@ -1535,6 +1536,64 @@ type: text
 The error message
 
 
+[[exported-fields-osquery]]
+== Osquery fields
+
+Fields exported by the `osquery` module
+
+
+
+[float]
+== osquery fields
+
+
+
+
+[float]
+== result fields
+
+Common fields exported by the result metricset.
+
+
+
+[float]
+=== `osquery.result.name`
+
+type: keyword
+
+The name of the query that generated this event.
+
+
+[float]
+=== `osquery.result.action`
+
+type: keyword
+
+For incremental data, marks whether the entry was added or removed. It can be one of "added", "removed", or "snapshot".
+
+
+[float]
+=== `osquery.result.host_identifier`
+
+type: keyword
+
+The identifier for the host on which the osquery agent is running. Normally the hostname.
+
+
+[float]
+=== `osquery.result.unix_time`
+
+type: long
+
+Unix timestamp of the event, in seconds since the epoch. Used for computing the `@timestamp` column.
+
+
+[float]
+=== `osquery.result.calendar_time`
+
+String representation of the collection time, as formatted by osquery.
+
+
 [[exported-fields-postgresql]]
 == PostgreSQL fields
 

filebeat/docs/modules/osquery.asciidoc

@@ -0,0 +1,80 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-osquery]]
+:modulename: osquery
+
+== Osquery module
+
+The +{modulename}+ module collects and decodes the result logs written by
+https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd]
+in the JSON format. To set up osqueryd follow the osquery installation
+instructions for your operating system and configure the `filesystem` logging
+driver (the default). Make sure UTC timestamps are enabled.
+
+include::../include/what-happens.asciidoc[]
+
+
+[float]
+=== Compatibility
+
+The  +{modulename}+ module was tested with logs from osquery version 2.10.2.
+Since the results are written in the JSON format, it is likely that this module
+works with any version of osquery.
+
+This module is available on Linux, macOS, and Windows.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard for visualizing the data collected by
+the "compliance" pack. To collect this data, enable the `id-compliance` pack in
+the osquery configuration file.
+
+[role="screenshot"]
+image::./images/kibana-osquery-compatibility.png[]
+
+include::../include/configuring-intro.asciidoc[]
+
+The following example shows how to set paths in the +modules.d/{modulename}.yml+
+file to override the default paths for the syslog and authorization logs:
+
+["source","yaml",subs="attributes"]
+-----
+- module: osquery
+  result:
+    enabled: true
+    var.paths: ["/path/to/osqueryd.results.log*"]
+-----
+
+
+To specify the same settings at the command line, you use:
+
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} --modules {modulename} -M "osquery.result.var.paths=[/path/to/osqueryd.results.log*]"
+-----
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `result` fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+*`var.use_namespace`*::
+
+If true, all fields exported by this module are prefixed with `osquery.result`.
+Set to false to copy the fields in the root of the document. If enabled, this
+setting also disables the renaming of some fields (e.g. `hostIdentifier` to
+`host_identifier`).  Note that if you set this to false, the sample dashboards
+coming with this module won't work correctly. The default is true.
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-osquery,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -10,6 +10,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-logstash>>
   * <<filebeat-module-mysql>>
   * <<filebeat-module-nginx>>
+  * <<filebeat-module-osquery>>
   * <<filebeat-module-postgresql>>
   * <<filebeat-module-redis>>
   * <<filebeat-module-system>>
@@ -26,6 +27,7 @@ include::modules/kafka.asciidoc[]
 include::modules/logstash.asciidoc[]
 include::modules/mysql.asciidoc[]
 include::modules/nginx.asciidoc[]
+include::modules/osquery.asciidoc[]
 include::modules/postgresql.asciidoc[]
 include::modules/redis.asciidoc[]
 include::modules/system.asciidoc[]

filebeat/filebeat.reference.yml

@@ -204,6 +204,20 @@ filebeat.modules:
     # can be added under this section.
     #prospector:
 
+#------------------------------- Osquery Module ------------------------------
+- module: osquery
+  result:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # If true, all fields created by this module are prefixed with
+    # `osquery.result`. Set to false to copy the fields in the root
+    # of the document. The default is true.
+    #var.use_namespace: true
+
 #----------------------------- PostgreSQL Module -----------------------------
 #- module: postgresql
   # Logs

filebeat/module/kafka/log/ingest/pipeline.json

@@ -45,7 +45,7 @@
     {
       "rename": {
         "field": "@timestamp",
-        "target_field": "beat.read_time"
+        "target_field": "read_timestamp"
       }
     },
     {

filebeat/module/osquery/_meta/config.reference.yml

@@ -0,0 +1,12 @@
+- module: osquery
+  result:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # If true, all fields created by this module are prefixed with
+    # `osquery.result`. Set to false to copy the fields in the root
+    # of the document. The default is true.
+    #var.use_namespace: true

filebeat/module/osquery/_meta/config.yml

@@ -0,0 +1,12 @@
+- module: osquery
+  result:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # If true, all fields created by this module are prefixed with
+    # `osquery.result`. Set to false to copy the fields in the root
+    # of the document. The default is true.
+    #var.use_namespace: true

filebeat/module/osquery/_meta/docs.asciidoc

@@ -0,0 +1,67 @@
+:modulename: osquery
+
+== Osquery module
+
+The +{modulename}+ module collects and decodes the result logs written by
+https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd]
+in the JSON format. To set up osqueryd follow the osquery installation
+instructions for your operating system and configure the `filesystem` logging
+driver (the default). Make sure UTC timestamps are enabled.
+
+include::../include/what-happens.asciidoc[]
+
+
+[float]
+=== Compatibility
+
+The  +{modulename}+ module was tested with logs from osquery version 2.10.2.
+Since the results are written in the JSON format, it is likely that this module
+works with any version of osquery.
+
+This module is available on Linux, macOS, and Windows.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard for visualizing the data collected by
+the "compliance" pack. To collect this data, enable the `id-compliance` pack in
+the osquery configuration file.
+
+[role="screenshot"]
+image::./images/kibana-osquery-compatibility.png[]
+
+include::../include/configuring-intro.asciidoc[]
+
+The following example shows how to set paths in the +modules.d/{modulename}.yml+
+file to override the default paths for the syslog and authorization logs:
+
+["source","yaml",subs="attributes"]
+-----
+- module: osquery
+  result:
+    enabled: true
+    var.paths: ["/path/to/osqueryd.results.log*"]
+-----
+
+
+To specify the same settings at the command line, you use:
+
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} --modules {modulename} -M "osquery.result.var.paths=[/path/to/osqueryd.results.log*]"
+-----
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `result` fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+*`var.use_namespace`*::
+
+If true, all fields exported by this module are prefixed with `osquery.result`.
+Set to false to copy the fields in the root of the document. If enabled, this
+setting also disables the renaming of some fields (e.g. `hostIdentifier` to
+`host_identifier`).  Note that if you set this to false, the sample dashboards
+coming with this module won't work correctly. The default is true.

filebeat/module/osquery/_meta/fields.yml

@@ -0,0 +1,9 @@
+- key: osquery
+  title: "Osquery"
+  description: >
+    Fields exported by the `osquery` module
+  fields:
+    - name: osquery
+      type: group
+      description: >
+      fields: