Filebeat: osquery module #5971
Filebeat: osquery module #5971
Conversation
tsg
added
Filebeat
| - name: calendar_time | ||
| tupe: keyword | ||
| description: > | ||
| String representation of the collection time, as formatted by osquery. |
There was a problem hiding this comment.
Interesting that this is called calendar_time. I like the description with collection time. I wonder if we should have a common field in the future for the collection_time (or a different name).
There was a problem hiding this comment.
In osquery, they have unixTime and calendarTime which are different representations of the same moment. These get translated to unix_time and calendar_time by the module, and @timestamp is computed from unix_time. So there's quite a bit of redundancy, we could consider dropping the calendar_time by default.
There was a problem hiding this comment.
Ok, my assumption was that the calendar_time is the time when the entry was actually read and unix_time when it was created. Like for a log line where we have @timestamp potentially from the log line and beat.read_time. Perhaps beat.read_time should be event.collection_timestamp?
There was a problem hiding this comment.
I added read_time (btw, it's not prefixed by beat) as being the read time when filebeat reads the log lines.
| - name: unix_time | ||
| type: long | ||
| description: > | ||
| Unix timestamp of the event. Used for computing the `@timestamp` column. |
There was a problem hiding this comment.
Is this in seconds since epoch?
There was a problem hiding this comment.
Yes, I'll update the docs for clarity.
|
Added another dashboard, addressed comments, and rebased to master. If green, this is good for merging from my POV. |
This adds a Filebeat module for centralising osquery logs. osqueryd writes the results in JSON, which makes it fairly easy to ingest to the Elastic stack. The module uses the JSON decoding support in Filebeat, and then renames the fields to match the Beats naming conventions (most fields prefixed with osquery.result). There is an option (use_namespace: false) to leave the fields as they were in the original JSON, but changing that settings makes the sample dashboards unusable. This module comes with two dashboards, one for the compliance pack in osquery, the other for the ossec-rootkit pack.
| @@ -45,7 +45,7 @@ | |||
| { | |||
| "rename": { | |||
| "field": "@timestamp", | |||
| "target_field": "beat.read_time" | |||
| "target_field": "read_timestamp" | |||
There was a problem hiding this comment.
@tsg This seems to be a breaking change? Also saw there is no CHANGELOG for this PR.
|
Hi, all! @tsg ^^^ |
|
@etursunbaev your filebeat version seems old. To use osquery module in older version, you can reference this: https://github.com/jjqq2013/misc/tree/master/elasticsearch5.4.0 |
|
Hi, all, do you know what will be the document type of osquery module? What is the value when use osquery module? |
|
Oh, no problem. I found it: I need define fields. log_topic in prospector. |
This adds a Filebeat module for centralising osquery logs.
osqueryd writes the results in JSON, which makes it fairly easy to ingest to the Elastic stack. The module uses the JSON decoding support in Filebeat, and then renames the fields to match the Beats naming conventions (most fields prefixed with
osquery.result). There is an option (use_namespace: false) to leave the fields as they were in the original JSON, but changing that settings makes the sample dashboards unusable.Another issue is that the osquery JSON format represents all data as strings (the numbers are quoted). This is both bad and good. It's bad because number aggregations don't work on what should be numbers, but it's good because it means there can't be any type conflicts. We could, potentially, do the type conversions ourselves based on the osquery schema, but that seems risky with regards to the schema changes. The osquery JSON output should also be fixed once osquery switches to RapidJSON, which seems to be in progress.