[Winlogbeat] Features / Status / Roadmap

[tbragin]

See background here: https://github.com/elastic/filebeat/issues/130
And current status here: https://github.com/elastic/winlogbeat/issues/1

[andrewkroh]

Winlogbeat Status Tracking

Updated: 2019-12-20

This issue tracks the overall status of the effort. We can open separate issues to discuss individual features in depth.

Features

• Add ability to read from Windows Event Log API (preferred API to get event log
  data in Windows Vista, Windows Server 2008, and newer). PR: Add Windows Event Log API support #576
• Read from Event Logging API (provides Windows XP and Windows 2003 compatibility and
  works on all newer versions, but some events cannot be read through this API).
• Event timestamp filtering (ignore_older)
• Event ID filtering [Winlogbeat] XML filtering #1054 (implemented in Winlogbeat - Select events by level, event_id, and provider #1218)
• Level (severity) filtering [Winlogbeat] XML filtering #1054 (implemented in Winlogbeat - Select events by level, event_id, and provider #1218)
• Provider (source) filtering (implemented in Winlogbeat - Select events by level, event_id, and provider #1218)
• Add EventData and UserData from the Windows event log records to the data published by Winlogbeat. Winlogbeat structured event data #1053
• Add ability to do more advanced queries including using raw XML. [Winlogbeat] XML filtering #1054
• Read registry to automatically detect all event logs. The user would not
  need to configure which event logs to read.
• Wildcard channel names. [Winlogbeat] Wildcard support for Channel Names #1193
• Add ability to configure UNC path for reading from remote event logs. (we prefer Windows Event Collector/Event Subscriptions but if those cannot be applied then this can be the fall-back.) Winlogbeat - Test Usage on Windows Event Collector #1031
• Reading .evt and .evtx files. Useful for ingesting archived event logs or for forensic purposes. References - .evt format: [1] (https://github.com/libyal/libevt/blob/master/documentation/Windows%20Event%20Log%20%28EVT%29%20format.asciidoc), .evtx format: 1, 2 [Winlogbeat] Add option to read .evt and .evtx files #4450
• Message regex filtering (likely unnecessary after adding other filters)
• Add additional read modes.
  • persistent - the default mode that read events and persist state (the current behavior)
  • new_only - read all events generated since the beat was started. No state
    is persisted. This is like tail -f.
  • read_then_exit - read all events that exist at startup, then exit.
    It ignores new events that are generated after the beat was started.
    No state is persisted.

Performance

• Share a single cache for message file handles.
• Add SID to name/domain/type caching.
• Asynchronous event publishing like Filebeat.
• Share a set of buffers between all event logs so that memory usage does not
  grow linearly with the number of logs being monitored.
• Optimize the slow methods (I suspect they are getStrings and
  utf16ToString).

Type Safety

• Investigate removing unsafe from eventlog_windows.go by parsing the buffer.

Test cases

• Add test case for messages with UTF-16 characters (i.e. Chinese characters). (update: seems like this is working, see http://dev.classmethod.jp/server-side/elasticsearch/winlogbeat-release/)

[elvarb]

Is it necessary to do regex filtering on the event? I'm pretty sure you can get all event field values as objects, fields you don't see in the event viewer but are special objects still.

[andrewkroh]

@elvarb I am not proposing regex filtering the event as a whole. I am proposing regex filtering based on the text of the event's message field.

[elvarb]

That's what I'm pointing out, you don't have to. For example if a user1 adds user2 to group1 the message field will say that. But also in the event are special fields like the eventid that are called something along the lines of param1, param2 and so on. The message field is an attempt to make the event human readable, and I truly mean an attempt.

[elvarb]

https://www.microsoft.com/technet/support/ee/transform.aspx?ProdName=Windows+Operating+System&ProdVer=5.2&EvtID=1074&EvtSrc=User32&LCID=1033

Here is an example that is showing what I mean, see the variables named %1,%2 and so on.

That url is a great resource, change the url parameters to get different events

[andrewkroh]

@elvarb Are you saying that regex based filtering is not required if Winlogbeat supports Event ID based filtering since each message is really just derived from a parameterized message template that is given by Event ID + Event Source?

[elvarb]

@andrewkroh Exactly, or I at least hope so. The reference to those fields are all over in the Windows documentation and I know that nxlog can read those fields.

Here is another example https://msdn.microsoft.com/en-us/library/aa368560(v=vs.85).aspx

And how to write event logs using parameters https://msdn.microsoft.com/en-us/library/windows/hardware/ff566411(v=vs.85).aspx

And code examples how to get those insertation strings
https://msdn.microsoft.com/en-us/library/windows/desktop/bb427356(v=vs.85).aspx

I have created regex for various windows event logs coming through OSSEC, which is always full text, not fun and prone to regex errors because of unexpected reasons. Try to avoid regex as much as you can.

In my logstash config I am also using the translate filter for various Windows events to translate from specific codes to human readable text. For example logon type:

"2": Interactive
"3": Network
"4": Batch
"5": Service
"7": Unlock
"8": NetworkClearText
"9": NewCredentials
"10": RemoteInteractive
"11": CachedInteractive

Would be fantastic if beats could do this to on the host level instead of having Logstash do this for well known native codes. (for some cases it is still better to do it in Logstash, custom filters and so on)

[davidhowell-tx]

You've probably already figured this out, but thought I would add some additional context to elvarb's comments from my non-programmer point of view. You can see the individual fields and values within the event log object in a few methods:

In Event Viewer, the XML View has an EventData node with the individual keys and values, and the message is just a more descriptive representation of the EventData values.

In PowerShell the individual values are accessible in the "Properties" Property. You can see this when you run the following:
$Z = Get-WinEvent -LogName Security -MaxEvents 1
$Z.Message
$Z.Properties

or with:
$Z = Get-EventLog -LogName Security -Newest 1
$Z.Message
$Z.ReplacementStrings

I hope that is helpful, I would really love to see this improvement so I can reduce my logstash parsing a bit.

[andrewkroh]

The above comment is related to the feature requested in #1053.

[mtmcgrew]

Reading in .evtx sounds amazing. I'm sure lots of people in the security community would love that!

[tomrade]

Currently we have an internal python based solution for offline files using python-evtx , which uses an Event ID yaml library but for beats these may be useful. Its worth nothing that your YAML might need to nested as different event channels will reuse eventids eg security , system etc

evtx go library
https://github.com/0xrawsec/golang-evtx

evtx event id database tool with example output
https://github.com/iadgov/Windows-Event-Log-Messages

[andrewkroh]

Here's and MIT licensed evtx parser I found that we could try: https://github.com/dutchcoders/evtxparser

The https://github.com/0xrawsec/golang-evtx project is GPL so it's incompatible with Apache 2.0.

[PChou]

I think reading .evt and .evtx files can be done with Microsoft native api(EvtQuery, EvtSeek,EvtNext ,EvtClose),
I am working on it, but since we need support glob path(like C:\Windows\System32\winevt\Logs*) and other features, it seems to be a filebeat type instead of winlogbeat module.
checkout:
PChou@d9e9523

[m4rcu5]

@PChou I've compiled your branch and just imported 47 archived evtx files (45GB on disk). It worked like a charm. I did notice when extending the glob to all 5 thousand files in the folder, that winlogbeat did choke, causing floods of messages about the event api being to busy.

Looking forward for this feature to make it into release 😄
We archive our Windows Events onto a central WEC server, which archives the logs into 500MB chunks. At the moment, that means, if you halt the Winlogbeat process for a couple of minutes, you will loose logs due to the rotation.

[mgoritzk]

+1

[urso]

@andrewkroh How relevant is this old meta issue?

[mgoritzk]

It's still relevant. Currently we need to execute complex grok parsing with Logstash to make the output pretty.

[ruflin]

I'm closing this general meta issue. In the meantime lots of changes happen. Lets track additional efforts for winlogbeat in separate issues. Many of them are already linked above.