From 663becbcc8ebcefb13e8579090ada09b53be6d8a Mon Sep 17 00:00:00 2001 From: Andrew Kroh Date: Wed, 23 Mar 2016 18:05:08 -0400 Subject: [PATCH 1/3] Add documentation of the event log query options Add documentation of the include_xml option Add new FAQ question Format lines that extended beyond 80 characters --- winlogbeat/docs/configuring-howto.asciidoc | 7 +- winlogbeat/docs/faq.asciidoc | 10 +- winlogbeat/docs/fields.asciidoc | 11 ++ winlogbeat/docs/getting-started.asciidoc | 27 ++-- winlogbeat/docs/overview.asciidoc | 4 +- .../configuration/winlogbeat-options.asciidoc | 146 +++++++++++++++--- winlogbeat/docs/troubleshooting.asciidoc | 5 +- winlogbeat/etc/fields.yml | 15 ++ 8 files changed, 184 insertions(+), 41 deletions(-) diff --git a/winlogbeat/docs/configuring-howto.asciidoc b/winlogbeat/docs/configuring-howto.asciidoc index 6a886baff3e..64eac7e9e02 100644 --- a/winlogbeat/docs/configuring-howto.asciidoc +++ b/winlogbeat/docs/configuring-howto.asciidoc @@ -2,7 +2,8 @@ == Configuring Winlogbeat After following the <> in the -Getting Started, you might want to fine tune the behavior of Winlogbeat. This section -describes some common use cases for changing configuration options. +Getting Started, you might want to fine tune the behavior of Winlogbeat. This +section describes some common use cases for changing configuration options. -For a complete description of all Winlogbeat configuration options, see <>. +For a complete description of all Winlogbeat configuration options, see +<>. diff --git a/winlogbeat/docs/faq.asciidoc b/winlogbeat/docs/faq.asciidoc index 3456f866b42..6c21e7d62cd 100644 --- a/winlogbeat/docs/faq.asciidoc +++ b/winlogbeat/docs/faq.asciidoc @@ -1,12 +1,18 @@ [[faq]] == Frequently Asked Questions -This section contains frequently asked questions about Winlogbeat. Also check out the -https://discuss.elastic.co/c/beats/winlogbeat[Winlogbeat discussion forum]. +This section contains frequently asked questions about Winlogbeat. Also check +out the https://discuss.elastic.co/c/beats/winlogbeat[Winlogbeat discussion +forum]. [[dashboard-fields-incorrect]] === Why is the dashboard in Kibana breaking up my data fields incorrectly? The index template might not be loaded correctly. See <>. +=== Why are there bogus computer_name fields reported in some events? + +Prior to the hostname configuration stage, during OS installation any event log +records generated may have a randomly assigned hostname. + include::../../libbeat/docs/shared-faq.asciidoc[] diff --git a/winlogbeat/docs/fields.asciidoc b/winlogbeat/docs/fields.asciidoc index 6f68c3eea9f..0c859380fef 100644 --- a/winlogbeat/docs/fields.asciidoc +++ b/winlogbeat/docs/fields.asciidoc @@ -276,3 +276,14 @@ required: False The version number of the event's definition. +==== xml + +type: string + +required: False + +The raw XML representation of the event obtained from Windows. This field is only available on operating systems supporting the Windows Event Log API (Microsoft Windows Vista and newer). This field is not included by default and must be enabled by setting `include_xml: true` as a configuration option for an individual event log. + +The XML representation of the event is useful for troubleshooting purposes. The data in the fields reported by Winlogbeat can be compared to the data in the XML to diagnose problems. + + diff --git a/winlogbeat/docs/getting-started.asciidoc b/winlogbeat/docs/getting-started.asciidoc index 5034bcc754f..c4ff81e5048 100644 --- a/winlogbeat/docs/getting-started.asciidoc +++ b/winlogbeat/docs/getting-started.asciidoc @@ -1,15 +1,18 @@ [[winlogbeat-getting-started]] == Getting Started With Winlogbeat -To get started with your own Winlogbeat setup, install and configure these related products: +To get started with your own Winlogbeat setup, install and configure these +related products: * Elasticsearch for storage and indexing the data. * Kibana for the UI. * Logstash (optional) for inserting data into Elasticsearch. -See {libbeat}/getting-started.html[Getting Started with Beats and the Elastic Stack] for more information. +See {libbeat}/getting-started.html[Getting Started with Beats and the Elastic +Stack] for more information. -After you have installed these products, you can start <>. +After you have installed these products, you can start +<>. [[winlogbeat-installation]] === Step 1: Installing Winlogbeat @@ -19,7 +22,8 @@ https://www.elastic.co/downloads/beats/winlogbeat[downloads page]. . Extract the contents into `C:\Program Files`. . Rename the `winlogbeat-` directory to `Winlogbeat`. . Open a PowerShell prompt as an Administrator (right-click on the PowerShell -icon and select Run As Administrator). If you are running Windows XP, you may need to download and install PowerShell. +icon and select Run As Administrator). If you are running Windows XP, you may +need to download and install PowerShell. . Run the following commands to install the service. ["source","sh",subs="attributes,callouts"] @@ -50,8 +54,8 @@ For more information about these options, see <>. . If you are sending output to Elasticsearch, set the IP address and port where @@ -159,7 +163,8 @@ we have created a few sample dashboards. The dashboards are maintained in this https://github.com/elastic/beats-dashboards[GitHub repository], which also includes instructions for loading the dashboards. -For more information about loading and viewing the dashboards, see {libbeat}/visualizing-data.html[Visualizing Your Data in Kibana]. +For more information about loading and viewing the dashboards, see +{libbeat}/visualizing-data.html[Visualizing Your Data in Kibana]. image:./images/winlogbeat-dashboard.png[Winlogbeat statistics] diff --git a/winlogbeat/docs/overview.asciidoc b/winlogbeat/docs/overview.asciidoc index b8e4f5f4cf7..451afc2544d 100644 --- a/winlogbeat/docs/overview.asciidoc +++ b/winlogbeat/docs/overview.asciidoc @@ -10,8 +10,8 @@ logs so that new event data is sent in a timely manner. The read position for each event log is persisted to disk to allow Winlogbeat to resume after restarts. -Winlogbeat can capture event data from any event logs running -on your system. For example, you can capture events such as: +Winlogbeat can capture event data from any event logs running on your system. +For example, you can capture events such as: * application events * hardware events diff --git a/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc b/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc index 797bffbb089..b5f74741693 100644 --- a/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc +++ b/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc @@ -1,3 +1,6 @@ +:vista_and_newer: This option is only available on operating systems + + supporting the Windows Event Log API (Microsoft Windows Vista and newer). + [[configuration-winlogbeat-options]] === Winlogbeat @@ -15,7 +18,6 @@ winlogbeat: ignore_older: 72h - name: Security - name: System - -------------------------------------------------------------------------------- ==== Winlogbeat Options @@ -36,16 +38,17 @@ winlogbeat: registry_file: C:/ProgramData/winlogbeat/.winlogbeat.yml -------------------------------------------------------------------------------- -NOTE: The forward slashes (/) in the path are automatically changed to backslashes -(\) for Windows compatibility. You can use either forward or backslashes. Forward -slashes are easier to work with in YAML because there is no need to escape them. +NOTE: The forward slashes (/) in the path are automatically changed to +backslashes (\) for Windows compatibility. You can use either forward or +backslashes. Forward slashes are easier to work with in YAML because there is no +need to escape them. ===== event_logs -A list of entries (called 'dictionaries' in YAML) that specify which event logs to -monitor. Each entry in the list defines an event log to monitor as well as any -information to be associated with the event log (filter, tags, and so on). The -`name` field is the only required field for each event log. +A list of entries (called 'dictionaries' in YAML) that specify which event logs +to monitor. Each entry in the list defines an event log to monitor as well as +any information to be associated with the event log (filter, tags, and so on). +The `name` field is the only required field for each event log. [source,yaml] -------------------------------------------------------------------------------- @@ -59,7 +62,8 @@ winlogbeat: The name of the event log to monitor. Each dictionary under `event_logs` must have a `name` field. You can get a list of available event logs by running -`Get-EventLog *` in PowerShell. Here is a sample of the output from the command: +`Get-EventLog *` in PowerShell. Here is a sample of the output from the +command: [source,sh] -------------------------------------------------------------------------------- @@ -76,10 +80,10 @@ PS C:\Users\vagrant> Get-EventLog * 15,360 0 OverwriteAsNeeded 464 Windows PowerShell -------------------------------------------------------------------------------- -Channel names can also be specified if running on Windows Vista or newer. -A channel is a named stream of events that transports events from an event -source to an event log. Most channels are tied to specific event publishers. -Here is an example showing how to list all channels using PowerShell. +Channel names can also be specified if running on Windows Vista or newer. A +channel is a named stream of events that transports events from an event source +to an event log. Most channels are tied to specific event publishers. Here is an +example showing how to list all channels using PowerShell. [source,sh] -------------------------------------------------------------------------------- @@ -110,11 +114,11 @@ winlogbeat: ===== event_logs.ignore_older -If this option is specified, Winlogbeat filters events that are -older than the specified amount of time. Valid time units are "ns", -"us" (or "µs"), "ms", "s", "m", "h". This option is useful when you are -beginning to monitor an event log that contains older records that you would -like to ignore. This field is optional. +If this option is specified, Winlogbeat filters events that are older than the +specified amount of time. Valid time units are "ns", "us" (or "µs"), "ms", "s", +"m", "h". This option is useful when you are beginning to monitor an event log +that contains older records that you would like to ignore. This field is +optional. [source,yaml] -------------------------------------------------------------------------------- @@ -124,6 +128,111 @@ winlogbeat: ignore_older: 168h -------------------------------------------------------------------------------- +===== event_logs.event_id + +A whitelist and blacklist of event IDs. The value is a comma-separated list. The +accepted values are single event IDs to include (e.g. 4624), a range of event +IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). +*{vista_and_newer}* + +[source,yaml] +-------------------------------------------------------------------------------- +winlogbeat: + event_logs: + - name: Security + event_id: 4624, 4625, 4700-4800, -4735 +-------------------------------------------------------------------------------- + +===== event_logs.level + +A list of event levels to include. The value is a comma-separated list of +levels. *{vista_and_newer}* + +[cols="3*", options="header"] +|=== +|Level +|Value + +|critical, crit +|1 + +|error, err +|2 + +|warning, warn +|3 + +|information, info +|0 or 4 + +|verbose +|5 +|=== + +[source,yaml] +-------------------------------------------------------------------------------- +winlogbeat: + event_logs: + - name: Security + level: critical, error, warning +-------------------------------------------------------------------------------- + +===== event_logs.provider + +A list of providers (source names) to include. The value is a YAML list. +*{vista_and_newer}* + +[source,yaml] +-------------------------------------------------------------------------------- +winlogbeat: + event_logs: + - name: Application + provider: + - Application Error + - Application Hang + - Windows Error Reporting + - EMET +-------------------------------------------------------------------------------- + +You can obtain a list of providers associated with a log by using PowerShell. +Here is an example showing the providers associated with the Security log. + +[source,sh] +-------------------------------------------------------------------------------- +PS C:\> (Get-WinEvent -ListLog Security).ProviderNames +DS +LSA +SC Manager +Security +Security Account Manager +ServiceModel 4.0.0.0 +Spooler +TCP/IP +VSSAudit +Microsoft-Windows-Security-Auditing +Microsoft-Windows-Eventlog +-------------------------------------------------------------------------------- + +===== event_logs.include_xml + +Boolean option that controls if the raw XML representation of an event is +included in the data sent by Winlogbeat. The default is false. +*{vista_and_newer}* + +The XML representation of the event is useful for troubleshooting purposes. The +data in the fields reported by Winlogbeat can be compared to the data in the XML +to diagnose problems. + +Example: + +[source,yaml] +-------------------------------------------------------------------------------- +winlogbeat: + event_logs: + - name: Microsoft-Windows-Windows Defender/Operational + include_xml: true +-------------------------------------------------------------------------------- + ===== event_logs.tags A list of tags that the Beat includes in the `tags` field of each published @@ -188,7 +297,6 @@ winlogbeat: The metrics are served as a JSON document. The metrics include: - memory stats -- number of filtered events from each log - number of published events from each log - total number of failures while publishing - total number of filtered events diff --git a/winlogbeat/docs/troubleshooting.asciidoc b/winlogbeat/docs/troubleshooting.asciidoc index 575e808c8a6..2e616bdea17 100644 --- a/winlogbeat/docs/troubleshooting.asciidoc +++ b/winlogbeat/docs/troubleshooting.asciidoc @@ -3,8 +3,7 @@ [partintro] -- -If you have issues installing or running Winlogbeat, read the -following tips. +If you have issues installing or running Winlogbeat, read the following tips. -- @@ -17,5 +16,3 @@ include::../../libbeat/docs/getting-help.asciidoc[] == Debugging include::../../libbeat/docs/debugging.asciidoc[] - - diff --git a/winlogbeat/etc/fields.yml b/winlogbeat/etc/fields.yml index 2ff4d2cb6dd..142a607e45b 100644 --- a/winlogbeat/etc/fields.yml +++ b/winlogbeat/etc/fields.yml @@ -226,6 +226,21 @@ eventlog: required: false description: The version number of the event's definition. + - name: xml + type: string + required: false + description: > + The raw XML representation of the event obtained from Windows. This + field is only available on operating systems supporting the Windows + Event Log API (Microsoft Windows Vista and newer). This field is not + included by default and must be enabled by setting `include_xml: true` + as a configuration option for an individual event log. + + + The XML representation of the event is useful for troubleshooting + purposes. The data in the fields reported by Winlogbeat can be compared + to the data in the XML to diagnose problems. + sections: - ["common", "Common Beat"] - ["eventlog", "Event Log Record"] From e5565f5da5349cb8e2925632b7ba4fd523500d6e Mon Sep 17 00:00:00 2001 From: Andrew Kroh Date: Wed, 23 Mar 2016 18:04:14 -0400 Subject: [PATCH 2/3] Add query by event ID, level, provider, and age (time) Use ucfg to unpack eventlog API config All system tests now require Windows. Add ignore_older filtering to eventlogging API since it is not provided by Windows Change expected time.ParseDuration error message due to ucfg change --- .travis.yml | 4 - CHANGELOG.asciidoc | 3 + winlogbeat/beater/winlogbeat.go | 75 ++--- winlogbeat/config/config.go | 71 +--- winlogbeat/config/config_test.go | 57 +--- winlogbeat/etc/beat.yml | 13 +- winlogbeat/eventlog/eventlog.go | 6 +- winlogbeat/eventlog/eventlogging.go | 63 +++- winlogbeat/eventlog/eventlogging_test.go | 20 +- winlogbeat/eventlog/factory.go | 70 +++- winlogbeat/eventlog/wineventlog.go | 67 +++- winlogbeat/sys/wineventlog/query.go | 214 ++++++++++++ winlogbeat/sys/wineventlog/query_test.go | 113 +++++++ .../sys/wineventlog/wineventlog_windows.go | 36 +- .../tests/system/config/winlogbeat.yml.j2 | 35 +- winlogbeat/tests/system/test_config.py | 44 ++- winlogbeat/tests/system/test_eventlog.py | 316 ------------------ winlogbeat/tests/system/test_eventlogging.py | 150 +++++++++ winlogbeat/tests/system/test_wineventlog.py | 286 ++++++++++++++++ winlogbeat/tests/system/winlogbeat.py | 114 ++++++- winlogbeat/winlogbeat.yml | 13 +- 21 files changed, 1201 insertions(+), 569 deletions(-) create mode 100644 winlogbeat/sys/wineventlog/query.go create mode 100644 winlogbeat/sys/wineventlog/query_test.go delete mode 100644 winlogbeat/tests/system/test_eventlog.py create mode 100644 winlogbeat/tests/system/test_eventlogging.py create mode 100644 winlogbeat/tests/system/test_wineventlog.py diff --git a/.travis.yml b/.travis.yml index 510755edca4..5d48a39ee6a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -18,7 +18,6 @@ env: - TARGETS="-C libbeat testsuite" - TARGETS="-C topbeat testsuite" - TARGETS="-C filebeat testsuite" - - TARGETS="-C winlogbeat testsuite" - TARGETS="-C packetbeat testsuite" - TARGETS="-C metricbeat testsuite" - TARGETS="-C libbeat crosscompile" @@ -36,8 +35,6 @@ matrix: env: TARGETS="-C filebeat crosscompile" - os: osx env: TARGETS="-C libbeat crosscompile" - - os: osx - env: TARGETS="-C winlogbeat testsuite" - os: osx env: TARGETS="-C winlogbeat crosscompile" - os: osx @@ -86,5 +83,4 @@ after_success: - test -f packetbeat/build/coverage/full.cov && bash <(curl -s https://codecov.io/bash) -f packetbeat/build/coverage/full.cov - test -f topbeat/build/coverage/full.cov && bash <(curl -s https://codecov.io/bash) -f topbeat/build/coverage/full.cov - test -f libbeat/build/coverage/full.cov && bash <(curl -s https://codecov.io/bash) -f libbeat/build/coverage/full.cov - - test -f winlogbeat/build/coverage/full.cov && bash <(curl -s https://codecov.io/bash) -f winlogbeat/build/coverage/full.cov - test -f metricbeat/build/coverage/full.cov && bash <(curl -s https://codecov.io/bash) -f metricbeat/build/coverage/full.cov diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 744f93a8b96..37bc6e90d72 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -65,6 +65,7 @@ https://github.com/elastic/beats/compare/v1.1.2...master[Check the HEAD diff] - Omit `fields` from Filebeat events when null {issue}899[899] *Winlogbeat* +- Fix invalid `event_id` on Windows XP and Windows 2003 {pull}1153[1153] ==== Added @@ -105,6 +106,8 @@ https://github.com/elastic/beats/compare/v1.1.2...master[Check the HEAD diff] - Add additional data to the events published by Winlogbeat. The new fields are `activity_id`, `event_data`, `keywords`, `opcode`, `process_id`, `provider_guid`, `related_activity_id`, `task`, `thread_id`, `user_data`. and `version`. {issue}1053[1053] +- Add `event_id`, `level`, and `provider` configuration options for filtering events {pull}1218[1218] +- Add `include_xml` configuration option for including the raw XML with the event {pull}1218[1218] ==== Deprecated diff --git a/winlogbeat/beater/winlogbeat.go b/winlogbeat/beater/winlogbeat.go index b6fe8a77eef..9c38f7abfa0 100644 --- a/winlogbeat/beater/winlogbeat.go +++ b/winlogbeat/beater/winlogbeat.go @@ -33,23 +33,17 @@ func init() { // Debug logging functions for this package. var ( debugf = logp.MakeDebug("winlogbeat") - detailf = logp.MakeDebug("winlogbeat_detail") memstatsf = logp.MakeDebug("memstats") ) // Time the application was started. var startTime = time.Now().UTC() -type log struct { - config.EventLogConfig - eventLog eventlog.EventLog -} - // Winlogbeat is used to conform to the beat interface type Winlogbeat struct { beat *beat.Beat // Common beat information. config *config.Settings // Configuration settings. - eventLogs []log // List of all event logs being monitored. + eventLogs []eventlog.EventLog // List of all event logs being monitored. done chan struct{} // Channel to initiate shutdown of main event loop. client publisher.Client // Interface to publish event. checkpoint *checkpoint.Checkpoint // Persists event log state to disk. @@ -116,15 +110,27 @@ func (eb *Winlogbeat) Setup(b *beat.Beat) error { err := http.Serve(sock, nil) if err != nil { logp.Warn("Unable to launch HTTP service for metrics. %v", err) - return } }() } + // Create the event logs. This will validate the event log specific + // configuration. + eb.eventLogs = make([]eventlog.EventLog, 0, len(eb.config.Winlogbeat.EventLogs)) + for _, config := range eb.config.Winlogbeat.EventLogs { + eventLog, err := eventlog.New(config) + if err != nil { + return fmt.Errorf("Failed to create new event log. %v", err) + } + debugf("Initialized EventLog[%s]", eventLog.Name()) + + eb.eventLogs = append(eb.eventLogs, eventLog) + } + return nil } -// Run is used within the beats interface to execute the winlogbeat. +// Run is used within the beats interface to execute the Winlogbeat workers. func (eb *Winlogbeat) Run(b *beat.Beat) error { persistedState := eb.checkpoint.States() @@ -133,40 +139,17 @@ func (eb *Winlogbeat) Run(b *beat.Beat) error { publishedEvents.Add("failures", 0) ignoredEvents.Add("total", 0) - // TODO: If no event_logs are specified in the configuration, use the - // Windows registry to discover the available event logs. - eb.eventLogs = make([]log, 0, len(eb.config.Winlogbeat.EventLogs)) - for _, eventLogConfig := range eb.config.Winlogbeat.EventLogs { - debugf("Initializing EventLog[%s]", eventLogConfig.Name) - - eventLog, err := eventlog.New(eventlog.Config{ - Name: eventLogConfig.Name, - API: eventLogConfig.API, - EventMetadata: eventLogConfig.EventMetadata, - }) - if err != nil { - return fmt.Errorf("Failed to create new event log for %s. %v", - eventLogConfig.Name, err) - } - - // Initialize per event log metrics. - publishedEvents.Add(eventLogConfig.Name, 0) - ignoredEvents.Add(eventLogConfig.Name, 0) - - eb.eventLogs = append(eb.eventLogs, log{ - EventLogConfig: eventLogConfig, - eventLog: eventLog, - }) - } - var wg sync.WaitGroup for _, log := range eb.eventLogs { - state, _ := persistedState[log.Name] - ignoreOlder, _ := config.IgnoreOlderDuration(log.IgnoreOlder) + state, _ := persistedState[log.Name()] + + // Initialize per event log metrics. + publishedEvents.Add(log.Name(), 0) + ignoredEvents.Add(log.Name(), 0) // Start a goroutine for each event log. wg.Add(1) - go eb.processEventLog(&wg, log.eventLog, state, ignoreOlder) + go eb.processEventLog(&wg, log, state) } wg.Wait() @@ -201,7 +184,6 @@ func (eb *Winlogbeat) processEventLog( wg *sync.WaitGroup, api eventlog.EventLog, state checkpoint.EventLogState, - ignoreOlder time.Duration, ) { defer wg.Done() @@ -243,21 +225,8 @@ loop: continue } - // Filter events. - var events []common.MapStr + events := make([]common.MapStr, 0, len(records)) for _, lr := range records { - // TODO: Move filters close to source. Short circuit processing - // of event if it is going to be filtered. - // TODO: Add a severity filter. - // TODO: Check the global IgnoreOlder filter. - if ignoreOlder != 0 && time.Since(lr.TimeCreated.SystemTime) > ignoreOlder { - detailf("EventLog[%s] ignore_older filter dropping event: %+v", - api.Name(), lr) - ignoredEvents.Add("total", 1) - ignoredEvents.Add(api.Name(), 1) - continue - } - events = append(events, lr.ToMapStr()) } diff --git a/winlogbeat/config/config.go b/winlogbeat/config/config.go index 46110ff0018..ab9ababb2db 100644 --- a/winlogbeat/config/config.go +++ b/winlogbeat/config/config.go @@ -7,9 +7,7 @@ import ( "sort" "strconv" "strings" - "time" - "github.com/elastic/beats/libbeat/common" "github.com/joeshaw/multierror" ) @@ -30,7 +28,7 @@ type Validator interface { // Settings is the root of the Winlogbeat configuration data hierarchy. type Settings struct { Winlogbeat WinlogbeatConfig `config:"winlogbeat"` - All map[string]interface{} `config:",inline"` + Raw map[string]interface{} `config:",inline"` } // Validate validates the Settings data and returns an error describing @@ -41,7 +39,7 @@ func (s Settings) Validate() error { // Check for invalid top-level keys. var errs multierror.Errors - for k := range s.All { + for k := range s.Raw { k = strings.ToLower(k) i := sort.SearchStrings(validKeys, k) if i >= len(validKeys) || validKeys[i] != k { @@ -60,32 +58,21 @@ func (s Settings) Validate() error { // WinlogbeatConfig contains all of Winlogbeat configuration data. type WinlogbeatConfig struct { - IgnoreOlder string `config:"ignore_older"` - EventLogs []EventLogConfig `config:"event_logs"` - Metrics MetricsConfig `config:"metrics"` - RegistryFile string `config:"registry_file"` + EventLogs []map[string]interface{} `config:"event_logs"` + Metrics MetricsConfig `config:"metrics"` + RegistryFile string `config:"registry_file"` } // Validate validates the WinlogbeatConfig data and returns an error describing // all problems or nil if there are none. func (ebc WinlogbeatConfig) Validate() error { var errs multierror.Errors - if _, err := IgnoreOlderDuration(ebc.IgnoreOlder); err != nil { - errs = append(errs, fmt.Errorf("Invalid top level ignore_older value "+ - "'%s' (%v)", ebc.IgnoreOlder, err)) - } if len(ebc.EventLogs) == 0 { errs = append(errs, fmt.Errorf("At least one event log must be "+ "configured as part of event_logs")) } - for _, eventLog := range ebc.EventLogs { - if err := eventLog.Validate(); err != nil { - errs = append(errs, err) - } - } - if err := ebc.Metrics.Validate(); err != nil { errs = append(errs, err) } @@ -129,51 +116,3 @@ func (mc MetricsConfig) Validate() error { return nil } - -// EventLogConfig holds the configuration data that specifies which event logs -// to monitor. -type EventLogConfig struct { - common.EventMetadata `config:",inline"` - Name string - IgnoreOlder string `config:"ignore_older"` - API string -} - -// Validate validates the EventLogConfig data and returns an error describing -// any problems or nil. -func (elc EventLogConfig) Validate() error { - var errs multierror.Errors - if elc.Name == "" { - err := fmt.Errorf("event log is missing a 'name'") - errs = append(errs, err) - } - - if _, err := IgnoreOlderDuration(elc.IgnoreOlder); err != nil { - err := fmt.Errorf("Invalid ignore_older value ('%s') for event_log "+ - "'%s' (%v)", elc.IgnoreOlder, elc.Name, err) - errs = append(errs, err) - } - - switch strings.ToLower(elc.API) { - case "", "eventlogging", "wineventlog": - break - default: - err := fmt.Errorf("Invalid api value ('%s') for event_log '%s'", - elc.API, elc.Name) - errs = append(errs, err) - } - - return errs.Err() -} - -// IgnoreOlderDuration returns the parsed value of the IgnoreOlder string. If -// IgnoreOlder is not set then (0, nil) is returned. If IgnoreOlder is not -// parsable as a duration then an error is returned. See time.ParseDuration. -func IgnoreOlderDuration(ignoreOlder string) (time.Duration, error) { - if ignoreOlder == "" { - return time.Duration(0), nil - } - - duration, err := time.ParseDuration(ignoreOlder) - return duration, err -} diff --git a/winlogbeat/config/config_test.go b/winlogbeat/config/config_test.go index 75f0c369623..2c37ddeb567 100644 --- a/winlogbeat/config/config_test.go +++ b/winlogbeat/config/config_test.go @@ -17,7 +17,10 @@ func (v validationTestCase) run(t *testing.T) { if v.errMsg == "" { assert.NoError(t, v.config.Validate()) } else { - assert.Contains(t, v.config.Validate().Error(), v.errMsg) + err := v.config.Validate() + if assert.Error(t, err, "expected '%s'", v.errMsg) { + assert.Contains(t, err.Error(), v.errMsg) + } } } @@ -26,8 +29,8 @@ func TestConfigValidate(t *testing.T) { // Top-level config { WinlogbeatConfig{ - EventLogs: []EventLogConfig{ - {Name: "App"}, + EventLogs: []map[string]interface{}{ + {"Name": "App"}, }, }, "", // No Error @@ -35,8 +38,8 @@ func TestConfigValidate(t *testing.T) { { Settings{ WinlogbeatConfig{ - EventLogs: []EventLogConfig{ - {Name: "App"}, + EventLogs: []map[string]interface{}{ + {"Name": "App"}, }, }, map[string]interface{}{"other": "value"}, @@ -49,29 +52,15 @@ func TestConfigValidate(t *testing.T) { "1 error: At least one event log must be configured as part of " + "event_logs", }, - { - WinlogbeatConfig{IgnoreOlder: "1"}, - "2 errors: Invalid top level ignore_older value '1' (time: " + - "missing unit in duration 1); At least one event log must be " + - "configured as part of event_logs", - }, { WinlogbeatConfig{ - EventLogs: []EventLogConfig{ - {Name: "App"}, + EventLogs: []map[string]interface{}{ + {"Name": "App"}, }, Metrics: MetricsConfig{BindAddress: "example.com"}, }, "1 error: bind_address", }, - { - WinlogbeatConfig{ - EventLogs: []EventLogConfig{ - {}, - }, - }, - "1 error: event log is missing a 'name'", - }, // MetricsConfig { MetricsConfig{}, @@ -103,32 +92,6 @@ func TestConfigValidate(t *testing.T) { MetricsConfig{BindAddress: "example.com:65536"}, "bind_address port must be within [1-65535] but was '65536'", }, - // EventLogConfig - { - EventLogConfig{Name: "System"}, - "", - }, - { - EventLogConfig{}, - "event log is missing a 'name'", - }, - { - EventLogConfig{Name: "System", IgnoreOlder: "24"}, - "Invalid ignore_older value ('24') for event_log 'System' " + - "(time: missing unit in duration 24)", - }, - { - EventLogConfig{Name: "System", API: "eventlogging"}, - "", - }, - { - EventLogConfig{Name: "System", API: "wineventlog"}, - "", - }, - { - EventLogConfig{Name: "System", API: "invalid"}, - "Invalid api value ('invalid') for event_log 'System'", - }, } for _, test := range testCases { diff --git a/winlogbeat/etc/beat.yml b/winlogbeat/etc/beat.yml index b584df49c59..abc145ac875 100644 --- a/winlogbeat/etc/beat.yml +++ b/winlogbeat/etc/beat.yml @@ -6,14 +6,17 @@ winlogbeat: # in the directory in which it was started. #registry_file: .winlogbeat.yml - # List of event logs to monitor. + # event_logs specifies a list of event logs to monitor as well as any + # accompanying options. The YAML data type of event_logs is a list of + # dictionaries. # - # Optionally, ignore_older may be specified to filter events that are older - # then the specified amount of time. If omitted then no filtering will - # occur. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h" + # The supported keys are name (required), tags, fields, fields_under_root, + # ignore_older, level, event_id, provider, and include_xml. Please visit the + # documentation for the complete details of each option. + # https://go.es.io/WinlogbeatConfig event_logs: - name: Application - ignore_older: 72h + ignore_older: 72h - name: Security - name: System diff --git a/winlogbeat/eventlog/eventlog.go b/winlogbeat/eventlog/eventlog.go index cd1d07f9864..19f6afe5163 100644 --- a/winlogbeat/eventlog/eventlog.go +++ b/winlogbeat/eventlog/eventlog.go @@ -46,9 +46,10 @@ type EventLog interface { // Record represents a single event from the log. type Record struct { - API string // The event log API type used to read the record. - common.EventMetadata // Fields and tags to add to the event. sys.Event + common.EventMetadata // Fields and tags to add to the event. + API string // The event log API type used to read the record. + XML string // XML representation of the event. } // ToMapStr returns a new MapStr containing the data from this Record. @@ -65,6 +66,7 @@ func (e Record) ToMapStr() common.MapStr { "event_id": e.EventIdentifier.ID, } + addOptional(m, "xml", e.XML) addOptional(m, "provider_guid", e.Provider.GUID) addOptional(m, "version", e.Version) addOptional(m, "level", e.Level) diff --git a/winlogbeat/eventlog/eventlogging.go b/winlogbeat/eventlog/eventlogging.go index b4a698ccb0f..33937b783ce 100644 --- a/winlogbeat/eventlog/eventlogging.go +++ b/winlogbeat/eventlog/eventlogging.go @@ -5,11 +5,13 @@ package eventlog import ( "fmt" "syscall" + "time" "github.com/elastic/beats/libbeat/common" "github.com/elastic/beats/libbeat/logp" "github.com/elastic/beats/winlogbeat/sys" win "github.com/elastic/beats/winlogbeat/sys/eventlogging" + "github.com/joeshaw/multierror" ) const ( @@ -18,13 +20,32 @@ const ( eventLoggingAPIName = "eventlogging" ) +var eventLoggingConfigKeys = append(commonConfigKeys, "ignore_older") + +type eventLoggingConfig struct { + configCommon `config:",inline"` + IgnoreOlder time.Duration `config:"ignore_older"` + Raw map[string]interface{} `config:",inline"` +} + +// Validate validates the eventLoggingConfig data and returns an error +// describing any problems or nil. +func (c *eventLoggingConfig) Validate() error { + var errs multierror.Errors + if c.Name == "" { + errs = append(errs, fmt.Errorf("event log is missing a 'name'")) + } + + return errs.Err() +} + // Validate that eventLogging implements the EventLog interface. var _ EventLog = &eventLogging{} // eventLogging implements the EventLog interface for reading from the Event // Logging API. type eventLogging struct { - uncServerPath string // UNC name of remote server. + config eventLoggingConfig name string // Name of the log that is opened. handle win.Handle // Handle to the event log. readBuf []byte // Buffer for reading in events. @@ -44,9 +65,9 @@ func (l eventLogging) Name() string { } func (l *eventLogging) Open(recordNumber uint64) error { - detailf("%s Open(recordNumber=%d) calling OpenEventLog(uncServerPath=%s, "+ - "providerName=%s)", l.logPrefix, recordNumber, l.uncServerPath, l.name) - handle, err := win.OpenEventLog(l.uncServerPath, l.name) + detailf("%s Open(recordNumber=%d) calling OpenEventLog(uncServerPath=, "+ + "providerName=%s)", l.logPrefix, recordNumber, l.name) + handle, err := win.OpenEventLog("", l.name) if err != nil { return err } @@ -147,6 +168,7 @@ func (l *eventLogging) Read() ([]Record, error) { l.ignoreFirst = false } + records = filter(records, l.ignoreOlder) debugf("%s Read() is returning %d records", l.logPrefix, len(records)) return records, nil } @@ -191,12 +213,39 @@ func readErrorHandler(err error) ([]Record, error) { return nil, err } +// Filter returns a new slice holding only the elements of s that satisfy the +// predicate fn(). +func filter(in []Record, fn func(*Record) bool) []Record { + var out []Record + for _, r := range in { + if fn(&r) { + out = append(out, r) + } + } + return out +} + +// ignoreOlder is a filter predicate that checks the record timestamp and +// returns true if the event was not matched by the filter. +func (l *eventLogging) ignoreOlder(r *Record) bool { + if l.config.IgnoreOlder != 0 && time.Since(r.TimeCreated.SystemTime) > l.config.IgnoreOlder { + return false + } + + return true +} + // newEventLogging creates and returns a new EventLog for reading event logs // using the Event Logging API. -func newEventLogging(c Config) (EventLog, error) { +func newEventLogging(options map[string]interface{}) (EventLog, error) { + var c eventLoggingConfig + if err := readConfig(options, &c, eventLoggingConfigKeys); err != nil { + return nil, err + } + return &eventLogging{ - uncServerPath: c.RemoteAddress, - name: c.Name, + config: c, + name: c.Name, handles: newMessageFilesCache(c.Name, win.QueryEventMessageFiles, win.FreeLibrary), logPrefix: fmt.Sprintf("EventLogging[%s]", c.Name), diff --git a/winlogbeat/eventlog/eventlogging_test.go b/winlogbeat/eventlog/eventlogging_test.go index ac1aa41286f..caf3aa15d09 100644 --- a/winlogbeat/eventlog/eventlogging_test.go +++ b/winlogbeat/eventlog/eventlogging_test.go @@ -1,4 +1,4 @@ -// +build windows,integration +// +build windows package eventlog @@ -167,7 +167,7 @@ func TestRead(t *testing.T) { } // Read messages: - eventlog, err := newEventLogging(Config{Name: providerName}) + eventlog, err := newEventLogging(map[string]interface{}{"name": providerName}) if err != nil { t.Fatal(err) } @@ -229,7 +229,7 @@ func TestReadUnknownEventId(t *testing.T) { } // Read messages: - eventlog, err := newEventLogging(Config{Name: providerName}) + eventlog, err := newEventLogging(map[string]interface{}{"name": providerName}) if err != nil { t.Fatal(err) } @@ -286,7 +286,7 @@ func TestReadTriesMultipleEventMsgFiles(t *testing.T) { } // Read messages: - eventlog, err := newEventLogging(Config{Name: providerName}) + eventlog, err := newEventLogging(map[string]interface{}{"name": providerName}) if err != nil { t.Fatal(err) } @@ -342,7 +342,7 @@ func TestReadMultiParameterMsg(t *testing.T) { } // Read messages: - eventlog, err := newEventLogging(Config{Name: providerName}) + eventlog, err := newEventLogging(map[string]interface{}{"name": providerName}) if err != nil { t.Fatal(err) } @@ -366,7 +366,7 @@ func TestReadMultiParameterMsg(t *testing.T) { if len(records) != 1 { t.FailNow() } - assert.Equal(t, eventID, records[0].EventIdentifier.ID) + assert.Equal(t, eventID&0xFFFF, records[0].EventIdentifier.ID) assert.Equal(t, fmt.Sprintf(template, msgs[0], msgs[1]), strings.TrimRight(records[0].Message, "\r\n")) } @@ -377,7 +377,7 @@ func TestOpenInvalidProvider(t *testing.T) { configureLogp() - el, err := newEventLogging(Config{Name: "nonExistentProvider"}) + el, err := newEventLogging(map[string]interface{}{"name": "nonExistentProvider"}) if err != nil { t.Fatal(err) } @@ -411,7 +411,7 @@ func TestReadNoParameterMsg(t *testing.T) { } // Read messages: - eventlog, err := newEventLogging(Config{Name: providerName}) + eventlog, err := newEventLogging(map[string]interface{}{"name": providerName}) if err != nil { t.Fatal(err) } @@ -435,7 +435,7 @@ func TestReadNoParameterMsg(t *testing.T) { if len(records) != 1 { t.FailNow() } - assert.Equal(t, eventID, records[0].EventIdentifier.ID) + assert.Equal(t, eventID&0xFFFF, records[0].EventIdentifier.ID) assert.Equal(t, template, strings.TrimRight(records[0].Message, "\r\n")) } @@ -456,7 +456,7 @@ func TestReadWhileCleared(t *testing.T) { } }() - eventlog, err := newEventLogging(Config{Name: providerName}) + eventlog, err := newEventLogging(map[string]interface{}{"name": providerName}) if err != nil { t.Fatal(err) } diff --git a/winlogbeat/eventlog/factory.go b/winlogbeat/eventlog/factory.go index e60c9399a7c..7b354f259cd 100644 --- a/winlogbeat/eventlog/factory.go +++ b/winlogbeat/eventlog/factory.go @@ -6,24 +6,67 @@ import ( "strings" "github.com/elastic/beats/libbeat/common" + "github.com/joeshaw/multierror" ) +var commonConfigKeys = []string{"api", "name", "fields", "fields_under_root", "tags"} + // Config is the configuration data used to instantiate a new EventLog. -type Config struct { - Name string // Name of the event log or channel. - RemoteAddress string // Remote computer to connect to. Optional. - common.EventMetadata // Fields and tags to add to each event. +type configCommon struct { + API string `config:"api"` // Name of the API to use. Optional. + Name string `config:"name"` // Name of the event log or channel. + common.EventMetadata `config:",inline"` // Fields and tags to add to each event. +} + +type validator interface { + Validate() error +} + +func readConfig( + data map[string]interface{}, + config interface{}, + validKeys []string, +) error { + c, err := common.NewConfigFrom(data) + if err != nil { + return fmt.Errorf("Failed reading config. %v", err) + } + + if err := c.Unpack(config); err != nil { + return fmt.Errorf("Failed unpacking config. %v", err) + } + + var errs multierror.Errors + if len(validKeys) > 0 { + sort.Strings(validKeys) - API string // Name of the API to use. Optional. + // Check for invalid keys. + for k := range data { + k = strings.ToLower(k) + i := sort.SearchStrings(validKeys, k) + if i >= len(validKeys) || validKeys[i] != k { + errs = append(errs, fmt.Errorf("Invalid event log key '%s' "+ + "found. Valid keys are %s", k, strings.Join(validKeys, ", "))) + } + } + } + + if v, ok := config.(validator); ok { + if err := v.Validate(); err != nil { + errs = append(errs, err) + } + } + + return errs.Err() } // Producer produces a new event log instance for reading event log records. -type producer func(Config) (EventLog, error) +type producer func(map[string]interface{}) (EventLog, error) // Channels lists the available channels (event logs). type channels func() ([]string, error) -// eventLogInfo is the registration info associate with an event log API. +// eventLogInfo is the registration info associated with an event log API. type eventLogInfo struct { apiName string priority int @@ -54,18 +97,23 @@ func Register(apiName string, priority int, producer producer, channels channels // New creates and returns a new EventLog instance based on the given config // and the registered EventLog producers. -func New(config Config) (EventLog, error) { +func New(options map[string]interface{}) (EventLog, error) { if len(eventLogs) == 0 { return nil, fmt.Errorf("No event log API is available on this system") } + var config configCommon + if err := readConfig(options, &config, nil); err != nil { + return nil, err + } + // A specific API is being requested (usually done for testing). if config.API != "" { for _, v := range eventLogs { - debugf("Testing %s", v.apiName) + debugf("Checking %s", v.apiName) if strings.EqualFold(v.apiName, config.API) { debugf("Using %s API for event log %s", v.apiName, config.Name) - e, err := v.producer(config) + e, err := v.producer(options) return e, err } } @@ -83,6 +131,6 @@ func New(config Config) (EventLog, error) { eventLog := eventLogs[keys[0]] debugf("Using highest priority API, %s, for event log %s", eventLog.apiName, config.Name) - e, err := eventLog.producer(config) + e, err := eventLog.producer(options) return e, err } diff --git a/winlogbeat/eventlog/wineventlog.go b/winlogbeat/eventlog/wineventlog.go index dab18bc464a..7c17733434a 100644 --- a/winlogbeat/eventlog/wineventlog.go +++ b/winlogbeat/eventlog/wineventlog.go @@ -6,11 +6,13 @@ import ( "fmt" "strconv" "syscall" + "time" "github.com/elastic/beats/libbeat/common" "github.com/elastic/beats/libbeat/logp" "github.com/elastic/beats/winlogbeat/sys" win "github.com/elastic/beats/winlogbeat/sys/wineventlog" + "github.com/joeshaw/multierror" "golang.org/x/sys/windows" ) @@ -26,13 +28,44 @@ const ( winEventLogAPIName = "wineventlog" ) +var winEventLogConfigKeys = append(commonConfigKeys, "ignore_older", "include_xml", + "event_id", "level", "provider") + +type winEventLogConfig struct { + configCommon `config:",inline"` + IncludeXML bool `config:"include_xml"` + SimpleQuery query `config:",inline"` + Raw map[string]interface{} `config:",inline"` +} + +// query contains parameters used to customize the event log data that is +// queried from the log. +type query struct { + IgnoreOlder time.Duration `config:"ignore_older"` // Ignore records older than this period of time. + EventID string `config:"event_id"` // White-list and black-list of events. + Level string `config:"level"` // Severity level. + Provider []string `config:"provider"` // Provider (source name). +} + +// Validate validates the winEventLogConfig data and returns an error describing +// any problems or nil. +func (c *winEventLogConfig) Validate() error { + var errs multierror.Errors + if c.Name == "" { + errs = append(errs, fmt.Errorf("event log is missing a 'name'")) + } + + return errs.Err() +} + // Validate that winEventLog implements the EventLog interface. var _ EventLog = &winEventLog{} // winEventLog implements the EventLog interface for reading from the Windows // Event Log API. type winEventLog struct { - remoteServer string // Name of the remote server from which to read. + config winEventLogConfig + query string channelName string // Name of the channel from which to read. subscription win.EvtHandle // Handle to the subscription. maxRead int // Maximum number returned in one Read. @@ -63,11 +96,12 @@ func (l *winEventLog) Open(recordNumber uint64) error { return nil } + debugf("%s using subscription query=%s", l.logPrefix, l.query) subscriptionHandle, err := win.Subscribe( - 0, // null session (used for connecting to remote event logs) + 0, // Session - nil for localhost signalEvent, - l.channelName, - "", // Query - nil means all events + "", // Channel - empty b/c channel is in the query + l.query, // Query - nil means all events bookmark, // Bookmark - for resuming from a specific event win.EvtSubscribeStartAfterBookmark) if err != nil { @@ -158,6 +192,10 @@ func (l *winEventLog) buildRecordFromXML(x string, recoveredErr error) (Record, Event: e, } + if l.config.IncludeXML { + r.XML = x + } + return r, nil } @@ -177,7 +215,23 @@ func reportDrop(reason interface{}) { // newWinEventLog creates and returns a new EventLog for reading event logs // using the Windows Event Log. -func newWinEventLog(c Config) (EventLog, error) { +func newWinEventLog(options map[string]interface{}) (EventLog, error) { + var c winEventLogConfig + if err := readConfig(options, &c, winEventLogConfigKeys); err != nil { + return nil, err + } + + query, err := win.Query{ + Log: c.Name, + IgnoreOlder: c.SimpleQuery.IgnoreOlder, + Level: c.SimpleQuery.Level, + EventID: c.SimpleQuery.EventID, + Provider: c.SimpleQuery.Provider, + }.Build() + if err != nil { + return nil, err + } + eventMetadataHandle := func(providerName, sourceName string) sys.MessageFiles { mf := sys.MessageFiles{SourceName: sourceName} h, err := win.OpenPublisherMetadata(0, sourceName, 0) @@ -195,8 +249,9 @@ func newWinEventLog(c Config) (EventLog, error) { } return &winEventLog{ + config: c, + query: query, channelName: c.Name, - remoteServer: c.RemoteAddress, maxRead: defaultMaxNumRead, renderBuf: make([]byte, renderBufferSize), cache: newMessageFilesCache(c.Name, eventMetadataHandle, freeHandle), diff --git a/winlogbeat/sys/wineventlog/query.go b/winlogbeat/sys/wineventlog/query.go new file mode 100644 index 00000000000..b5c856e5691 --- /dev/null +++ b/winlogbeat/sys/wineventlog/query.go @@ -0,0 +1,214 @@ +package wineventlog + +import ( + "bytes" + "fmt" + "regexp" + "strconv" + "strings" + "text/template" + "time" + + "github.com/joeshaw/multierror" +) + +const ( + query = ` + + {{if .Suppress}} + *[System[({{join .Suppress " or "}})]]{{end}} + +` +) + +var ( + templateFuncMap = template.FuncMap{"join": strings.Join} + queryTemplate = template.Must(template.New("query").Funcs(templateFuncMap).Parse(query)) + incEventIDRegex = regexp.MustCompile(`^\d+$`) + incEventIDRangeRegex = regexp.MustCompile(`^(\d+)\s*-\s*(\d+)$`) + excEventIDRegex = regexp.MustCompile(`^-(\d+)$`) +) + +// Query that identifies the source of the events and one or more selectors or +// suppressors. +type Query struct { + // Name of the channel or the path to the log file that contains the events + // to query. + Log string + + IgnoreOlder time.Duration // Ignore records older than this time period. + + // Whitelist and blacklist of event IDs. The value is a comma-separated + // list. The accepted values are single event IDs to include (e.g. 4634), a + // range of event IDs to include (e.g. 4400-4500), and single event IDs to + // exclude (e.g. -4410). + EventID string + + // Level or levels to include. The value is a comma-separated list of levels + // to include. The accepted levels are verbose (5), information (4), + // warning (3), error (2), and critical (1). + Level string + + // Providers (sources) to include records from. + Provider []string +} + +// Build builds a query from the given parameters. The query is returned as a +// XML string and can be used with Subscribe function. +func (q Query) Build() (string, error) { + var errs multierror.Errors + if q.Log == "" { + errs = append(errs, fmt.Errorf("empty log name")) + } + + qp := &queryParams{Path: q.Log} + builders := []func(Query) error{ + qp.ignoreOlderSelect, + qp.eventIDSelect, + qp.levelSelect, + qp.providerSelect, + } + for _, build := range builders { + if err := build(q); err != nil { + errs = append(errs, err) + } + } + if len(errs) > 0 { + return "", errs.Err() + } + return executeTemplate(queryTemplate, qp) +} + +// queryParams are the parameters that are used to create a query from a +// template. +type queryParams struct { + Path string + Select []string + Suppress []string +} + +func (qp *queryParams) ignoreOlderSelect(q Query) error { + if q.IgnoreOlder <= 0 { + return nil + } + + ms := q.IgnoreOlder.Nanoseconds() / int64(time.Millisecond) + qp.Select = append(qp.Select, + fmt.Sprintf("TimeCreated[timediff(@SystemTime) <= %d]", ms)) + return nil +} + +func (qp *queryParams) eventIDSelect(q Query) error { + if q.EventID == "" { + return nil + } + + var includes []string + var excludes []string + components := strings.Split(q.EventID, ",") + for _, c := range components { + c = strings.TrimSpace(c) + switch { + case incEventIDRegex.MatchString(c): + includes = append(includes, fmt.Sprintf("EventID=%s", c)) + case excEventIDRegex.MatchString(c): + m := excEventIDRegex.FindStringSubmatch(c) + excludes = append(excludes, fmt.Sprintf("EventID=%s", m[1])) + case incEventIDRangeRegex.MatchString(c): + m := incEventIDRangeRegex.FindStringSubmatch(c) + r1, _ := strconv.Atoi(m[1]) + r2, _ := strconv.Atoi(m[2]) + if r1 >= r2 { + return fmt.Errorf("event ID range '%s' is invalid", c) + } + includes = append(includes, + fmt.Sprintf("(EventID >= %d and EventID <= %d)", r1, r2)) + default: + return fmt.Errorf("invalid event ID query component ('%s')", c) + } + } + + if len(includes) == 1 { + qp.Select = append(qp.Select, includes...) + } else if len(includes) > 1 { + qp.Select = append(qp.Select, "("+strings.Join(includes, " or ")+")") + } + if len(excludes) == 1 { + qp.Suppress = append(qp.Suppress, excludes...) + } else if len(excludes) > 1 { + qp.Suppress = append(qp.Suppress, "("+strings.Join(excludes, " or ")+")") + } + + return nil +} + +// levelSelect returns a xpath selector for the event Level. The returned +// selector will select events with levels less than or equal to the specified +// level. Note that level 0 is used as a catch-all/unknown level. +// +// Accepted levels: +// verbose - 5 +// information, info - 4 or 0 +// warning, warn - 3 +// error, err - 2 +// critical, crit - 1 +func (qp *queryParams) levelSelect(q Query) error { + if q.Level == "" { + return nil + } + + l := func(level int) string { return fmt.Sprintf("Level = %d", level) } + + var levelSelect []string + for _, expr := range strings.Split(q.Level, ",") { + expr = strings.TrimSpace(expr) + switch strings.ToLower(expr) { + default: + return fmt.Errorf("invalid level ('%s') for query", q.Level) + case "verbose", "5": + levelSelect = append(levelSelect, l(5)) + case "information", "info", "4": + levelSelect = append(levelSelect, l(0), l(4)) + case "warning", "warn", "3": + levelSelect = append(levelSelect, l(3)) + case "error", "err", "2": + levelSelect = append(levelSelect, l(2)) + case "critical", "crit", "1": + levelSelect = append(levelSelect, l(1)) + case "0": + levelSelect = append(levelSelect, l(0)) + } + } + + if len(levelSelect) > 0 { + qp.Select = append(qp.Select, "("+strings.Join(levelSelect, " or ")+")") + } + + return nil +} + +func (qp *queryParams) providerSelect(q Query) error { + if len(q.Provider) == 0 { + return nil + } + + var selects []string + for _, p := range q.Provider { + selects = append(selects, fmt.Sprintf("@Name='%s'", p)) + } + + qp.Select = append(qp.Select, + fmt.Sprintf("Provider[%s]", strings.Join(selects, " or "))) + return nil +} + +// executeTemplate populates a template with the given data and returns the +// value as a string. +func executeTemplate(t *template.Template, data interface{}) (string, error) { + var buf bytes.Buffer + err := t.Execute(&buf, data) + if err != nil { + return "", err + } + return buf.String(), nil +} diff --git a/winlogbeat/sys/wineventlog/query_test.go b/winlogbeat/sys/wineventlog/query_test.go new file mode 100644 index 00000000000..f13ef34659f --- /dev/null +++ b/winlogbeat/sys/wineventlog/query_test.go @@ -0,0 +1,113 @@ +// +build !integration + +package wineventlog + +import ( + "fmt" + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +func ExampleQuery() { + q, _ := Query{Log: "System", EventID: "10, 200-500, -311", Level: "info"}.Build() + fmt.Println(q) + // Output: + // + // + // *[System[(EventID=311)]] + // + // +} + +func TestIgnoreOlderQuery(t *testing.T) { + const expected = ` + + + +` + + q, err := Query{Log: "Application", IgnoreOlder: time.Hour}.Build() + if assert.NoError(t, err) { + assert.Equal(t, expected, q) + fmt.Println(q) + } +} + +func TestEventIDQuery(t *testing.T) { + const expected = ` + + + *[System[(EventID=75)]] + +` + + q, err := Query{Log: "Application", EventID: "1, 1-100, -75"}.Build() + if assert.NoError(t, err) { + assert.Equal(t, expected, q) + fmt.Println(q) + } +} + +func TestLevelQuery(t *testing.T) { + const expected = ` + + + +` + + q, err := Query{Log: "Application", Level: "Verbose"}.Build() + if assert.NoError(t, err) { + assert.Equal(t, expected, q) + fmt.Println(q) + } +} + +func TestProviderQuery(t *testing.T) { + const expected = ` + + + +` + + q, err := Query{Log: "Application", Provider: []string{"mysrc"}}.Build() + if assert.NoError(t, err) { + assert.Equal(t, expected, q) + fmt.Println(q) + } +} + +func TestCombinedQuery(t *testing.T) { + const expected = ` + + + *[System[(EventID=75)]] + +` + + q, err := Query{ + Log: "Application", + IgnoreOlder: time.Hour, + EventID: "1, 1-100, -75", + Level: "Warning", + }.Build() + if assert.NoError(t, err) { + assert.Equal(t, expected, q) + fmt.Println(q) + } +} + +func TestQueryNoParams(t *testing.T) { + const expected = ` + + + +` + + q, err := Query{Log: "Application"}.Build() + if assert.NoError(t, err) { + assert.Equal(t, expected, q) + fmt.Println(q) + } +} diff --git a/winlogbeat/sys/wineventlog/wineventlog_windows.go b/winlogbeat/sys/wineventlog/wineventlog_windows.go index 44cf47d19f2..45be1edd827 100644 --- a/winlogbeat/sys/wineventlog/wineventlog_windows.go +++ b/winlogbeat/sys/wineventlog/wineventlog_windows.go @@ -181,12 +181,29 @@ func RenderEvent( } // Ignore the error and return the original error with the response. - xml, _ = evtRenderXML(renderBuf, eventHandle) + xml, _ = RenderEventNoMessage(eventHandle, renderBuf) } return xml, err } +// RenderEventNoMessage render the events as XML but without the RenderingInfo (message). +func RenderEventNoMessage(eventHandle EvtHandle, renderBuf []byte) (string, error) { + var bufferUsed, propertyCount uint32 + err := _EvtRender(0, eventHandle, EvtRenderEventXml, uint32(len(renderBuf)), + &renderBuf[0], &bufferUsed, &propertyCount) + bufferUsed *= 2 // It returns the number of utf-16 chars. + if err == ERROR_INSUFFICIENT_BUFFER { + return "", sys.InsufficientBufferError{err, int(bufferUsed)} + } + if err != nil { + return "", err + } + + xml, _, err := sys.UTF16BytesToString(renderBuf[0:bufferUsed]) + return xml, err +} + // CreateBookmark creates a new handle to a bookmark. Close must be called on // returned EvtHandle when finished with the handle. func CreateBookmark(channel string, recordID uint64) (EvtHandle, error) { @@ -384,20 +401,3 @@ func evtRenderProviderName(renderBuf []byte, eventHandle EvtHandle) (string, err reader := bytes.NewReader(renderBuf) return readString(renderBuf, reader) } - -// evtRenderXML render the events as XML but without the RenderingInfo (message). -func evtRenderXML(renderBuf []byte, eventHandle EvtHandle) (string, error) { - var bufferUsed, propertyCount uint32 - err := _EvtRender(0, eventHandle, EvtRenderEventXml, uint32(len(renderBuf)), - &renderBuf[0], &bufferUsed, &propertyCount) - bufferUsed *= 2 // It returns the number of utf-16 chars. - if err == ERROR_INSUFFICIENT_BUFFER { - return "", sys.InsufficientBufferError{err, int(bufferUsed)} - } - if err != nil { - return "", err - } - - xml, _, err := sys.UTF16BytesToString(renderBuf[0:bufferUsed]) - return xml, err -} diff --git a/winlogbeat/tests/system/config/winlogbeat.yml.j2 b/winlogbeat/tests/system/config/winlogbeat.yml.j2 index f3ad748c980..db12a394711 100644 --- a/winlogbeat/tests/system/config/winlogbeat.yml.j2 +++ b/winlogbeat/tests/system/config/winlogbeat.yml.j2 @@ -1,33 +1,50 @@ ############################################################################### ############################# Winlogbeat ###################################### winlogbeat: - {%- if ignore_older %} - ignore_older: {{ignore_older}} - {% endif %} - {%- if event_logs %} event_logs: {% for log in event_logs -%} - name: {{ log.name }} - {%- if log.ignore_older %} + {%- if log.api is defined %} + api: {{ log.api }} + {% endif %} + {%- if log.ignore_older is defined %} ignore_older: {{ log.ignore_older }} {% endif %} - {%- if log.api %} - api: {{ log.api }} + {%- if log.event_id is defined %} + event_id: {{ log.event_id }} + {% endif %} + {%- if log.level is defined %} + level: {{ log.level }} {% endif %} - {%- if log.tags %} + {%- if log.provider %} + provider: + {% for p in log.provider -%} + - {{ p }} + {% endfor -%} + {% endif -%} + {%- if log.include_message is defined %} + include_message: {{ log.include_message }} + {% endif %} + {%- if log.include_xml is defined %} + include_xml: {{ log.include_xml }} + {% endif %} + {%- if log.tags is defined %} tags: {% for tag in log.tags -%} - {{ tag }} {% endfor -%} {% endif -%} - {%- if log.fields %} + {%- if log.fields is defined %} {% if log.fields_under_root %}fields_under_root: true{% endif %} fields: {% for k, v in log.fields.items() -%} {{ k }}: {{ v }} {% endfor -%} {% endif %} + {%- if log.invalid is defined %} + invalid: {{ log.invalid }} + {% endif %} {% endfor -%} {% endif %} diff --git a/winlogbeat/tests/system/test_config.py b/winlogbeat/tests/system/test_config.py index d581fcd59c8..825fd6f9024 100644 --- a/winlogbeat/tests/system/test_config.py +++ b/winlogbeat/tests/system/test_config.py @@ -1,3 +1,5 @@ +import sys +import unittest from winlogbeat import BaseTest """ @@ -5,12 +7,11 @@ """ +@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") class Test(BaseTest): - def test_valid_config(self): """ - With -configtest and valid config, it should return a non-zero error - code. + configtest - valid config """ self.render_config_template( ignore_older="1h", @@ -22,15 +23,42 @@ def test_valid_config(self): def test_invalid_ignore_older(self): """ - With -configtest and an error in the configuration, it should - return a non-zero error code. + configtest - invalid ignore_older units (1 hour) + """ + self.render_config_template( + event_logs=[ + {"name": "Application", "ignore_older": "1 hour"} + ] + ) + self.start_beat(extra_args=["-configtest"]).check_wait(exit_code=1) + assert self.log_contains( + "can not convert 'string' into 'duration' accessing 'ignore_older'") + # Latest ucfg update affected this error message and we lost the + # time.ParseDuration error message. + # "time: unknown unit hour in duration 1 hour") + + def test_invalid_level(self): + """ + configtest - invalid level (errors) """ self.render_config_template( - ignore_older="1 hour", event_logs=[ - {"name": "Application"} + {"name": "Application", "level": "errors"} ] ) self.start_beat(extra_args=["-configtest"]).check_wait(exit_code=1) assert self.log_contains( - "Invalid top level ignore_older value '1 hour'") + "invalid level ('errors') for query") + + def test_invalid_api(self): + """ + configtest - invalid api (file) + """ + self.render_config_template( + event_logs=[ + {"name": "Application", "api": "file"} + ] + ) + self.start_beat(extra_args=["-configtest"]).check_wait(exit_code=1) + assert self.log_contains(("Failed to create new event log. file API is " + "not available")) diff --git a/winlogbeat/tests/system/test_eventlog.py b/winlogbeat/tests/system/test_eventlog.py deleted file mode 100644 index dcc3747431d..00000000000 --- a/winlogbeat/tests/system/test_eventlog.py +++ /dev/null @@ -1,316 +0,0 @@ -import sys -import unittest -from winlogbeat import BaseTest - -if sys.platform.startswith("win"): - import win32api - import win32con - import win32evtlog - import win32security - import win32evtlogutil - -""" -Contains tests for reading from the Windows Event Log (both APIs). -""" - -class Test(BaseTest): - providerName = "WinlogbeatTestPython" - applicationName = "SystemTest" - sid = None - sidString = None - - def setUp(self): - super(Test, self).setUp() - win32evtlogutil.AddSourceToRegistry(self.applicationName, - "%systemroot%\\system32\\EventCreate.exe", - self.providerName) - - def tearDown(self): - super(Test, self).tearDown() - win32evtlogutil.RemoveSourceFromRegistry( - self.applicationName, self.providerName) - self.clear_event_log() - - def clear_event_log(self): - hlog = win32evtlog.OpenEventLog(None, self.providerName) - win32evtlog.ClearEventLog(hlog, None) - win32evtlog.CloseEventLog(hlog) - - def write_event_log(self, message, eventID=10, sid=None): - if sid == None: - sid = self.get_sid() - - level = win32evtlog.EVENTLOG_INFORMATION_TYPE - descr = [message] - - win32evtlogutil.ReportEvent(self.applicationName, eventID, - eventType=level, strings=descr, sid=sid) - - def get_sid(self): - if self.sid == None: - ph = win32api.GetCurrentProcess() - th = win32security.OpenProcessToken(ph, win32con.TOKEN_READ) - self.sid = win32security.GetTokenInformation( - th, win32security.TokenUser)[0] - - return self.sid - - def get_sid_string(self): - if self.sidString == None: - self.sidString = win32security.ConvertSidToStringSid(self.get_sid()) - - return self.sidString - - def assert_common_fields(self, evt, api, msg=None, eventID=10, sid=None, extra=None): - assert evt["computer_name"].lower() == win32api.GetComputerName().lower() - assert "record_number" in evt - self.assertDictContainsSubset({ - "count": 1, - "event_id": eventID, - "level": "Information", - "log_name": self.providerName, - "source_name": self.applicationName, - "type": api, - }, evt) - - if msg == None: - assert "message" not in evt - else: - self.assertEquals(evt["message"], msg) - self.assertDictContainsSubset({"event_data.param1": msg}, evt) - - if sid == None: - self.assertEquals(evt["user.identifier"], self.get_sid_string()) - self.assertEquals(evt["user.name"].lower(), win32api.GetUserName().lower()) - self.assertEquals(evt["user.type"], "User") - assert "user.domain" in evt - else: - self.assertEquals(evt["user.identifier"], sid) - assert "user.name" not in evt - assert "user.type" not in evt - - if extra != None: - self.assertDictContainsSubset(extra, evt) - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_eventlogging_read_one_event(self): - """ - Event Logging - Read one event - """ - self.read_one_event("eventlogging") - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_wineventlog_read_one_event(self): - """ - Win Event Log - Read one event - """ - evt = self.read_one_event("wineventlog") - self.assertDictContainsSubset({ - "keywords": ["Classic"], - "opcode": "Info", - }, evt) - - def read_one_event(self, api): - msg = "Read One Event Testcase" - self.write_event_log(msg) - - # Run Winlogbeat - self.render_config_template( - event_logs=[ - {"name": self.providerName, "api": api} - ] - ) - proc = self.start_beat() - self.wait_until(lambda: self.output_has(1)) - proc.check_kill_and_wait() - - # Verify output - events = self.read_output() - assert len(events) == 1 - evt = events[0] - self.assert_common_fields(evt, api, msg) - return evt - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_eventlogging_read_unknown_event_id(self): - """ - Event Logging - Read unknown event ID - """ - evt = self.read_unknown_event_id("eventlogging") - - assert evt["message_error"].lower() == ("The system cannot find " - "message text for message number 1111 in the message file for " - "C:\\Windows\\system32\\EventCreate.exe.").lower() - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_wineventlog_read_unknown_event_id(self): - """ - Win Event Log - Read unknown event ID - """ - evt = self.read_unknown_event_id("wineventlog") - self.assertDictContainsSubset({ - "keywords": ["Classic"], - "opcode": "Info", - }, evt) - - # No rendering error is being given. - #assert evt["message_error"] == ("the message resource is present but " - # "the message is not found in the string/message table") - - def read_unknown_event_id(self, api): - msg = "Unknown Event ID Testcase" - eventID = 1111 - self.write_event_log(msg, eventID) - - # Run Winlogbeat - self.render_config_template( - event_logs=[ - {"name": self.providerName, "api": api} - ] - ) - proc = self.start_beat() - self.wait_until(lambda: self.output_has(1)) - proc.check_kill_and_wait() - - # Verify output - events = self.read_output() - assert len(events) == 1 - evt = events[0] - self.assert_common_fields(evt, api, None, eventID=1111, extra={"event_data.param1": msg}) - return evt - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_eventlogging_read_unknown_sid(self): - """ - Event Logging - Read event with unknown SID - """ - self.read_unknown_sid("eventlogging") - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_wineventlog_read_unknown_sid(self): - """ - Win Event Log - Read event with unknown SID - """ - evt = self.read_unknown_sid("wineventlog") - self.assertDictContainsSubset({ - "keywords": ["Classic"], - "opcode": "Info", - }, evt) - - def read_unknown_sid(self, api): - # Fake SID that was made up. - accountIdentifier = "S-1-5-21-3623811015-3361044348-30300820-1013" - sid = win32security.ConvertStringSidToSid(accountIdentifier) - - msg = "Unknown SID of " + accountIdentifier - self.write_event_log(msg, sid=sid) - - # Run Winlogbeat - self.render_config_template( - event_logs=[ - {"name": self.providerName, "api": api} - ] - ) - proc = self.start_beat() - self.wait_until(lambda: self.output_has(1)) - proc.check_kill_and_wait() - - # Verify output - events = self.read_output() - assert len(events) == 1 - evt = events[0] - self.assert_common_fields(evt, api, msg, sid=accountIdentifier) - return evt - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_eventlogging_fields_under_root(self): - """ - Event Logging - Fields Under Root - """ - self.fields_under_root("eventlogging") - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_wineventlog_fields_under_root(self): - """ - Win Event Log - Fields Under Root - """ - self.fields_under_root("wineventlog") - - def fields_under_root(self, api): - msg = "Add fields under root" - self.write_event_log(msg) - - # Run Winlogbeat - self.render_config_template( - tags = ["global"], - fields = {"global": "field", "env": "prod", "type": "overwrite"}, - fields_under_root = True, - event_logs = [ - {"name": self.providerName, - "api": api, - "tags": ["local"], - "fields_under_root": True, - "fields": {"local": "field", "env": "dev"}} - ] - ) - proc = self.start_beat() - self.wait_until(lambda: self.output_has(1)) - proc.check_kill_and_wait() - - # Verify output - events = self.read_output() - self.assertEqual(len(events), 1) - evt = events[0] - self.assertDictContainsSubset({ - "global": "field", - "env": "dev", - "type": "overwrite", - "local": "field", - "tags": ["global", "local"], - }, evt) - return evt - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_eventlogging_fields_not_under_root(self): - """ - Event Logging - Fields Not Under Root - """ - self.fields_not_under_root("eventlogging") - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_wineventlog_fields_not_under_root(self): - """ - Win Event Log - Fields Not Under Root - """ - self.fields_not_under_root("wineventlog") - - def fields_not_under_root(self, api): - msg = "Add fields" - self.write_event_log(msg) - - # Run Winlogbeat - self.render_config_template( - fields = {"global": "field", "env": "prod", "type": "overwrite"}, - event_logs = [ - {"name": self.providerName, - "api": api, - "fields": {"local": "field", "env": "dev", "num": 1}} - ] - ) - proc = self.start_beat() - self.wait_until(lambda: self.output_has(1)) - proc.check_kill_and_wait() - - # Verify output - events = self.read_output() - self.assertEqual(len(events), 1) - evt = events[0] - assert "tags" not in evt, "tags present in event" - self.assertDictContainsSubset({ - "fields.global": "field", - "fields.env": "dev", - "fields.type": "overwrite", - "fields.local": "field", - "fields.num": 1, - }, evt) - return evt diff --git a/winlogbeat/tests/system/test_eventlogging.py b/winlogbeat/tests/system/test_eventlogging.py new file mode 100644 index 00000000000..b6dd02a50ef --- /dev/null +++ b/winlogbeat/tests/system/test_eventlogging.py @@ -0,0 +1,150 @@ +import sys +import time +import unittest +from winlogbeat import WriteReadTest + +if sys.platform.startswith("win"): + import win32security + +""" +Contains tests for reading from the Event Logging API (pre MS Vista). +""" + + +@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") +class Test(WriteReadTest): + @classmethod + def setUpClass(self): + self.api = "eventlogging" + super(WriteReadTest, self).setUpClass() + + def test_read_one_event(self): + """ + eventlogging - Read one classic event + """ + msg = "Hello world!" + self.write_event_log(msg) + evts = self.read_events() + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg) + + def test_read_unknown_event_id(self): + """ + eventlogging - Read unknown event ID + """ + msg = "Unknown event ID" + event_id = 1111 + self.write_event_log(msg, eventID=event_id) + evts = self.read_events() + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], eventID=event_id) + self.assertEqual(evts[0]["message_error"].lower(), + ("The system cannot find message text for message " + "number 1111 in the message file for " + "C:\\Windows\\system32\\EventCreate.exe.").lower()) + + def test_read_unknown_sid(self): + """ + eventlogging - Read event with unknown SID + """ + # Fake SID that was made up. + accountIdentifier = "S-1-5-21-3623811015-3361044348-30300820-1013" + sid = win32security.ConvertStringSidToSid(accountIdentifier) + + msg = "Unknown SID " + accountIdentifier + self.write_event_log(msg, sid=sid) + evts = self.read_events() + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, sid=accountIdentifier) + + def test_fields_under_root(self): + """ + eventlogging - Add tags and custom fields under root + """ + msg = "Add tags and fields under root" + self.write_event_log(msg) + evts = self.read_events(config={ + "tags": ["global"], + "fields": {"global": "field", "env": "prod", "level": "overwrite"}, + "fields_under_root": True, + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "tags": ["local"], + "fields_under_root": True, + "fields": {"local": "field", "env": "dev"} + } + ] + }) + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, level="overwrite", extra={ + "global": "field", + "env": "dev", + "local": "field", + "tags": ["global", "local"], + }) + + def test_fields_not_under_root(self): + """ + eventlogging - Add custom fields (not under root) + """ + msg = "Add fields (not under root)" + self.write_event_log(msg) + evts = self.read_events(config={ + "fields": {"global": "field", "env": "prod", "level": "overwrite"}, + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "fields": {"local": "field", "env": "dev", "num": 1} + } + ] + }) + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, extra={ + "fields.global": "field", + "fields.env": "dev", + "fields.level": "overwrite", + "fields.local": "field", + "fields.num": 1, + }) + self.assertTrue("tags" not in evts[0]) + + def test_ignore_older(self): + """ + eventlogging - Query by time (ignore_older than 4s) + """ + self.write_event_log(">=4 seconds old", eventID=20) + time.sleep(4) + self.write_event_log("~0 seconds old", eventID=10) + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "ignore_older": "2s" + } + ] + }, expected_events=1) + self.assertTrue(len(evts), 1) + self.assertEqual(evts[0]["event_id"], 10) + + def test_unknown_eventlog_config(self): + """ + eventlogging - Unknown config parameter + """ + self.render_config_template( + event_logs=[ + { + "name": self.providerName, + "api": self.api, + "event_id": "10, 12", + "level": "info", + "provider": ["me"], + "include_xml": True, + } + ] + ) + self.start_beat(extra_args=["-configtest"]).check_wait(exit_code=1) + assert self.log_contains("4 errors: Invalid event log key") diff --git a/winlogbeat/tests/system/test_wineventlog.py b/winlogbeat/tests/system/test_wineventlog.py new file mode 100644 index 00000000000..8ec6e5c42a7 --- /dev/null +++ b/winlogbeat/tests/system/test_wineventlog.py @@ -0,0 +1,286 @@ +import sys +import time +import unittest +from winlogbeat import WriteReadTest + +if sys.platform.startswith("win"): + import win32evtlog + import win32security + +""" +Contains tests for reading from the Windows Event Log API (MS Vista and newer). +""" + + +@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") +class Test(WriteReadTest): + @classmethod + def setUpClass(self): + self.api = "wineventlog" + super(WriteReadTest, self).setUpClass() + + def test_read_one_event(self): + """ + wineventlog - Read one classic event + """ + msg = "Hello world!" + self.write_event_log(msg) + evts = self.read_events() + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, extra={ + "keywords": ["Classic"], + "opcode": "Info", + }) + + def test_read_unknown_event_id(self): + """ + wineventlog - Read unknown event ID + """ + msg = "Unknown event ID" + event_id = 1111 + self.write_event_log(msg, eventID=event_id) + evts = self.read_events() + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], eventID=event_id, extra={ + "keywords": ["Classic"], + "opcode": "Info", + }) + # Oddly, no rendering error is being given. + self.assertTrue("message_error" not in evts[0]) + + def test_read_unknown_sid(self): + """ + wineventlog - Read event with unknown SID + """ + # Fake SID that was made up. + accountIdentifier = "S-1-5-21-3623811015-3361044348-30300820-1013" + sid = win32security.ConvertStringSidToSid(accountIdentifier) + + msg = "Unknown SID " + accountIdentifier + self.write_event_log(msg, sid=sid) + evts = self.read_events() + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, sid=accountIdentifier, extra={ + "keywords": ["Classic"], + "opcode": "Info", + }) + + def test_fields_under_root(self): + """ + wineventlog - Add tags and custom fields under root + """ + msg = "Add tags and fields under root" + self.write_event_log(msg) + evts = self.read_events(config={ + "tags": ["global"], + "fields": {"global": "field", "env": "prod", "level": "overwrite"}, + "fields_under_root": True, + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "tags": ["local"], + "fields_under_root": True, + "fields": {"local": "field", "env": "dev"} + } + ] + }) + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, level="overwrite", extra={ + "keywords": ["Classic"], + "opcode": "Info", + "global": "field", + "env": "dev", + "local": "field", + "tags": ["global", "local"], + }) + + def test_fields_not_under_root(self): + """ + wineventlog - Add custom fields (not under root) + """ + msg = "Add fields (not under root)" + self.write_event_log(msg) + evts = self.read_events(config={ + "fields": {"global": "field", "env": "prod", "level": "overwrite"}, + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "fields": {"local": "field", "env": "dev", "num": 1} + } + ] + }) + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg, extra={ + "keywords": ["Classic"], + "opcode": "Info", + "fields.global": "field", + "fields.env": "dev", + "fields.level": "overwrite", + "fields.local": "field", + "fields.num": 1, + }) + self.assertTrue("tags" not in evts[0]) + + def test_include_xml(self): + """ + wineventlog - Include raw XML event + """ + msg = "Include raw XML event" + self.write_event_log(msg) + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "include_xml": True, + } + ] + }) + self.assertTrue(len(evts), 1) + self.assert_common_fields(evts[0], msg=msg) + self.assertTrue("xml" in evts[0]) + + def test_query_event_id(self): + """ + wineventlog - Query by event IDs + """ + msg = "event_id test case" + self.write_event_log(msg, eventID=10) # Excluded + self.write_event_log(msg, eventID=50) + self.write_event_log(msg, eventID=100) + self.write_event_log(msg, eventID=150) # Excluded + self.write_event_log(msg, eventID=175) + self.write_event_log(msg, eventID=200) + evts = self.read_events(config={ + "tags": ["event_id"], + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "event_id": "50, 100-200, -150" + } + ] + }, expected_events=4) + self.assertTrue(len(evts), 4) + self.assertEqual(evts[0]["event_id"], 50) + self.assertEqual(evts[1]["event_id"], 100) + self.assertEqual(evts[2]["event_id"], 175) + self.assertEqual(evts[3]["event_id"], 200) + + def test_query_level_single(self): + """ + wineventlog - Query by level (warning) + """ + self.write_event_log("success", level=win32evtlog.EVENTLOG_SUCCESS) + self.write_event_log("error", level=win32evtlog.EVENTLOG_ERROR_TYPE) + self.write_event_log("warning", level=win32evtlog.EVENTLOG_WARNING_TYPE) + self.write_event_log("information", level=win32evtlog.EVENTLOG_INFORMATION_TYPE) + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "level": "warning" + } + ] + }) + self.assertTrue(len(evts), 1) + self.assertEqual(evts[0]["level"], "Warning") + + + def test_query_level_multiple(self): + """ + wineventlog - Query by level (error, warning) + """ + self.write_event_log("success", level=win32evtlog.EVENTLOG_SUCCESS) # Level 0, Info + self.write_event_log("error", level=win32evtlog.EVENTLOG_ERROR_TYPE) # Level 2 + self.write_event_log("warning", level=win32evtlog.EVENTLOG_WARNING_TYPE) # Level 3 + self.write_event_log("information", level=win32evtlog.EVENTLOG_INFORMATION_TYPE) # Level 4 + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "level": "error, warning" + } + ] + }, expected_events=2) + self.assertTrue(len(evts), 2) + self.assertEqual(evts[0]["level"], "Error") + self.assertEqual(evts[1]["level"], "Warning") + + def test_query_ignore_older(self): + """ + wineventlog - Query by time (ignore_older than 2s) + """ + self.write_event_log(">=2 seconds old", eventID=20) + time.sleep(2) + self.write_event_log("~0 seconds old", eventID=10) + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "ignore_older": "2s" + } + ] + }) + self.assertTrue(len(evts), 1) + self.assertEqual(evts[0]["event_id"], 10) + + def test_query_provider(self): + """ + wineventlog - Query by provider (event source) + """ + self.write_event_log("selected", source=self.otherAppName) + self.write_event_log("filtered") + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "provider": [self.otherAppName] + } + ] + }) + self.assertTrue(len(evts), 1) + self.assertEqual(evts[0]["source_name"], self.otherAppName) + + def test_query_multi_param(self): + """ + wineventlog - Query by multiple params + """ + self.write_event_log("selected", source=self.otherAppName, + eventID=556, level=win32evtlog.EVENTLOG_ERROR_TYPE) + self.write_event_log("filtered", source=self.otherAppName, eventID=556) + self.write_event_log("filtered", level=win32evtlog.EVENTLOG_WARNING_TYPE) + evts = self.read_events(config={ + "event_logs": [ + { + "name": self.providerName, + "api": self.api, + "event_id": "10-20, 30-40, -35, -18, 400-1000, -432", + "level": "warn, error", + "provider": [self.otherAppName] + } + ] + }) + self.assertTrue(len(evts), 1) + self.assertEqual(evts[0]["message"], "selected") + + def test_unknown_eventlog_config(self): + """ + wineventlog - Unknown config parameter + """ + self.render_config_template( + event_logs=[ + { + "name": self.providerName, + "api": self.api, + "invalid": "garbage"} + ] + ) + self.start_beat(extra_args=["-configtest"]).check_wait(exit_code=1) + assert self.log_contains("1 error: Invalid event log key 'invalid' found.") diff --git a/winlogbeat/tests/system/winlogbeat.py b/winlogbeat/tests/system/winlogbeat.py index b63253901a3..0e973e5c32e 100644 --- a/winlogbeat/tests/system/winlogbeat.py +++ b/winlogbeat/tests/system/winlogbeat.py @@ -1,13 +1,123 @@ import sys -sys.path.append('../../../libbeat/tests/system') +if sys.platform.startswith("win"): + import win32api + import win32con + import win32evtlog + import win32security + import win32evtlogutil +sys.path.append('../../../libbeat/tests/system') from beat.beat import TestCase class BaseTest(TestCase): - @classmethod def setUpClass(self): self.beat_name = "winlogbeat" super(BaseTest, self).setUpClass() + +class WriteReadTest(BaseTest): + providerName = "WinlogbeatTestPython" + applicationName = "SystemTest" + otherAppName = "OtherSystemTestApp" + sid = None + sidString = None + api = None + + def setUp(self): + super(WriteReadTest, self).setUp() + win32evtlogutil.AddSourceToRegistry(self.applicationName, + "%systemroot%\\system32\\EventCreate.exe", + self.providerName) + win32evtlogutil.AddSourceToRegistry(self.otherAppName, + "%systemroot%\\system32\\EventCreate.exe", + self.providerName) + + def tearDown(self): + super(WriteReadTest, self).tearDown() + win32evtlogutil.RemoveSourceFromRegistry( + self.applicationName, self.providerName) + win32evtlogutil.RemoveSourceFromRegistry( + self.otherAppName, self.providerName) + self.clear_event_log() + + def clear_event_log(self): + hlog = win32evtlog.OpenEventLog(None, self.providerName) + win32evtlog.ClearEventLog(hlog, None) + win32evtlog.CloseEventLog(hlog) + + def write_event_log(self, message, eventID=10, sid=None, + level=None, source=None): + if sid == None: + sid = self.get_sid() + if source == None: + source = self.applicationName + if level == None: + level = win32evtlog.EVENTLOG_INFORMATION_TYPE + + win32evtlogutil.ReportEvent(source, eventID, + eventType=level, strings=[message], sid=sid) + + def get_sid(self): + if self.sid == None: + ph = win32api.GetCurrentProcess() + th = win32security.OpenProcessToken(ph, win32con.TOKEN_READ) + self.sid = win32security.GetTokenInformation( + th, win32security.TokenUser)[0] + + return self.sid + + def get_sid_string(self): + if self.sidString == None: + self.sidString = win32security.ConvertSidToStringSid(self.get_sid()) + + return self.sidString + + def read_events(self, config=None, expected_events=1): + if config == None: + config = { + "event_logs": [ + {"name": self.providerName, "api": self.api} + ] + } + + self.render_config_template(**config) + proc = self.start_beat() + self.wait_until(lambda: self.output_has(expected_events)) + proc.check_kill_and_wait() + + return self.read_output() + + def assert_common_fields(self, evt, msg=None, eventID=10, sid=None, + level="Information", extra=None): + assert evt["computer_name"].lower() == win32api.GetComputerName().lower() + assert "record_number" in evt + self.assertDictContainsSubset({ + "count": 1, + "event_id": eventID, + "level": level, + "log_name": self.providerName, + "source_name": self.applicationName, + "type": self.api, + }, evt) + + if msg == None: + assert "message" not in evt + else: + self.assertEquals(evt["message"], msg) + self.assertDictContainsSubset({"event_data.param1": msg}, evt) + + if sid == None: + self.assertEquals(evt["user.identifier"], self.get_sid_string()) + self.assertEquals(evt["user.name"].lower(), + win32api.GetUserName().lower()) + self.assertEquals(evt["user.type"], "User") + assert "user.domain" in evt + else: + self.assertEquals(evt["user.identifier"], sid) + assert "user.name" not in evt + assert "user.type" not in evt + + if extra != None: + self.assertDictContainsSubset(extra, evt) diff --git a/winlogbeat/winlogbeat.yml b/winlogbeat/winlogbeat.yml index dc4134ada13..03fc7d58284 100644 --- a/winlogbeat/winlogbeat.yml +++ b/winlogbeat/winlogbeat.yml @@ -6,14 +6,17 @@ winlogbeat: # in the directory in which it was started. #registry_file: .winlogbeat.yml - # List of event logs to monitor. + # event_logs specifies a list of event logs to monitor as well as any + # accompanying options. The YAML data type of event_logs is a list of + # dictionaries. # - # Optionally, ignore_older may be specified to filter events that are older - # then the specified amount of time. If omitted then no filtering will - # occur. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h" + # The supported keys are name (required), tags, fields, fields_under_root, + # ignore_older, level, event_id, provider, and include_xml. Please visit the + # documentation for the complete details of each option. + # https://go.es.io/WinlogbeatConfig event_logs: - name: Application - ignore_older: 72h + ignore_older: 72h - name: Security - name: System From 10da1ce8c87bd7edc69f6eb9e53b19a3f18c6189 Mon Sep 17 00:00:00 2001 From: Andrew Kroh Date: Mon, 28 Mar 2016 14:25:27 -0400 Subject: [PATCH 3/3] Remove count field from Winlogbeat --- winlogbeat/docs/fields.asciidoc | 9 --------- winlogbeat/etc/fields.yml | 7 ------- winlogbeat/eventlog/eventlog.go | 1 - winlogbeat/tests/system/winlogbeat.py | 1 - 4 files changed, 18 deletions(-) diff --git a/winlogbeat/docs/fields.asciidoc b/winlogbeat/docs/fields.asciidoc index 0c859380fef..0d04660d146 100644 --- a/winlogbeat/docs/fields.asciidoc +++ b/winlogbeat/docs/fields.asciidoc @@ -50,15 +50,6 @@ The event log API type used to read the record. The possible values are "wineven The Event Logging API was designed for Windows Server 2003, Windows XP, or Windows 2000 operating systems. In Windows Vista, the event logging infrastructure was redesigned. On Windows Vista or later operating systems, the Windows Event Log API is used. Winlogbeat automatically detects which API to use for reading event logs. -==== count - -type: int - -required: True - -The number of event log records represented in the event. This field is always set to 1. - - [[exported-fields-eventlog]] === Event Log Record Fields diff --git a/winlogbeat/etc/fields.yml b/winlogbeat/etc/fields.yml index 142a607e45b..3656d79044a 100644 --- a/winlogbeat/etc/fields.yml +++ b/winlogbeat/etc/fields.yml @@ -44,13 +44,6 @@ common: systems, the Windows Event Log API is used. Winlogbeat automatically detects which API to use for reading event logs. - - name: count - type: int - required: true - description: > - The number of event log records represented in the event. This field is - always set to 1. - eventlog: type: group description: > diff --git a/winlogbeat/eventlog/eventlog.go b/winlogbeat/eventlog/eventlog.go index 19f6afe5163..3da81c4d830 100644 --- a/winlogbeat/eventlog/eventlog.go +++ b/winlogbeat/eventlog/eventlog.go @@ -57,7 +57,6 @@ func (e Record) ToMapStr() common.MapStr { m := common.MapStr{ "type": e.API, common.EventMetadataKey: e.EventMetadata, - "count": 1, "@timestamp": common.Time(e.TimeCreated.SystemTime), "log_name": e.Channel, "source_name": e.Provider.Name, diff --git a/winlogbeat/tests/system/winlogbeat.py b/winlogbeat/tests/system/winlogbeat.py index 0e973e5c32e..ed42a8c7639 100644 --- a/winlogbeat/tests/system/winlogbeat.py +++ b/winlogbeat/tests/system/winlogbeat.py @@ -94,7 +94,6 @@ def assert_common_fields(self, evt, msg=None, eventID=10, sid=None, assert evt["computer_name"].lower() == win32api.GetComputerName().lower() assert "record_number" in evt self.assertDictContainsSubset({ - "count": 1, "event_id": eventID, "level": level, "log_name": self.providerName,