Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fileset for parsing linux auditd logs #3750

Merged
merged 4 commits into from
Apr 3, 2017
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions filebeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -865,6 +865,112 @@ Fields from the system log files.



[float]
== audit Fields

Fields from the Linux audit logs. Not all fields are documented here because they are dynamic and vary by audit event type.



[float]
=== system.audit.type

The audit event type.


[float]
=== system.audit.epoch

The unix epoch timestamp from the audit log.


[float]
=== system.audit.counter

type: long

The audit event counter.


[float]
=== system.audit.pid

type: long

The ID of the process.


[float]
=== system.audit.ppid

type: long

The ID of the process.


[float]
=== system.audit.items

type: long

The number of items in an event.


[float]
=== system.audit.item

type: long

The item field indicates which item out of the total number of items. This number is zero-based; a value of 0 means it is the first item.


[float]
== geoip Fields

Contains GeoIP information gathered based on the `system.audit.addr` field. Only present if the GeoIP Elasticsearch plugin is available and used.



[float]
=== system.audit.geoip.continent_name

type: keyword

The name of the continent.


[float]
=== system.audit.geoip.city_name

type: keyword

The name of the city.


[float]
=== system.audit.geoip.region_name

type: keyword

The name of the region.


[float]
=== system.audit.geoip.country_iso_code

type: keyword

Country ISO code.


[float]
=== system.audit.geoip.location

type: geo_point

The longitude and latitude.


[float]
== auth Fields

Expand Down
Binary file added filebeat/docs/images/kibana-system-audit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
16 changes: 12 additions & 4 deletions filebeat/docs/modules/system.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,29 @@ This file is generated! See scripts/docs_collector.py
[[filebeat-module-system]]
== System module

This module collects and parses logs created by system logging server of common Unix/Linux based
distributions.
This module collects and parses logs created by system logging server of common
Unix/Linux based distributions.

[float]
=== Compatibility

This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS Sierra, and others.
This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS
Sierra, and others.

This module is not available for Windows.

[float]
=== Dashboard

This module comes with a sample dashboard.
This module comes with sample dashboards. The first is a more generic syslog
dashboard that shows syslog data.

image::./images/kibana-system.png[]

And second is the audit dashboard that shows audit log data.

image::./images/kibana-system-audit.png[]

[float]
=== Syslog fileset settings

Expand Down
8 changes: 4 additions & 4 deletions filebeat/fileset/modules.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,15 @@ func newModuleRegistry(modulesPath string,
fcfg = &defaultFilesetConfig
}

if fcfg.Enabled != nil && (*fcfg.Enabled) == false {
continue
}

fcfg, err = applyOverrides(fcfg, mcfg.Module, filesetName, overrides)
if err != nil {
return nil, fmt.Errorf("Error applying overrides on fileset %s/%s: %v", mcfg.Module, filesetName, err)
}

if fcfg.Enabled != nil && (*fcfg.Enabled) == false {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific reason this was moved?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This applies the overrides first, which allows an override to disable a fileset. It allows a CLI config like:

-M system.audit.enabled=false

continue
}

fileset, err := New(modulesPath, filesetName, &mcfg, fcfg)
if err != nil {
return nil, err
Expand Down
4 changes: 2 additions & 2 deletions filebeat/fileset/modules_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func TestNewModuleRegistry(t *testing.T) {
expectedModules := map[string][]string{
"nginx": {"access", "error"},
"mysql": {"slowlog", "error"},
"system": {"syslog", "auth"},
"system": {"syslog", "audit", "auth"},
}

assert.Equal(t, len(expectedModules), len(reg.registry))
Expand Down Expand Up @@ -98,7 +98,7 @@ func TestNewModuleRegistryConfig(t *testing.T) {
assert.NotContains(t, reg.registry["nginx"], "error")
}

func TestAppplyOverrides(t *testing.T) {
func TestApplyOverrides(t *testing.T) {

falseVar := false
trueVar := true
Expand Down
16 changes: 12 additions & 4 deletions filebeat/module/system/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -1,20 +1,28 @@
== System module

This module collects and parses logs created by system logging server of common Unix/Linux based
distributions.
This module collects and parses logs created by system logging server of common
Unix/Linux based distributions.

[float]
=== Compatibility

This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS Sierra, and others.
This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS
Sierra, and others.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "and others" might come back to us as an issue in support, maybe it's better to leave it out.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, probably a good idea to drop that. Will do.


This module is not available for Windows.

[float]
=== Dashboard

This module comes with a sample dashboard.
This module comes with sample dashboards. The first is a more generic syslog
dashboard that shows syslog data.

image::./images/kibana-system.png[]

And second is the audit dashboard that shows audit log data.

image::./images/kibana-system-audit.png[]

[float]
=== Syslog fileset settings

Expand Down
60 changes: 60 additions & 0 deletions filebeat/module/system/audit/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
- name: audit
type: group
description: >
Fields from the Linux audit logs. Not all fields are documented here because
they are dynamic and vary by audit event type.
fields:
- name: type
description: >
The audit event type.
- name: epoch
description: >
The unix epoch timestamp from the audit log.
- name: counter
type: long
description: >
The audit event counter.
- name: pid
type: long
description: >
The ID of the process.
- name: ppid
type: long
description: >
The ID of the process.
- name: items
type: long
description: >
The number of items in an event.
- name: item
type: long
description: >
The item field indicates which item out of the total number of items.
This number is zero-based; a value of 0 means it is the first item.
- name: geoip
type: group
description: >
Contains GeoIP information gathered based on the `system.audit.addr`
field. Only present if the GeoIP Elasticsearch plugin is available and
used.
fields:
- name: continent_name
type: keyword
description: >
The name of the continent.
- name: city_name
type: keyword
description: >
The name of the city.
- name: region_name
type: keyword
description: >
The name of the region.
- name: country_iso_code
type: keyword
description: >
Country ISO code.
- name: location
type: geo_point
description: >
The longitude and latitude.
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"hits": 0,
"timeRestore": false,
"description": "",
"title": "Filebeat System Audit",
"uiStateJSON": "{\"P-2\":{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}}",
"panelsJSON": "[{\"col\":1,\"id\":\"6295bdd0-0a0e-11e7-825f-6748cda7d858\",\"panelIndex\":1,\"row\":1,\"size_x\":4,\"size_y\":4,\"type\":\"visualization\"},{\"col\":9,\"id\":\"5ebdbe50-0a0f-11e7-825f-6748cda7d858\",\"panelIndex\":2,\"row\":1,\"size_x\":4,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"2bb0fa70-0a11-11e7-9e84-43da493ad0c7\",\"panelIndex\":3,\"row\":5,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"columns\":[\"message\"],\"id\":\"4ac0a370-0a11-11e7-8b04-eb22a5669f27\",\"panelIndex\":4,\"row\":8,\"size_x\":12,\"size_y\":3,\"sort\":[\"@timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":7,\"id\":\"d1726930-0a7f-11e7-8b04-eb22a5669f27\",\"panelIndex\":5,\"row\":5,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"size_x\":4,\"size_y\":4,\"panelIndex\":6,\"type\":\"visualization\",\"id\":\"c5411910-0a87-11e7-8b04-eb22a5669f27\",\"col\":5,\"row\":1}]",
"optionsJSON": "{\"darkTheme\":false}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}],\"highlightAll\":true,\"version\":true}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"sort": [
"@timestamp",
"desc"
],
"hits": 0,
"description": "",
"title": "Audit Events",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"highlightAll\":true,\"version\":true,\"query\":{\"query_string\":{\"query\":\"_exists_:system.audit\",\"analyze_wildcard\":true}},\"filter\":[]}"
},
"columns": [
"message"
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"visState": "{\"type\":\"timelion\",\"title\":\"Audit Event Results\",\"params\":{\"expression\":\".es(q=\\\"_exists_:system.audit NOT system.audit.res:failure\\\").label(\\\"Success\\\") .es(q=\\\"system.audit.res:failed\\\").label(\\\"Failure\\\").title(\\\"Audit Event Results\\\")\",\"interval\":\"auto\"}}",
"description": "",
"title": "Audit Event Results",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"visState": "{\"title\":\"Audit Top Exec Commands\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"system.audit.a0\",\"size\":30,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Command (arg 0)\"}}],\"listeners\":{}}",
"description": "",
"title": "Audit Top Exec Commands",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"query_string\":{\"query\":\"system.audit.type:EXECVE\",\"analyze_wildcard\":true}},\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"visState": "{\"title\":\"Audit Event Types\",\"type\":\"pie\",\"params\":{\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"system.audit.type\",\"size\":50,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}",
"description": "",
"title": "Audit Event Types",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"visState": "{\"title\":\"Audit Event Account Tag Cloud\",\"type\":\"tagcloud\",\"params\":{\"scale\":\"linear\",\"orientation\":\"single\",\"minFontSize\":15,\"maxFontSize\":42,\"hideLabel\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"system.audit.acct\",\"size\":15,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}",
"description": "",
"title": "Audit Event Account Tag Cloud",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"visState": "{\"title\":\"Audit Event Address Geo Location\",\"type\":\"tile_map\",\"params\":{\"mapType\":\"Scaled Circle Markers\",\"isDesaturated\":true,\"addTooltip\":true,\"heatMaxZoom\":16,\"heatMinOpacity\":0.1,\"heatRadius\":25,\"heatBlur\":15,\"heatNormalizeData\":true,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\":[15,5],\"wms\":{\"enabled\":false,\"url\":\"https://basemap.nationalmap.gov/arcgis/services/USGSTopo/MapServer/WMSServer\",\"options\":{\"version\":\"1.3.0\",\"layers\":\"0\",\"format\":\"image/png\",\"transparent\":true,\"attribution\":\"Maps provided by USGS\",\"styles\":\"\"}}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"schema\":\"segment\",\"params\":{\"field\":\"system.audit.geoip.location\",\"autoPrecision\":true,\"precision\":2}}],\"listeners\":{}}",
"description": "",
"title": "Audit Event Address Geo Location",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
}
}
6 changes: 6 additions & 0 deletions filebeat/module/system/audit/config/audit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
input_type: log
paths:
{{ range $i, $path := .paths }}
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
Loading