Skip to content

Commit

Permalink
Disable host fields for "cloud", panw, cef modules (elastic#18223)
Browse files Browse the repository at this point in the history
This changes the default configuration of Filebeat to not add `host` fields to events that
originated in other places. The `host` field is defined in ECS as "host on which the event happened"
but for data pulled from cloud APIs for data forwarded to Filebeat from other sources (PANW, CEF)
this `host` field is inaccurate.

The affected "cloud" modules are azure, aws, googlecloud, o365, and okta. By default they will
tag events with `forwarded`. This will cause the module to not add `host.name` at the input state. And then
the default configuration for Filebeat was updated to add a `when` condition to the `add_host_metadata`
processors to skip events containing the `forwarded` tag.

For PANW and CEF when data is forwarded to Filebeat from another host/device (this is most of the time)
you don't want Filebeat to add `host`. So by default this modules add a `forwarded` tag to events. If you configure the module to not include the `forwarded` tag
(e.g. `var.tags: [my_tag]`) then Filebeat will add the `host.*` fields.

And for PANW I added some additional static `observer.*` fields.

Relates: elastic#13920
  • Loading branch information
andrewkroh authored May 15, 2020
1 parent 3a73e88 commit e990740
Show file tree
Hide file tree
Showing 126 changed files with 3,948 additions and 566 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. {issue}16025[16025] {pull}17910[17910]
- Improve ECS categorization field mappings for nginx module. http.request.referrer is now lowercase & http.request.referrer only populated when nginx sets a value {issue}16174[16174] {pull}17844[17844]
- Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . {issue}16180[16180] {pull}17982[17982]
- With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta)
will no longer send the `host` field that contains information about the host Filebeat is
running on. This is because the `host` field specifies the host on which the event
happened. {issue}13920[13920] {pull}18223[18223]
- With the default configuration the cef and panw modules will no longer send the `host`
field. You can revert this change by configuring tags for the module and omitting
`forwarded` from the list. {issue}13920[13920] {pull}18223[18223]

*Heartbeat*

Expand Down Expand Up @@ -325,6 +332,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve ECS categorization field mappings in system module. {issue}16031[16031] {pull}18065[18065]
- Change the `json.*` input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
- Improve ECS categorization field mappings in osquery module. {issue}16176[16176] {pull}17881[17881]
- Added `observer.vendor`, `observer.product`, and `observer.type` to PANW module events. {pull}18223[18223]
- The `logstash` module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. {issue}9964[9964] {pull}18095[18095]

*Heartbeat*
Expand Down
7 changes: 7 additions & 0 deletions filebeat/_meta/config/processors.yml.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{{header "Processors"}}
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
6 changes: 6 additions & 0 deletions filebeat/docs/modules/cef.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ The UDP port to listen for syslog traffic. Defaults to `9003`

NOTE: Ports below 1024 require Filebeat to run as root.

*`var.tags`*::

A list of tags to include in events. Including `forwarded` indicates that the
events did not originate on this host and causes `host.name` to not be added to
events. Defaults to `[cef, forwarded]`.

[float]
==== Forcepoint NGFW Security Management Center

Expand Down
7 changes: 2 additions & 5 deletions filebeat/filebeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,16 +172,13 @@ output.elasticsearch:
#ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
- add_host_metadata: ~
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
Expand Down
13 changes: 13 additions & 0 deletions filebeat/fileset/fileset.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"io/ioutil"
"os"
"path/filepath"
"reflect"
"runtime"
"strings"
"text/template"
Expand Down Expand Up @@ -290,6 +291,18 @@ func getTemplateFunctions(vars map[string]interface{}) (template.FuncMap, error)
}

return template.FuncMap{
"inList": func(collection []interface{}, item string) bool {
for _, h := range collection {
if reflect.DeepEqual(item, h) {
return true
}
}
return false
},
"tojson": func(v interface{}) (string, error) {
bytes, err := json.Marshal(v)
return string(bytes), err
},
"IngestPipeline": func(shortID string) string {
return formatPipelineID(
builtinVars["prefix"].(string),
Expand Down
3 changes: 2 additions & 1 deletion filebeat/fileset/fileset_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,7 @@ func TestGetTemplateFunctions(t *testing.T) {
templateFunctions, err := getTemplateFunctions(vars)
assert.NoError(t, err)
assert.IsType(t, template.FuncMap{}, templateFunctions)
assert.Len(t, templateFunctions, 1)
assert.Contains(t, templateFunctions, "inList")
assert.Contains(t, templateFunctions, "tojson")
assert.Contains(t, templateFunctions, "IngestPipeline")
}
7 changes: 2 additions & 5 deletions x-pack/filebeat/filebeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,16 +172,13 @@ output.elasticsearch:
#ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
- add_host_metadata: ~
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
Expand Down
3 changes: 3 additions & 0 deletions x-pack/filebeat/module/aws/cloudtrail/config/file.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
- add_fields:
target: ''
Expand Down
3 changes: 3 additions & 0 deletions x-pack/filebeat/module/aws/cloudtrail/config/s3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ session_token: {{ .session_token }}
role_arn: {{ .role_arn }}
{{ end }}

tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
- add_fields:
target: ''
Expand Down
2 changes: 2 additions & 0 deletions x-pack/filebeat/module/aws/cloudtrail/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ var:
- name: secret_access_key
- name: session_token
- name: role_arn
- name: tags
default: [forwarded]

ingest_pipeline: ingest/pipeline.yml
input: config/{{.input}}.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@
"source.geo.region_iso_code": "CN-CQ",
"source.geo.region_name": "Chongqing",
"source.ip": "123.145.67.89",
"tags": [
"forwarded"
],
"user.id": "AROAIN5ATK5U7KEXAMPLE:JohnRole1",
"user_agent.device.name": "Spider",
"user_agent.name": "aws-cli",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down Expand Up @@ -58,6 +61,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
"service.type": "aws",
"source.address": "192.0.2.110",
"source.ip": "192.0.2.110",
"tags": [
"forwarded"
],
"user.id": "AIDACKCEVSQ6C2EXAMPLE",
"user.name": "JohnDoe",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -66,6 +69,9 @@
"service.type": "aws",
"source.address": "192.0.2.100",
"source.ip": "192.0.2.100",
"tags": [
"forwarded"
],
"user.id": "AIDACKCEVSQ6C2EXAMPLE",
"user.name": "JaneDoe",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -111,6 +117,9 @@
"service.type": "aws",
"source.address": "192.0.2.100",
"source.ip": "192.0.2.100",
"tags": [
"forwarded"
],
"user.id": "AROAIDPPEZS35WEXAMPLE:AssumedRoleSessionName",
"user.name": "RoleToBeAssumed",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -63,6 +66,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
"source.geo.region_iso_code": "US-VA",
"source.geo.region_name": "Virginia",
"source.ip": "72.21.198.64",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
"service.type": "aws",
"source.address": "192.0.2.1",
"source.ip": "192.0.2.1",
"tags": [
"forwarded"
],
"user.id": "AIDAQRSTUVWXYZEXAMPLE:devdsk",
"user_agent.device.name": "Spider",
"user_agent.name": "aws-cli",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -62,6 +65,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_PRINCIPLE",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Loading

0 comments on commit e990740

Please sign in to comment.