Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

Merged
merged 10 commits into from
Jan 29, 2019
Merged

[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
84fbe13
Reformat ingest pipeline
adriansr Jan 28, 2019
6d57056
Alias suricata fields to point to ECS counterparts
adriansr Jan 28, 2019
312ac43
Update fields docs
adriansr Jan 28, 2019
1e2af11
Dashboards use ECS fields
adriansr Jan 28, 2019
163013b
Restore suricata.eve.alert.category
adriansr Jan 28, 2019
4476225
Regenerate golden files
adriansr Jan 28, 2019
4331bf9
Replace convert,delete pairs with renames in pipeline
adriansr Jan 29, 2019
7246103
Add aliases to ecs-migration file
adriansr Jan 29, 2019
08f9092
Add alias for suricata.eve.http.url to url.original
adriansr Jan 29, 2019
073e1de
Fix suricata.eve.http.http_refer
adriansr Jan 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 48 additions & 24 deletions x-pack/filebeat/module/suricata/eve/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
Fields exported by the EVE JSON logs
fields:
- name: event_type
type: keyword
type: alias
path: event.type

- name: app_proto_orig
type: keyword
Expand Down Expand Up @@ -46,7 +47,8 @@
type: keyword

- name: filename
type: keyword
type: alias
path: file.path

- name: tx_id
type: long
Expand All @@ -67,25 +69,30 @@
type: keyword

- name: size
type: long
type: alias
path: file.size

- name: icmp_type
type: long

- name: dest_port
type: long
type: alias
path: destination.port

- name: src_port
type: long
type: alias
path: source.port

- name: proto
type: keyword
type: alias
path: network.transport

- name: pcap_cnt
type: long

- name: src_ip
type: ip
type: alias
path: source.ip

- name: dns
type: group
Expand Down Expand Up @@ -124,7 +131,8 @@
type: keyword

- name: dest_ip
type: ip
type: alias
path: destination.ip

- name: icmp_code
type: long
Expand All @@ -133,37 +141,44 @@
type: group
fields:
- name: status
type: long
type: alias
path: http.response.status_code

- name: redirect
type: keyword

- name: http_user_agent
type: keyword
type: alias
path: user_agent.original

- name: protocol
type: keyword

- name: http_refer
type: keyword
type: alias
path: http.request.referrer

- name: url
type: keyword

- name: hostname
type: keyword
type: alias
path: url.domain

- name: length
type: long
type: alias
path: http.response.body.bytes

- name: http_method
type: keyword
type: alias
path: http.request.method

- name: http_content_type
type: keyword

- name: timestamp
type: date
type: alias
path: '@timestamp'

- name: in_iface
type: keyword
Expand All @@ -172,10 +187,12 @@
type: group
fields:
- name: category
type: keyword
type: alias
path: message

- name: severity
type: long
type: alias
path: event.severity

- name: rev
type: long
Expand All @@ -187,7 +204,8 @@
type: keyword

- name: action
type: keyword
type: alias
path: event.outcome

- name: signature_id
type: long
Expand Down Expand Up @@ -655,13 +673,16 @@
type: group
fields:
- name: bytes_toclient
type: long
type: alias
path: destination.bytes

- name: start
type: date
type: alias
path: event.start

- name: pkts_toclient
type: long
type: alias
path: destination.packets

- name: age
type: long
Expand All @@ -670,13 +691,15 @@
type: keyword

- name: bytes_toserver
type: long
type: alias
path: source.bytes

- name: reason
type: keyword

- name: pkts_toserver
type: long
type: alias
path: source.packets

- name: end
type: date
Expand All @@ -685,7 +708,8 @@
type: boolean

- name: app_proto
type: keyword
type: alias
path: network.protocol

- name: tx_id
type: long
Expand Down
63 changes: 42 additions & 21 deletions x-pack/filebeat/module/suricata/eve/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
"convert": {
"field": "suricata.eve.http.status",
"target_field": "http.response.status_code",
"type": "string",
"type": "integer",
"ignore_missing": true
}
},
Expand Down Expand Up @@ -119,7 +119,7 @@
{
"convert": {
"field": "suricata.eve.http.http_user_agent",
"target_field": "user_agent.original",
"target_field": "user_agent_original",
"type": "string",
"ignore_missing": true
}
Expand All @@ -133,11 +133,6 @@
]
}
},
{
"remove": {
"field": "suricata.eve.timestamp"
}
},
{
"lowercase": {
"field": "suricata.eve.event_type",
Expand Down Expand Up @@ -217,18 +212,6 @@
"ignore_failure": true
}
},
{
"remove": {
"field": "suricata.eve.flow.start",
"ignore_missing": true
}
},
{
"remove": {
"field": "suricata.eve.flow.end",
"ignore_missing": true
}
},
{
"set": {
"field": "event.end",
Expand Down Expand Up @@ -257,8 +240,14 @@
},
{
"user_agent": {
"field": "user_agent.original",
"target_field": "user_agent",
"field": "user_agent_original",
"ignore_missing": true
}
},
{
"rename": {
"field": "user_agent_original",
"target_field": "user_agent.original",
"ignore_missing": true
}
},
Expand Down Expand Up @@ -317,6 +306,38 @@
"target_field": "destination.geo",
"ignore_missing": true
}
},
{
"remove": {
"field": [
"suricata.eve.alert.action",
"suricata.eve.alert.category",
"suricata.eve.alert.severity",
"suricata.eve.app_proto",
"suricata.eve.dest_ip",
"suricata.eve.dest_port",
"suricata.eve.event_type",
"suricata.eve.fileinfo.filename",
"suricata.eve.fileinfo.size",
"suricata.eve.flow.end",
"suricata.eve.flow.start",
"suricata.eve.flow.bytes_toclient",
"suricata.eve.flow.bytes_toserver",
"suricata.eve.flow.pkts_toclient",
"suricata.eve.flow.pkts_toserver",
"suricata.eve.http.hostname",
"suricata.eve.http.http_method",
"suricata.eve.http.http_refer",
"suricata.eve.http.http_user_agent",
"suricata.eve.http.length",
"suricata.eve.http.status",
"suricata.eve.proto",
"suricata.eve.src_ip",
"suricata.eve.src_port",
"suricata.eve.timestamp"
],
"ignore_missing": true
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For most of these fields, you could actually get rid of the convert { type: string} and do a straight field rename. Would look much more like the other modules, and would remove the need to have this huge remove operation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are completely right. There's a lot of cleanup that can be done

}
],
"on_failure": [
Expand Down
Loading