[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

adriansr · 2019-01-28T14:37:51Z

This PR avoids duplicate data in documents ingested via the Suricata Eve fileset by replacing a few fields with aliases to ECS fields.

This allows to maintain the full set of fields Suricata users may expect while at the same time reducing the size of the events.

The aliased fields are:

Aliased field	Target ECS field
suricata.eve.alert.action	event.outcome
suricata.eve.alert.severity	event.severity
suricata.eve.app_proto	network.protocol
suricata.eve.dest_ip	destination.ip
suricata.eve.dest_port	destination.port
suricata.eve.event_type	event.type
suricata.eve.fileinfo.filename	file.path
suricata.eve.fileinfo.size	file.size
suricata.eve.flow.start	event.start
suricata.eve.flow.bytes_toclient	destination.bytes
suricata.eve.flow.bytes_toserver	source.bytes
suricata.eve.flow.pkts_toclient	destination.packets
suricata.eve.flow.pkts_toserver	source.packets
suricata.eve.http.hostname	url.domain
suricata.eve.http.http_method	http.request.method
suricata.eve.http.http_refer	http.request.referrer
suricata.eve.http.http_user_agent	user_agent.original
suricata.eve.http.length	http.response.body.bytes
suricata.eve.http.status	http.response.status_code
suricata.eve.proto	network.transport
suricata.eve.src_ip	source.ip
suricata.eve.src_port	source.port
suricata.eve.timestamp	`@timestamp`

Also, the following non-related fixes are performed:

"http.response.status_code" was being set from a string and not an integer.
"user_agent.original" was being inadvertedly removed by the user_agent processor.
Reformatted the ingest pipeline with standard JSON formatting.

Aliased fields don't seem to be handled in dashboards.

This field cannot be aliased to `message` as we need this data in a keyword field. There is no suitable field for that in ECS.

elasticmachine · 2019-01-28T19:26:09Z

Pinging @elastic/secops

adriansr · 2019-01-28T19:36:21Z

Note to reviewers: To review the ingest pipeline it's easier if you look at the changes added in every individual commit in the PR.

webmat

This is almost ready to go. Good work!

I've noticed one missing alias, for url.path. That's the only change required in my review.

I also have one optional suggestion to make the IN code a little more straightforward.

webmat · 2019-01-28T21:44:26Z

x-pack/filebeat/module/suricata/eve/test/eve-alerts.log-expected.json

@@ -76,6 +58,7 @@
        "user_agent.major": "7",
        "user_agent.minor": "58",
        "user_agent.name": "curl",
+        "user_agent.original": "curl/7.58.0",


Good catch!

webmat · 2019-01-28T21:49:00Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

+                    "suricata.eve.timestamp"
+                ],
+                "ignore_missing": true
+            }


For most of these fields, you could actually get rid of the convert { type: string} and do a straight field rename. Would look much more like the other modules, and would remove the need to have this huge remove operation.

You are completely right. There's a lot of cleanup that can be done

webmat · 2019-01-28T21:55:27Z

x-pack/filebeat/module/suricata/eve/test/eve-small.log-expected.json

        "suricata.eve.http.protocol": "HTTP/1.1",
-        "suricata.eve.http.status": 200,
        "suricata.eve.http.url": "/dd.xml",


suricata.eve.http.url => url.path

I'm not so sure about that one, as we will be losing information if s.e.http.url has query and/or fragment part (those go into url.query and url.fragment).

perhaps then url.original?

Yeah, in this case url.original should be used. This is for the value as observed.

Then optionally (out of scope here) the other fields could be populated to break it down (url.path, url.query, url.fragment), or rebuild full context (url.full).

Oh, I overlooked url.original. Done!

ruflin

LGTM (after addressing review from Mat). I mostly focused on the change in the golden files.

ruflin · 2019-01-29T07:34:38Z

x-pack/filebeat/module/suricata/eve/test/eve-alerts.log-expected.json

        "suricata.eve.flow_id": 2191386088856669,
-        "suricata.eve.http.hostname": "example.net",
        "suricata.eve.http.http_content_type": "text/html",


@webmat This and the one below could be something for ECS in the future.

Yeah I took a note to add content_type, when reviewing this PR, indeed :-)

What's the other field you're referring to? The field below is method LOL. And .hostname => .domain ;-)

ruflin · 2019-01-29T07:58:46Z

I initially thought these should not go into the ecs-migration.yml file but now I think it would make sense to add them there. This has two advantages:

In case you missed renaming one of the fields above in the dashboards, the migration script will catch it.
We can show a full list of files that we migrated including this out of ecs-migration.yml file.

adriansr · 2019-01-29T13:22:14Z

@ruflin I've updated the ecs-migration.yml, please have a look. I'm unsure if I used the right set of flags. Is there a way to test this migration script?

ruflin · 2019-01-29T13:33:30Z

@adriansr The migration script for the dashboards can be found here: #9998 I still have to create the script to generate a nice changelog entry.

ruflin

I'm good with getting this in as is and have follow up PR's if needed.

webmat · 2019-01-29T16:28:50Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

-                "type": "string",
-                "ignore_missing": true
-            }
-        },
        {
            "convert": {
                "field": "suricata.eve.http.http_refer",


This one's straightforward as well. Any reason why you're not renaming it?

Nope, just skipped it by mistake. Good catch

webmat

LGTM as well

One last nit about doing a straight rename for http_refer.

adriansr · 2019-01-29T16:54:07Z

jenkins, test this

adriansr added 2 commits January 28, 2019 11:05

Reformat ingest pipeline

84fbe13

Alias suricata fields to point to ECS counterparts

6d57056

adriansr added the in progress Pull request is currently in progress. label Jan 28, 2019

adriansr requested a review from a team as a code owner January 28, 2019 14:37

Update fields docs

312ac43

adriansr requested a review from a team as a code owner January 28, 2019 15:57

adriansr added 2 commits January 28, 2019 17:59

Dashboards use ECS fields

1e2af11

Aliased fields don't seem to be handled in dashboards.

Restore suricata.eve.alert.category

163013b

This field cannot be aliased to `message` as we need this data in a keyword field. There is no suitable field for that in ECS.

adriansr changed the title ~~[WIP] Alias redundant fields in the Suricata module~~ [Filebeat] Replace Suricata/Eve fields with aliases to ECS fields Jan 28, 2019

adriansr added review Filebeat Filebeat SecOps ecs and removed in progress Pull request is currently in progress. labels Jan 28, 2019

adriansr requested a review from ruflin January 28, 2019 19:26

Regenerate golden files

4476225

webmat reviewed Jan 28, 2019

View reviewed changes

ruflin approved these changes Jan 29, 2019

View reviewed changes

adriansr added 2 commits January 29, 2019 13:43

Replace convert,delete pairs with renames in pipeline

4331bf9

Add aliases to ecs-migration file

7246103

ruflin approved these changes Jan 29, 2019

View reviewed changes

Add alias for suricata.eve.http.url to url.original

08f9092

webmat reviewed Jan 29, 2019

View reviewed changes

webmat approved these changes Jan 29, 2019

View reviewed changes

Fix suricata.eve.http.http_refer

073e1de

adriansr merged commit e01e798 into elastic:master Jan 29, 2019

andrewkroh mentioned this pull request Feb 4, 2019

[Filebeat] Field aliases yield confusing Kibana Discovery results #10535

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

adriansr commented Jan 28, 2019 •

edited

Loading

elasticmachine commented Jan 28, 2019

adriansr commented Jan 28, 2019 •

edited

Loading

webmat left a comment

webmat Jan 28, 2019

webmat Jan 28, 2019

adriansr Jan 29, 2019

webmat Jan 28, 2019

adriansr Jan 29, 2019

ruflin Jan 29, 2019

webmat Jan 29, 2019

adriansr Jan 29, 2019

ruflin left a comment

ruflin Jan 29, 2019

webmat Jan 29, 2019

ruflin commented Jan 29, 2019

adriansr commented Jan 29, 2019

ruflin commented Jan 29, 2019

ruflin left a comment

webmat Jan 29, 2019

adriansr Jan 29, 2019

webmat left a comment

adriansr commented Jan 29, 2019

[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

[Filebeat] Replace Suricata/Eve fields with aliases to ECS fields #10377

Conversation

adriansr commented Jan 28, 2019 • edited Loading

elasticmachine commented Jan 28, 2019

adriansr commented Jan 28, 2019 • edited Loading

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jan 29, 2019

adriansr commented Jan 29, 2019

ruflin commented Jan 29, 2019

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

adriansr commented Jan 29, 2019

adriansr commented Jan 28, 2019 •

edited

Loading

adriansr commented Jan 28, 2019 •

edited

Loading