Convert Filebeat kibana.log to ECS #9301

webmat · 2018-11-29T20:34:38Z

Caveats

One thing I think should be addressed by this PR:

I think we should append kibana.log.tags to tags. However what's the best way to approach it? append expects a JSON array of values or field names, not a field name that contains an array (see doc example)
- Decision: Keeping under kibana. field set.

Actual caveats / disclaimers

Current Kibana module populates service.name with an array of one service name. Not sure what the purpose is for this. This seems like a minor annoyance, and not sure if stack monitoring depends on this. Not touching this.
This module populates http.response.elapsed_time which is not in ECS. Noted in [WIP] Filebeat module issues found during ECS conversion #9208
http.response.content_length needs to migrate after Update the HTTP field set with ECS definitions as of beta 2 #9645 (noted in WIP Filebeat modules adjustments for ECS Beta 2 #9684)
filebeat/_meta/fields.common.yml defines http.response.content_length which should become an alias to http.response.body.size after Update the HTTP field set with ECS definitions as of beta 2 #9645 (may impact many modules). This has been added to the issue WIP Filebeat modules adjustments for ECS Beta 2 #9684.

Not previously done by this module. I say we should not add this now, as it adds dependencies:

Perform user agent parsing
Perform geoip on source.ip

Renames

kibana.log.meta.req.headers.referer => http.request.referrer
kibana.log.meta.req.headers.user-agent => user_agent.original
kibana.log.meta.req.remoteAddress => source.address
kibana.log.meta.req.url => url.original

Duplicate fields

They have been removed, are aliased as well

kibana.log.meta.req.referer => http.request.referrer
kibana.log.meta.statusCode => http.response.status_code (ECS field was already populated)
kibana.log.meta.method => http.request.method

TODO

Populate event.duration based on http.response.elapsed_time
Alias renamed fields to their ECS counterpart
Document field migrations in ecs-migration.yml
Changelog
Finalize once Painless / module failing with: "Too many dynamic script compilations within, max" #9600 is fixed
Figure out why the -expected.json file now only contains 1 event. I should contain 3.

ruflin · 2018-12-03T09:13:21Z

Caveats:

service.name: I remember it has to do on how we add service.name. From a lucene perspective it's the same so not need to change. But there is an issue. This should become service.type as service.name can be a user configured name.
We should open an issue in the KB repo to start a discussion around ECS logs
Missing fields: Let's open track this during the discussion related to http fields

webmat · 2018-12-20T21:52:23Z

@ruflin Apart from one thing that I wonder about (kibana.log.tags >> tags?), this should be pretty much ready to go.

webmat · 2018-12-21T01:20:09Z

jenkins, test this

ruflin

If I understand this correctly, the fields which were removed and were duplicates, still have an alias so old stuff would keep working?

ruflin · 2018-12-21T11:49:56Z

@webmat I'm good with keep kibana.logs.tags as I think these are tags specific to kibana.

webmat · 2018-12-21T14:02:11Z

@ruflin This is correct. Some fields were present twice in the output format, under different names. Now they are present only once, in line with ECS, and all the old names are aliases to the ECS fields.

webmat · 2018-12-21T14:05:35Z

@ruflin One more caveat I'd like your opinion on: service.name, we leave as an array of one?

ruflin · 2018-12-21T14:21:02Z

@webmat Yes, for the array. It does not make a difference. Can explain an other time why it's like.

After we done all the modules, will go through an rename service.name to service.type anyways.

webmat · 2018-12-21T17:29:00Z

@ruflin Rebasing. Good to go, once tests are green?

webmat · 2019-01-03T20:56:42Z

@ruflin You had approved this before the holidays, I tweaked one small detail. Instead of renaming source.address to source.ip, I copy it (keeping source.address).

Can I get one last look, please? Thanks

webmat · 2019-01-07T14:52:54Z

Ping @elastic/stack-monitoring for awareness. If this works for you as well, I'd like to merge on Tuesday at the latest.

webmat · 2019-01-08T15:28:43Z

@ycombinator Hey could you have a quick look at this one as well, when you have a moment, please? Thanks!

ycombinator · 2019-01-09T20:19:15Z

filebeat/module/kibana/log/ingest/pipeline.json

+                "lang": "painless",
+                "source": "ctx.event.duration = Math.round(ctx.kibana.log.meta.res.responseTime * params.scale)",
+                "params": { "scale": 1000000 },
+                "if": "ctx.kibana.log.containsKey('meta') && ctx.kibana.log.meta.containsKey('res') && ctx.kibana.log.meta.res.containsKey('responseTime')"


Nit: You could probably shorten this to something like: ctx.kibana.log.meta?.res?.responseTime != null.

Oh man, Painless has that? Thanks for pointing out ❤️

Yep, it was added at some point: https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-operators-reference.html#null-safe-operator

@ycombinator Do you know which version this was added? Asking for compatibility reasons.

5.1.1: elastic/elasticsearch#21239

Ah, then we are good :-)

ycombinator

LGTM. Thanks for explaining the various changes to me off-PR. Left a nit comment that can feel free to ignore.

ruflin · 2019-01-10T08:05:42Z

jenkins, test this

- kibana.log.meta.req.headers.referer => http.request.referrer - kibana.log.meta.req.headers.user-agent => user_agent.original - kibana.log.meta.req.remoteAddress => source.ip - kibana.log.meta.req.url => url.original

- kibana.log.meta.req.referer - kibana.log.meta.statusCode - kibana.log.meta.method

webmat · 2019-01-10T18:52:26Z

Rebased and updated the pipeline to use the null-safe operators. Thanks @ycombinator for pointing those out, this will make my life much easier in places like this experimental PR: #9905 😆

Tests were green across the board. Will merge tomorrow AM if it's still the case.

ruflin mentioned this pull request Nov 29, 2018

Transition Beats to ECS #8655

Closed

webmat self-assigned this Nov 29, 2018

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 29, 2018

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

webmat force-pushed the ecs-kibana-fb branch from e3cda26 to b8325bb Compare December 20, 2018 21:34

webmat requested a review from a team as a code owner December 20, 2018 21:34

webmat added review and removed in progress Pull request is currently in progress. labels Dec 20, 2018

webmat changed the title ~~WIP Convert Filebeat kibana.log to ECS~~ Convert Filebeat kibana.log to ECS Dec 20, 2018

webmat requested a review from ruflin December 20, 2018 21:38

webmat force-pushed the ecs-kibana-fb branch from b8325bb to ee72842 Compare December 21, 2018 01:59

ruflin reviewed Dec 21, 2018

View reviewed changes

webmat force-pushed the ecs-kibana-fb branch from ee72842 to eb880f3 Compare December 21, 2018 17:29

webmat added the blocked label Dec 21, 2018

ruflin approved these changes Dec 24, 2018

View reviewed changes

webmat force-pushed the ecs-kibana-fb branch from eb880f3 to 3502d12 Compare January 3, 2019 18:56

webmat requested review from a team as code owners January 3, 2019 18:56

urso removed the request for review from a team January 4, 2019 01:57

webmat removed the blocked label Jan 4, 2019

ruflin approved these changes Jan 7, 2019

View reviewed changes

webmat force-pushed the ecs-kibana-fb branch from 69f679c to 14a1e4e Compare January 8, 2019 15:32

ycombinator reviewed Jan 9, 2019

View reviewed changes

ycombinator approved these changes Jan 9, 2019

View reviewed changes

Mathieu Martin added 12 commits January 10, 2019 13:40

Perform straightforward ECS renames:

971f065

- kibana.log.meta.req.headers.referer => http.request.referrer - kibana.log.meta.req.headers.user-agent => user_agent.original - kibana.log.meta.req.remoteAddress => source.ip - kibana.log.meta.req.url => url.original

Remove fields that were straight duplicates (will still be aliased):

722c830

- kibana.log.meta.req.referer - kibana.log.meta.statusCode - kibana.log.meta.method

Uppercase http.request.method field

d66e681

Compute event.duration

89a171f

Alias transitioned fields

a3f15c4

Document transitioned fields in ecs-migration.yml

5edc9eb

Fix a typo in one of the transitioned fields

50cb96b

Changelog

2b58151

Don't rename source.address, but copy it to source.ip

4f76e40

Only copy source.address to source.ip if address is present

9a8aa8a

Regenerate expected file. Was missing the recent event.dataset fix

b7592a6

Use the very nice null-safe operators to shorten if clauses

dfd5670

webmat force-pushed the ecs-kibana-fb branch from 1376e78 to dfd5670 Compare January 10, 2019 18:46

webmat removed the review label Jan 11, 2019

webmat merged commit 6c1d73b into elastic:master Jan 11, 2019

webmat deleted the ecs-kibana-fb branch January 11, 2019 02:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Filebeat kibana.log to ECS #9301

Convert Filebeat kibana.log to ECS #9301

webmat commented Nov 29, 2018 •

edited

Loading

ruflin commented Dec 3, 2018

webmat commented Dec 20, 2018

webmat commented Dec 21, 2018

ruflin left a comment

ruflin commented Dec 21, 2018

webmat commented Dec 21, 2018 •

edited

Loading

webmat commented Dec 21, 2018 •

edited

Loading

ruflin commented Dec 21, 2018

webmat commented Dec 21, 2018

webmat commented Jan 3, 2019

webmat commented Jan 7, 2019

webmat commented Jan 8, 2019

ycombinator Jan 9, 2019

webmat Jan 9, 2019

ycombinator Jan 9, 2019

ruflin Jan 10, 2019

ycombinator Jan 10, 2019

ruflin Jan 10, 2019

ycombinator left a comment

ruflin commented Jan 10, 2019

webmat commented Jan 10, 2019

Convert Filebeat kibana.log to ECS #9301

Convert Filebeat kibana.log to ECS #9301

Conversation

webmat commented Nov 29, 2018 • edited Loading

ruflin commented Dec 3, 2018

webmat commented Dec 20, 2018

webmat commented Dec 21, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin commented Dec 21, 2018

webmat commented Dec 21, 2018 • edited Loading

webmat commented Dec 21, 2018 • edited Loading

ruflin commented Dec 21, 2018

webmat commented Dec 21, 2018

webmat commented Jan 3, 2019

webmat commented Jan 7, 2019

webmat commented Jan 8, 2019

ycombinator Jan 9, 2019

Choose a reason for hiding this comment

webmat Jan 9, 2019

Choose a reason for hiding this comment

ycombinator Jan 9, 2019

Choose a reason for hiding this comment

ruflin Jan 10, 2019

Choose a reason for hiding this comment

ycombinator Jan 10, 2019

Choose a reason for hiding this comment

ruflin Jan 10, 2019

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

ruflin commented Jan 10, 2019

webmat commented Jan 10, 2019

webmat commented Nov 29, 2018 •

edited

Loading

webmat commented Dec 21, 2018 •

edited

Loading

webmat commented Dec 21, 2018 •

edited

Loading