Convert Filebeat icinga.* to ECS #9294

webmat · 2018-11-29T19:06:50Z

Caveats

ECS doesn't have support for "facility" yet. Not sure if we should mix Icinga and Syslog facilities in any case. So haven't touched this field
This module doesn't do deep parsing of the Icinga messages. There are interesting bits (e.g. PID, execution_time) that may eventually be interesting to extract out. This is out of scope for this initial mapping to ECS, however.
Not copying original message to log.original, as this is out of scope for this mapping. Will be addressed with an optional toggle at the Beats level.

Renames

icinga.debug.message => message
icinga.main.message => message
icinga.startup.message => message
icinga.debug.severity => log.level
icinga.main.severity => log.level
icinga.startup.severity => log.level

TODO

Commit missing updates to the -expected.json file
Alias renamed fields to their ECS counterpart
Document field migrations in ecs-migration.yml
Changelog

webmat · 2018-11-30T16:22:25Z

jenkins, test this bro

ruflin · 2018-12-03T09:14:17Z

Caveats: Agree to "ignore" all of them for now.

webmat · 2018-12-19T18:13:08Z

jenkins, test this

webmat · 2018-12-19T21:25:32Z

Ready for final review

webmat · 2018-12-20T02:08:03Z

jenkins, test this

dev-tools/ecs-migration.yml

filebeat/module/icinga/debug/_meta/fields.yml

webmat · 2018-12-20T15:05:02Z

Timeouts on ML tests. Will restart the filebeat tests, should be unrelated to my very simple changes.

Test modules list command ... ok (0.0913s)
======================================================================
ERROR: Test ML are installed in all possible ways [with setup_flag=True, modules_flag=False]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/go/src/github.com/elastic/beats/filebeat/build/python-env/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 46, in test_ml_setup
    self._run_ml_test(setup_flag, modules_flag)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 136, in _run_ml_test
    max_timeout=60)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 363, in wait_log_contains
    name=name)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 340, in wait_until
    "Waited {} seconds.".format(max_timeout))
TimeoutError: Timeout waiting for 'log_contains' to be true. Waited 60 seconds.
-------------------- >> begin captured stdout << ---------------------
Using elasticsearch: http://elasticsearch:9200
Test setup_flag: True, modules_flag: False
--------------------- >> end captured stdout << ----------------------
======================================================================
ERROR: Test ML are installed in all possible ways [with setup_flag=True, modules_flag=True]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/go/src/github.com/elastic/beats/filebeat/build/python-env/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 46, in test_ml_setup
    self._run_ml_test(setup_flag, modules_flag)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 136, in _run_ml_test
    max_timeout=60)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 363, in wait_log_contains
    name=name)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 340, in wait_until
    "Waited {} seconds.".format(max_timeout))
TimeoutError: Timeout waiting for 'log_contains' to be true. Waited 60 seconds.
-------------------- >> begin captured stdout << ---------------------
Using elasticsearch: http://elasticsearch:9200
License already enabled
Test setup_flag: True, modules_flag: True
--------------------- >> end captured stdout << ----------------------
======================================================================
ERROR: Test ML are installed in all possible ways [with setup_flag=False, modules_flag=False]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/go/src/github.com/elastic/beats/filebeat/build/python-env/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 46, in test_ml_setup
    self._run_ml_test(setup_flag, modules_flag)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 136, in _run_ml_test
    max_timeout=60)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 363, in wait_log_contains
    name=name)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 340, in wait_until
    "Waited {} seconds.".format(max_timeout))
TimeoutError: Timeout waiting for 'log_contains' to be true. Waited 60 seconds.
-------------------- >> begin captured stdout << ---------------------
Using elasticsearch: http://elasticsearch:9200
License already enabled
Test setup_flag: False, modules_flag: False
--------------------- >> end captured stdout << ----------------------
======================================================================
ERROR: Test ML are installed in all possible ways [with setup_flag=False, modules_flag=True]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/go/src/github.com/elastic/beats/filebeat/build/python-env/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 392, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 46, in test_ml_setup
    self._run_ml_test(setup_flag, modules_flag)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_ml.py", line 136, in _run_ml_test
    max_timeout=60)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 363, in wait_log_contains
    name=name)
  File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 340, in wait_until
    "Waited {} seconds.".format(max_timeout))
TimeoutError: Timeout waiting for 'log_contains' to be true. Waited 60 seconds.
-------------------- >> begin captured stdout << ---------------------
Using elasticsearch: http://elasticsearch:9200
License already enabled
Test setup_flag: False, modules_flag: True
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
XML: /go/src/github.com/elastic/beats/filebeat/build/TEST-system.xml
[error] 15.34% test_ml.Test.test_ml_setup_0: 92.3972s
[success] 11.59% test_autodiscover.TestAutodiscover.test_docker: 69.8451s
[error] 9.97% test_ml.Test.test_ml_setup_1: 60.0760s
[error] 9.97% test_ml.Test.test_ml_setup_2: 60.0727s
[error] 9.97% test_ml.Test.test_ml_setup_3: 60.0460s
[success] 1.56% test_registrar.Test.test_restart_state: 9.4150s

ruflin · 2018-12-20T15:09:41Z

It seems something changed on master in ML, ignore them for now: #9718

webmat · 2018-12-20T15:34:01Z

@ruflin I'll wait for this fix to the ML tests to be in and rebase, since my PRs are specifically Filebeat modules. I want the full test run.

However can you give the changes one last look and approve the PR?

- icinga.debug.message => message - icinga.debug.severity => log.level

ruflin

LGTM. No need to wait on the ml change, it does not affect this build. But it seems like you already rebased.

ruflin · 2018-12-20T16:15:18Z

filebeat/module/icinga/main/test/test.log-expected.json

        "input.type": "log", 
-        "log.offset": 0
+        "log.level": "information", 


we should probably also standardise these in ECS so it's always "INFO". Nothing todo here, just a general comment.

Yes there's already been a comprehensive discussion on normalizing log levels here elastic/ecs#129

ruflin mentioned this pull request Nov 29, 2018

Transition Beats to ECS #8655

Closed

webmat self-assigned this Nov 29, 2018

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 29, 2018

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

webmat force-pushed the ecs-fb-icinga branch from dc19418 to 8294f19 Compare December 18, 2018 21:49

webmat requested a review from a team as a code owner December 19, 2018 16:59

webmat added review and removed in progress Pull request is currently in progress. labels Dec 19, 2018

webmat force-pushed the ecs-fb-icinga branch from 11709ed to 30a4f69 Compare December 19, 2018 17:07

webmat changed the title ~~WIP Convert Filebeat icinga.* to ECS~~ Convert Filebeat icinga.* to ECS Dec 19, 2018

webmat force-pushed the ecs-fb-icinga branch from 30a4f69 to 88f03b0 Compare December 19, 2018 20:59

webmat requested a review from ruflin December 19, 2018 21:00

ruflin reviewed Dec 20, 2018

View reviewed changes

dev-tools/ecs-migration.yml Show resolved Hide resolved

filebeat/module/icinga/debug/_meta/fields.yml Show resolved Hide resolved

webmat force-pushed the ecs-fb-icinga branch from 88f03b0 to 01902c4 Compare December 20, 2018 14:13

Mathieu Martin added 5 commits December 20, 2018 11:10

Map 2 Icinga module fields to ECS:

9afff8f

- icinga.debug.message => message - icinga.debug.severity => log.level

Clean up the comments a bit in ecs-migration.yml.

db9ab6b

Document icinga field moves in ecs-migration

e0d3bd6

Alias migrated Icinga fields, including the new migration attribute

aeabd4d

Changelog

ee106ea

Fix the icinga.*.severity field aliases in ecs-migration.yml

e8eb3dd

webmat force-pushed the ecs-fb-icinga branch from 01902c4 to e8eb3dd Compare December 20, 2018 16:10

ruflin approved these changes Dec 20, 2018

View reviewed changes

webmat merged commit 08fc351 into elastic:master Dec 20, 2018

webmat deleted the ecs-fb-icinga branch December 20, 2018 17:25

widhalmt mentioned this pull request Oct 18, 2019

Make compatible to plain logs *and* filebeat icinga module Icinga/icinga-logstash-pipeline#45

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Filebeat icinga.* to ECS #9294

Convert Filebeat icinga.* to ECS #9294

webmat commented Nov 29, 2018 •

edited

Loading

webmat commented Nov 30, 2018

ruflin commented Dec 3, 2018

webmat commented Dec 19, 2018

webmat commented Dec 19, 2018

webmat commented Dec 20, 2018

webmat commented Dec 20, 2018

ruflin commented Dec 20, 2018

webmat commented Dec 20, 2018

ruflin left a comment

ruflin Dec 20, 2018

webmat Dec 20, 2018

Convert Filebeat icinga.* to ECS #9294

Convert Filebeat icinga.* to ECS #9294

Conversation

webmat commented Nov 29, 2018 • edited Loading

webmat commented Nov 30, 2018

ruflin commented Dec 3, 2018

webmat commented Dec 19, 2018

webmat commented Dec 19, 2018

webmat commented Dec 20, 2018

webmat commented Dec 20, 2018

ruflin commented Dec 20, 2018

webmat commented Dec 20, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin Dec 20, 2018

Choose a reason for hiding this comment

webmat Dec 20, 2018

Choose a reason for hiding this comment

webmat commented Nov 29, 2018 •

edited

Loading