IBM MQ Filebeat module #8782

felix-lessoer · 2018-10-29T15:17:54Z

This module is able to read errorlogs from IBM MQ.

@ruflin supporting this.

ruflin · 2018-10-29T20:20:18Z

Still need to have a closer look. Two things so far:

Could you add an entry to the changelog?
Could you add some example logs to the tests directory?

filebeat/docs/modules/ibmmq.asciidoc

ruflin · 2019-05-21T19:51:26Z

x-pack/filebeat/module/ibmmq/errorlog/_meta/fields.yml

+  type: group
+  fields:
+  - name: process
+    description: Please add description


@felix-lessoer Help on describing these fields from your side would be super helpful as you know more about these then I do.

x-pack/filebeat/module/ibmmq/errorlog/_meta/fields.yml

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01_QM2.log-expected.json

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01.log

webmat · 2019-05-22T13:34:13Z

@ruflin Thanks for pinging me on this!

Don't wait for me for a review of the PR. But I'll definitely be coming back to this module, to see patterns for doing structured logging

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01.log-expected.json

ruflin

@exekias @felix-lessoer I pushed quite a few changes to the module around the fields and some docs cleanup. I left some comments where I still need more details on the fields to have a proper description and for the arithinsert* and commentinsert* we need to decide on how to handle it.

ruflin · 2019-06-07T12:25:11Z

filebeat/tests/system/test_modules.py

@@ -162,8 +162,8 @@ def run_on_file(self, module, fileset, test_file, cfgfile):
            assert obj["event"]["module"] == module, "expected event.module={} but got {}".format(
                module, obj["event"]["module"])

-            assert "error" not in obj, "not error expected but got: {}".format(
-                obj)
+            #assert "error" not in obj, "not error expected but got: {}".format(


This is currently commented out to allow for error.id. But not sure we should store it into this value, it depends a bit on what we want to use it for long term in the context of ECS. @webmat

ruflin · 2019-06-07T12:51:39Z

x-pack/filebeat/module/ibmmq/_meta/docs.asciidoc

+[float]
+=== Compatibility
+
+TODO: document with what versions of the software is this tested


@felix-lessoer Can you provide these details here?

I tested in v 9.1.0.0 .. But the logs are the same in older versions

I think we can just write that, it's good to explain what versions have we tested against, even if it's expected to work with others

ruflin · 2019-06-07T12:54:09Z

x-pack/filebeat/module/ibmmq/_meta/kibana/7/dashboard/Filebeat-IBMMQ-Overview.json

@@ -0,0 +1,814 @@
+{


This dashboard will probably need an update as soon as we finished the field renaming.

ruflin · 2019-06-07T12:54:53Z

x-pack/filebeat/module/ibmmq/errorlog/_meta/fields.yml

+  type: group
+  fields:
+  - name: installation
+    description: Please add description


@felix-lessoer I need some details here on the description.

This is the installation name which can be given at installation time. Default: Installation 1

From IBM documentation

Each installation of IBM® MQ on UNIX, Linux, and Windows, has a unique identifier known as an installation name. The installation name is used to associate things such as queue managers and configuration files with an installation.

updated. I wonder if we should in some cases even provide links to the docs.

x-pack/filebeat/module/ibmmq/errorlog/_meta/fields.yml

ruflin · 2019-06-11T11:55:32Z

x-pack/filebeat/module/ibmmq/errorlog/ingest/pipeline.json

+            "remove": {
+                "field": [
+                    "log_timestamp",
+                    "message"


Note for myself: Should we really fully remove message or keep at least something inside?

ruflin · 2019-06-13T13:42:21Z

@felix-lessoer I have a more high level question: We call this fileset errorlog. Are there also other logs from ibmmq that we should add in the future? Or is this the only log and should just be ibmmq.log?

felix-lessoer · 2019-06-19T14:08:25Z

Thats fine. Lets call it log for now

felix-lessoer · 2019-07-19T10:51:53Z

I updated the pipeline to handle error when fields are missing. I also added the @timestamp and message field to be able to use the default configuration of the Logs UI .
I also updated to sample Dashboard to fit to ECS . The Dashboard Import JSON looks the same as the others. But when I run filebeat setup I run into an error. No sure why.
@exekias Can you check on this??

exekias · 2019-07-24T09:39:16Z

Checked the dashboard and it's now loading properly, we need to add filters to make sure that showed metrics and saved search only apply to data coming out of this module (event.dataset: ibmmq.errorlog).

exekias · 2019-07-25T10:00:25Z

I've updated expected files, let's wait for CI 🤞

exekias · 2019-07-25T10:56:07Z

Tests are failing, it seems arithinsert1 doesn't get removed for the sample log AMQERR01.log. @felix-lessoer I guess the pipeline is failing for that one?

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01.log-expected.json

exekias · 2019-07-26T10:03:31Z

failing tests are unrelated

ruflin · 2019-07-29T08:39:02Z

🎉 Thanks @felix-lessoer and @exekias for pushing this through!

ruflin added module review Filebeat Filebeat labels Oct 29, 2018

ruflin removed Team:Integrations Label for the Integrations team labels Nov 27, 2018

ruflin added the Team:Integrations Label for the Integrations team label May 10, 2019

felix-lessoer added 3 commits May 10, 2019 10:17

Initial commit for the IBM MQ Filebeat module

63c61b6

Updated manifest to fit for windows files

599fa45

Add example log data

c2447d2

ruflin force-pushed the mq branch from e63ccca to c2447d2 Compare May 10, 2019 08:19

felix-lessoer requested a review from a team as a code owner May 10, 2019 08:19

add expected files

95aa65c

andresrc added the [zube]: In Review label May 20, 2019

andresrc assigned exekias May 20, 2019

update docs

016baa1

felix-lessoer requested a review from a team as a code owner May 21, 2019 19:46

ruflin reviewed May 22, 2019

View reviewed changes

webmat reviewed May 22, 2019

View reviewed changes

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01.log Show resolved Hide resolved

added kibana export

ecc341d

ruflin reviewed Jun 7, 2019

View reviewed changes

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01.log-expected.json Outdated Show resolved Hide resolved

ruflin reviewed Jun 7, 2019

View reviewed changes

ruflin reviewed Jun 11, 2019

View reviewed changes

felix-lessoer and others added 6 commits June 13, 2019 15:28

Initial commit for the IBM MQ Filebeat module

93363c2

Updated manifest to fit for windows files

c4003d8

Add example log data

8d5cc45

add expected files

abc3390

update docs

9349f86

added kibana export

a7d9da0

update ecs files

a56f235

felix-lessoer added 3 commits July 18, 2019 10:16

Merge branch 'mq' with local changes

eef1ed5

Update pipeline to improve error handling + add @timestamp and message

1a3e8b9

Updated Dashboard to fit to ECS

cd3d613

andresrc added the v7.4.0 label Jul 22, 2019

felix-lessoer and others added 5 commits July 24, 2019 10:07

Updated Dashboard to fit to the right index pattern

ee63457

Bugfixing dashboard

cd027bc

Merge branch 'master' into felix-lessoer-mq

8a7620f

Add compatibility note

0439eed

Update changelog

7938cc0

felix-lessoer and others added 2 commits July 24, 2019 11:44

Dashboard is now based on Errorlog Saved search

6e294c6

Update expected files

d58f8bd

Use common error handling

c0b5726

exekias reviewed Jul 25, 2019

View reviewed changes

x-pack/filebeat/module/ibmmq/errorlog/test/AMQERR01.log-expected.json Outdated Show resolved Hide resolved

Carlos Pérez-Aradros Herce added 3 commits July 25, 2019 13:54

Use generic error reporting

f911b5f

Fix grok pattern

14b07a6

Fix pipeline remove processor

b26b443

exekias approved these changes Jul 26, 2019

View reviewed changes

exekias merged commit 68519cb into elastic:master Jul 29, 2019

zube bot added [zube]: Done and removed [zube]: In Review labels Jul 29, 2019

andresrc removed the [zube]: Done label Jul 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IBM MQ Filebeat module #8782

IBM MQ Filebeat module #8782

felix-lessoer commented Oct 29, 2018

ruflin commented Oct 29, 2018

ruflin May 21, 2019

webmat commented May 22, 2019

ruflin left a comment

ruflin Jun 7, 2019

ruflin Jun 7, 2019

felix-lessoer Jun 11, 2019

exekias Jul 23, 2019

ruflin Jun 7, 2019

ruflin Jun 7, 2019

felix-lessoer Jun 11, 2019

ruflin Jun 11, 2019

ruflin Jun 11, 2019

ruflin commented Jun 13, 2019

felix-lessoer commented Jun 19, 2019 •

edited

Loading

felix-lessoer commented Jul 19, 2019 •

edited

Loading

exekias commented Jul 24, 2019

exekias commented Jul 25, 2019

exekias commented Jul 25, 2019

exekias commented Jul 26, 2019

ruflin commented Jul 29, 2019

IBM MQ Filebeat module #8782

IBM MQ Filebeat module #8782

Conversation

felix-lessoer commented Oct 29, 2018

ruflin commented Oct 29, 2018

Choose a reason for hiding this comment

webmat commented May 22, 2019

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jun 13, 2019

felix-lessoer commented Jun 19, 2019 • edited Loading

felix-lessoer commented Jul 19, 2019 • edited Loading

exekias commented Jul 24, 2019

exekias commented Jul 25, 2019

exekias commented Jul 25, 2019

exekias commented Jul 26, 2019

ruflin commented Jul 29, 2019

felix-lessoer commented Jun 19, 2019 •

edited

Loading

felix-lessoer commented Jul 19, 2019 •

edited

Loading