Adding another grok pattern to the filebeats mongo module pipeline. #7560

cwray · 2018-07-10T15:27:21Z

Adding another grok pattern to the filebeats mongo module, ingest pipeline.

We have some extended logging turned on in mongo. The log file format that mongo creates for these extended logs is not parseable by the current grok filter. Adding another grok statement to the list of Grok statements to handle these extended mongo logs.

see.
https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/4

…eline.

elasticmachine · 2018-07-10T15:27:23Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

karmi · 2018-07-10T15:27:24Z

Hi @cwray, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

cwray · 2018-07-10T15:57:01Z

Added.

ruflin · 2018-07-11T07:44:49Z

@cwray Thanks for the contribution. Could you share a few log lines from a mongodb log file to test against the new pattern?

I initially wanted to ask you to add it to our existing test file but just realised it's missing :-( Will try to open a PR to add one so you can add it then.

Could you add an entry to the changelog file?

kvch · 2018-07-11T10:38:49Z

Do you mind rebasing your PR on top of mine? This contains the missing example logs: #7568

cwray · 2018-07-11T13:50:57Z

For the logs that don't match.

2018-07-09T17:05:25.294+0000 I -        [conn212435] end connection 127.0.0.1:43454 (479 connections now open)
2018-07-09T17:05:26.003+0000 I -        [conn212344]   Index Build (background): 18324200/222053333 8%

The log lines will work fine.

2018-07-09T17:05:25.243+0000 I NETWORK  [thread1] connection accepted from 127.0.0.1:43454 #212435 (479 connections now open)
2018-07-09T17:05:25.266+0000 I ACCESS   [conn212435] Successfully authenticated as principal adcellerantClusterAdministrator on admin
2018-07-09T17:05:26.923+0000 I COMMAND  [conn212367] command adc command: aggregate ----REDACTED---- protocol:op_query 350ms

It might aslo be good to catch anyting with a COMMAND statement and parse them out diffrently To be able to look at what commands are running and how long they are taking. But that is for another day.

James-Quigley · 2018-07-11T17:28:07Z

Experiencing an issue because of this as well. Would love an update for this to make it in by default so I don't have to manage my own ingest pipelines

James-Quigley · 2018-07-11T20:47:58Z

FWIW, I was experiencing an issue related to this: https://discuss.elastic.co/t/mongodb-module-grok-expression-does-not-match/139241/4

However creating a pipeline manually with the grok patterns in this PR still did not fix my problem. The patterns pass when using the simulate pipeline, but not when using filebeat.

cwray · 2018-07-11T21:30:33Z

@James-Quigley

What does your pipeline look like?

Here is what is currently working for me.

"filebeat-6.3.0-mongodb-log-pipeline": {
    "on_failure": [
      {
        "set": {
          "field": "error.message",
          "value": "{{ _ingest.on_failure_message }}"
        }
      }
    ],
    "description": "Pipeline for parsing MongoDB logs",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}",
            "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} - *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}"
          ],
          "ignore_missing": true
        }
      }
    ]
  }

You will need to add this pipeline, Then remove the current index and restart filebeats. Im not sure why I had to remove the current index to get this all to work.

James-Quigley · 2018-07-12T15:48:00Z

@cwray I copied the default filebeat pipeline, but just changed the grok patterns. I've totally purged ES of indexes related to filebeat and I removed the existing pipelines before adding the custom pipeline.

When the documents get ingested into ES though, I get "Grok expressions do not match".

Just so I'm clear though, what do you mean when you say "Remove the current index"

ruflin · 2018-07-16T07:38:08Z

@cwray As #7568 is merged, could you rebase you changes on top of master?

cwray · 2018-07-17T00:30:25Z

@ruflin I would love to but I'm a bit new to this process. What is the best way to handle the rebase and push back to this branch? I tried and I think I screwed it up.

houndci-bot · 2018-07-17T00:54:45Z

packetbeat/procs/zsyscall_windows.go

@@ -77,3 +78,22 @@ func _GetExtendedTcpTable(pTcpTable uintptr, pdwSize *uint32, bOrder bool, ulAf
 	}
 	return
 }
+
+func _GetExtendedUdpTable(pTcpTable uintptr, pdwSize *uint32, bOrder bool, ulAf uint32, tableClass uint32, reserved uint32) (code syscall.Errno, err error) {


func _GetExtendedUdpTable should be _GetExtendedUDPTable
func parameter pTcpTable should be pTCPTable

houndci-bot · 2018-07-17T00:54:45Z

packetbeat/procs/zsyscall_windows.go

@@ -57,6 +57,7 @@ var (
 	modiphlpapi = windows.NewLazySystemDLL("iphlpapi.dll")

 	procGetExtendedTcpTable = modiphlpapi.NewProc("GetExtendedTcpTable")
+	procGetExtendedUdpTable = modiphlpapi.NewProc("GetExtendedUdpTable")


var procGetExtendedUdpTable should be procGetExtendedUDPTable

houndci-bot · 2018-07-17T00:54:45Z

packetbeat/procs/syscall_windows.go

+	owningPID uint32
+}
+
+type UDP6RowOwnerPID struct {


exported type UDP6RowOwnerPID should have comment or be unexported

houndci-bot · 2018-07-17T00:54:45Z

packetbeat/procs/syscall_windows.go

@@ -51,6 +52,19 @@ type TCP6RowOwnerPID struct {
 	owningPID     uint32
 }

+type UDPRowOwnerPID struct {


exported type UDPRowOwnerPID should have comment or be unexported

houndci-bot · 2018-07-17T00:54:46Z

packetbeat/procs/syscall_windows.go

@@ -24,6 +24,7 @@ import (
 )

 const (
+	UDP_TABLE_OWNER_PID     = 1


don't use ALL_CAPS in Go names; use CamelCase
exported const UDP_TABLE_OWNER_PID should have comment (or a comment on this block) or be unexported

houndci-bot · 2018-07-17T00:54:47Z

metricbeat/module/elasticsearch/index_recovery/data.go

+// specific language governing permissions and limitations
+// under the License.
+
+package index_recovery


don't use an underscore in package name

houndci-bot · 2018-07-17T00:54:47Z

libbeat/processors/add_kubernetes_metadata/kubernetes_test.go

+// specific language governing permissions and limitations
+// under the License.
+
+package add_kubernetes_metadata


don't use an underscore in package name

houndci-bot · 2018-07-17T00:54:47Z

libbeat/common/tuples.go


 	return tuple
 }

-func (t *TCPTuple) ComputeHashebles() {
+func (t *TCPTuple) ComputeHashables() {


exported method TCPTuple.ComputeHashables should have comment or be unexported

houndci-bot · 2018-07-17T00:54:47Z

libbeat/common/tuples.go


 	return tuple
 }

-func (t *IPPortTuple) ComputeHashebles() {
+func (t *IPPortTuple) ComputeHashables() {


exported method IPPortTuple.ComputeHashables should have comment or be unexported

houndci-bot · 2018-07-17T00:54:48Z

filebeat/reader/multiline/multiline.go

@@ -66,7 +66,7 @@ const (
 type matcher func(last, current []byte) bool

 var (
-	sigMultilineTimeout = errors.New("multline timeout")
+	sigMultilineTimeout = errors.New("multiline timeout")


error var sigMultilineTimeout should have name of the form errFoo

kvch · 2018-07-17T09:20:05Z

Add elastic/beats to your remotes, if you don't have it already.

git remote add upstream https://github.com/elastic/beats
git fetch upstream

Rebase your branch.

git checkout add_mongo_grok_pattern
git rebase upstream/master

Resolve conflicts.

git add filebeat/module/mongodb/log/ingest/pipeline.json
git rebase --continue

Or if you can't add your patch run git rebase --skip, to skip that patch.

Force push it to your repo.

git push origin add_mongo_grok_pattern -f

Also, it's a good idea to create a backup branch before start rebasing. If things get ugly, you can revert to your backup.
Let me know if you need further help.

…eline.

cwray · 2018-07-17T18:58:21Z

Ok, I think I managed to unscrew my commit from yesterday and rebase. Let me know if I can do anything else.

James-Quigley · 2018-07-17T19:25:58Z

@cwray I'm still not able to get this working. I copied exactly what is in the commit and created a new pipeline. Deleted the indices I'm going to be writing to. Then restarted filebeat using the new pipeline and the grok expressions still don't match.

Am I missing something obvious? The exact same pipeline passes when using the simulate API

ruflin

@cwray Rebase looks good and all green.

What would be great now is if you could add one or multiple log lines which were not working before to this file here. https://github.com/elastic/beats/blob/master/filebeat/module/mongodb/log/test/mongodb-debian-3.2.11.log After that you can run GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=mongodb nosetests tests/system/test_modules.py to update the generated content. You will need to have an Elasticsearch instance running on localhost:9200 for this to work.

If it does not work, could also do it for you if you give me access to your branch.

Could you also add a CHANGELOG entry?

ruflin · 2018-07-18T08:51:21Z

filebeat/module/mongodb/log/ingest/pipeline.json

@@ -4,7 +4,9 @@
    "grok": {
      "field": "message",
      "patterns":[
-        "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} \\s*\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}"
+


Can you remove this empty line?

Sure I will give all this build stuff a go if I can get a chance today.

@ruflin I tried to give this a go having a few problems. I have given you access to my repo If you would like to give it a try.

What is the error you are getting?
You need to put you example logs in filebeat/module/mongodb/log/test/mongodb-debian-3.2.11.log. Before running the generator command, you need to build filebeat using make filebeat.test and then run GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=mongodb nosetests tests/system/test_modules.py.

James-Quigley · 2019-01-16T21:02:30Z

FWIW I'm still having this issue. Can't get our mongo logs into ES because of it

cwray · 2019-01-17T16:30:24Z

Sorry, this has not been revisited in a while. I have been pulled off this project at my company and resigned to a different task.

kaiyan-sheng · 2019-06-14T21:37:14Z

This problem is already fixed with the latest filebeat mongodb grok pattern.
With input log:

2018-07-10T20:42:37.863+0000 I NETWORK [thread1] connection accepted from 10.6.4.44:56076 #26676 (42 connections now open)

Output expected json is:

    {
        "@timestamp": "2018-07-10T20:42:37.863Z",
        "event.dataset": "mongodb.log",
        "event.module": "mongodb",
        "fileset.name": "log",
        "input.type": "log",
        "log.level": "I",
        "log.offset": 102,
        "message": "connection accepted from 10.6.4.44:56076 #26676 (42 connections now open)",
        "mongodb.log.component": "NETWORK",
        "mongodb.log.context": "thread1",
        "service.type": "mongodb"
    }

Will close this PR now.

cwray added 2 commits July 10, 2018 09:19

Adding another grok pattern to the filebeats mongo module, ingest pip…

7a76343

…eline.

fixed some spacing

5931d3c

ruflin added module review Filebeat Filebeat labels Jul 11, 2018

houndci-bot reviewed Jul 17, 2018

View reviewed changes

cwray force-pushed the add_mongo_grok_pattern branch from f1c681f to f8687bd Compare July 17, 2018 15:38

Adding another grok pattern to the filebeats mongo module, ingest pip…

069b5e7

…eline.

cwray force-pushed the add_mongo_grok_pattern branch from f8687bd to 069b5e7 Compare July 17, 2018 16:15

Merge branch 'master' into add_mongo_grok_pattern

582947b

ruflin reviewed Jul 18, 2018

View reviewed changes

ruflin added the Team:Integrations Label for the Integrations team label Jan 17, 2019

andresrc assigned kaiyan-sheng Jun 11, 2019

andresrc added the [zube]: In Review label Jun 11, 2019

kaiyan-sheng closed this Jun 14, 2019

zube bot added [zube]: Done and removed [zube]: In Review labels Jun 14, 2019

andresrc removed the [zube]: Done label Jun 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding another grok pattern to the filebeats mongo module pipeline. #7560

Adding another grok pattern to the filebeats mongo module pipeline. #7560

cwray commented Jul 10, 2018

elasticmachine commented Jul 10, 2018

karmi commented Jul 10, 2018

cwray commented Jul 10, 2018

ruflin commented Jul 11, 2018

kvch commented Jul 11, 2018

cwray commented Jul 11, 2018

James-Quigley commented Jul 11, 2018

James-Quigley commented Jul 11, 2018

cwray commented Jul 11, 2018

James-Quigley commented Jul 12, 2018

ruflin commented Jul 16, 2018

cwray commented Jul 17, 2018 •

edited

Loading

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

houndci-bot Jul 17, 2018

kvch commented Jul 17, 2018

cwray commented Jul 17, 2018

James-Quigley commented Jul 17, 2018

ruflin left a comment

ruflin Jul 18, 2018

cwray Jul 18, 2018

cwray Jul 23, 2018

kvch Jul 23, 2018

James-Quigley commented Jan 16, 2019

cwray commented Jan 17, 2019

kaiyan-sheng commented Jun 14, 2019

Adding another grok pattern to the filebeats mongo module pipeline. #7560

Adding another grok pattern to the filebeats mongo module pipeline. #7560

Conversation

cwray commented Jul 10, 2018

elasticmachine commented Jul 10, 2018

karmi commented Jul 10, 2018

cwray commented Jul 10, 2018

ruflin commented Jul 11, 2018

kvch commented Jul 11, 2018

cwray commented Jul 11, 2018

James-Quigley commented Jul 11, 2018

James-Quigley commented Jul 11, 2018

cwray commented Jul 11, 2018

James-Quigley commented Jul 12, 2018

ruflin commented Jul 16, 2018

cwray commented Jul 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Jul 17, 2018

cwray commented Jul 17, 2018

James-Quigley commented Jul 17, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

James-Quigley commented Jan 16, 2019

cwray commented Jan 17, 2019

kaiyan-sheng commented Jun 14, 2019

cwray commented Jul 17, 2018 •

edited

Loading