[Filebeat] Add ZooKeeper Module #25128

legoguy1000 · 2021-04-18T04:27:25Z

What does this PR do?

Adds a new module for ZooKeeper logs.

Why is it important?

Requested by Elastic Cloud team to ingest ZooKeeper logs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/filebeat
TESTING_FILEBEAT_MODULES=zookeeper mage -v pythonIntegTest

Related issues

Closes Filebeat module for zookeeper logs #25061

Use cases

Screenshots

Logs

elasticmachine · 2021-04-18T04:32:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: jsoriano commented: /test
Start Time: 2021-04-28T13:23:29.428+0000
Duration: 137 min 56 sec
Commit: 1b1807f

Test stats 🧪

Test	Results
Failed	0
Passed	13709
Skipped	2285
Total	15994

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13709
Skipped	2285
Total	15994

legoguy1000 · 2021-04-21T12:45:05Z

@matschaffer any thoughts on if more should be done for this? If not I'll take the PR out of draft.

matschaffer · 2021-04-22T00:08:53Z

@legoguy1000 I haven't run it and not sure when I'll be able to to give it a proper review (starting on-call tomorrow).

Just reading it through looks good though so taking out of draft gets 👍 from me.

Maybe @jsoriano can help recommend a reviewer. I'll mention it to some other folks to get eyes on it.

legoguy1000 · 2021-04-22T00:17:06Z

Oooh, on-call, been there. Sounds good, I'll take it out of draft. It's an initial release so can always be improved later.

elasticmachine · 2021-04-22T01:37:24Z

Pinging @elastic/integrations (Team:Integrations)

mergify · 2021-04-22T01:37:55Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 25061-zookeeper-module upstream/25061-zookeeper-module
git merge upstream/master
git push upstream 25061-zookeeper-module

jsoriano · 2021-04-22T08:18:11Z

I will take a look, thanks!

jsoriano · 2021-04-22T08:18:18Z

/test

jsoriano

This is looking great, thanks @legoguy1000 for this!

I have added some suggestions, let me know what you think. Some CI jobs are also complaining about some outdated files, these can be probably solved by running mage update from filebeat and x-pack/filebeat directories and committing the changes.

Other comment I have is that in principle new modules are only added to x-pack/filebeat/module, would you mind to move the module there?

CHANGELOG.next.asciidoc

filebeat/module/zookeeper/_meta/config.yml

filebeat/module/zookeeper/_meta/docs.asciidoc

filebeat/module/zookeeper/audit/manifest.yml

filebeat/module/zookeeper/audit/test/audit.log-expected.json

filebeat/module/zookeeper/audit/config/audit.yml

filebeat/module/zookeeper/audit/test/audit.log

filebeat/module/zookeeper/module.yml

filebeat/docs/modules/zookeeper.asciidoc

legoguy1000 · 2021-04-22T12:40:11Z

This is looking great, thanks @legoguy1000 for this!

I have added some suggestions, let me know what you think. Some CI jobs are also complaining about some outdated files, these can be probably solved by running mage update from filebeat and x-pack/filebeat directories and committing the changes.

Other comment I have is that in principle new modules are only added to x-pack/filebeat/module, would you mind to move the module there?

I can move it. My thought was zookeeper fit alongside Kafka and apache... more than all the security tools but doesn't matter to me.

legoguy1000 · 2021-04-22T12:47:31Z

PR updated

legoguy1000 · 2021-04-26T11:59:51Z

@jsoriano I've updated the PR to account for timestamps in the logs and set accordingly. If it looks good to you, can u run the tests?

henrikno

This is pretty rad! I haven't tested the audit log but the app log seems to work nicely, especially given how non-parseable the logs are. I like that it also captures event.created an ingested so we can detect delays. Thanks for making this.

x-pack/filebeat/module/zookeeper/log/ingest/pipeline.yml

henrikno · 2021-04-27T03:30:50Z

x-pack/filebeat/module/zookeeper/audit/ingest/pipeline.yml

+- convert:
+    field: client.address
+    target_field: client.ip
+    type: ip


Note to future self; this needs 7.13 since it was added recently elastic/elasticsearch#71285

legoguy1000 · 2021-04-27T03:49:28Z

This is pretty rad! I haven't tested the audit log but the app log seems to work nicely, especially given how non-parseable the logs are. I like that it also captures event.created an ingested so we can detect delays. Thanks for making this.

Do u have examples of real audit logs? I just used the example logs from the docs and then had to add additional data to match the log pattern.

jsoriano

Thanks! I think this is almost ready to go, added some small comments, but nothing serious.

x-pack/filebeat/module/zookeeper/_meta/fields.yml

x-pack/filebeat/module/zookeeper/audit/_meta/fields.yml

x-pack/filebeat/module/zookeeper/audit/test/audit_with_dates.log

x-pack/filebeat/module/zookeeper/_meta/docs.asciidoc

filebeat/docs/modules/zookeeper.asciidoc

jsoriano · 2021-04-27T10:15:53Z

/test

legoguy1000 · 2021-04-27T12:46:18Z

@jsoriano I've update the PR with your comments and added your provided sample logs. I think we should be good to test and if it passes merge.

legoguy1000 · 2021-04-27T19:43:04Z

@jsoriano good to retest.

jsoriano · 2021-04-27T19:44:06Z

/test

CHANGELOG.next.asciidoc

legoguy1000 · 2021-04-27T21:30:54Z

@jsoriano same issue with the @timestamp https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-25128/20/pipeline/1930#step-2215-log-668. I haven't seen this before. I thought the tests ignore the time fields that change every time.

legoguy1000 · 2021-04-27T21:36:39Z

@jsoriano same issue with the @timestamp https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-25128/20/pipeline/1930#step-2215-log-668. I haven't seen this before. I thought the tests ignore the time fields that change every time.

I know why. its because the 1 set of example logs that doesn't have an timestamp field to parse so the time changes every time. I'm going to get rid of those since u provided logs from the default pattern and it has a timestamp.

jsoriano · 2021-04-28T13:22:58Z

Oh, I also thought that tests ignored this field for logs without timestamp, but maybe there are no other cases of that. I am ok in any case with testing only with logs with timestamp as they seem to be the usual default.

jsoriano · 2021-04-28T13:23:04Z

/test

legoguy1000 · 2021-04-28T13:26:48Z

Oh, I also thought that tests ignored this field for logs without timestamp, but maybe there are no other cases of that. I am ok in any case with testing only with logs with timestamp as they seem to be the usual default.

Ya there's a list in the python script to declare the filesets to ignore because they don't have a timestamp but decided to just remove them for the same reason.

legoguy1000 · 2021-04-28T15:49:05Z

@jsoriano finally passed :)

Adds a new module for ZooKeeper audit and service logs. (cherry picked from commit d09dfb0)

jsoriano · 2021-04-28T16:00:03Z

Merged and backport open, this will be likely released in 7.14.0. Thanks @legoguy1000 for the good work here!

Adds a new module for ZooKeeper audit and service logs. (cherry picked from commit d09dfb0) Co-authored-by: Alex Resnick <[email protected]> Co-authored-by: DeDe Morton <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 18, 2021

legoguy1000 force-pushed the 25061-zookeeper-module branch from fed9913 to fb0205c Compare April 18, 2021 04:27

jsoriano added the Team:Integrations Label for the Integrations team label Apr 19, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 19, 2021

legoguy1000 force-pushed the 25061-zookeeper-module branch 2 times, most recently from 2acd2d8 to d2e76c4 Compare April 19, 2021 17:18

legoguy1000 marked this pull request as ready for review April 22, 2021 01:37

legoguy1000 force-pushed the 25061-zookeeper-module branch from d2e76c4 to e6de063 Compare April 22, 2021 01:38

jsoriano self-requested a review April 22, 2021 08:18

jsoriano self-assigned this Apr 22, 2021

jsoriano reviewed Apr 22, 2021

View reviewed changes

legoguy1000 force-pushed the 25061-zookeeper-module branch 3 times, most recently from 6ce4965 to 6d45f78 Compare April 22, 2021 18:41

henrikno reviewed Apr 27, 2021

View reviewed changes

jsoriano reviewed Apr 27, 2021

View reviewed changes

legoguy1000 added 9 commits April 27, 2021 19:36

elastic#25061: Add ZooKeeper Module

3e69b5a

Add zookeeper.log fileset

e112e7f

update zookeeper module based on comments.

3e7e46e

update thread name

d26267f

add audit logs with dates

afaf64a

changes from @jsoriano comments

bbe6b15

remove use split

afc0ce7

add user field

e309f8e

update generated data

00de4e9

legoguy1000 force-pushed the 25061-zookeeper-module branch from 0a35fcd to 00de4e9 Compare April 27, 2021 19:42

legoguy1000 commented Apr 27, 2021

View reviewed changes

CHANGELOG.next.asciidoc Show resolved Hide resolved

fix the CI errors

1b1807f

legoguy1000 force-pushed the 25061-zookeeper-module branch from 084a5ee to 1b1807f Compare April 27, 2021 21:46

jsoriano merged commit d09dfb0 into elastic:master Apr 28, 2021

jsoriano mentioned this pull request Apr 28, 2021

Cherry-pick #25128 to 7.x: [Filebeat] Add ZooKeeper Module #25395

Merged

6 tasks

jsoriano added the v7.14.0 label Apr 28, 2021

jsoriano pushed a commit to jsoriano/beats that referenced this pull request Apr 28, 2021

Add ZooKeeper Module to Filebeat (elastic#25128)

3ffa854

Adds a new module for ZooKeeper audit and service logs. (cherry picked from commit d09dfb0)

legoguy1000 deleted the 25061-zookeeper-module branch April 28, 2021 16:02

jsoriano mentioned this pull request Apr 29, 2021

Fix includes to fix broken doc build #25401

Merged

jsoriano mentioned this pull request May 5, 2021

Obtain the zookeeper server id from its logs #25546

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add ZooKeeper Module #25128

[Filebeat] Add ZooKeeper Module #25128

legoguy1000 commented Apr 18, 2021 •

edited

Loading

elasticmachine commented Apr 18, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

legoguy1000 commented Apr 21, 2021

matschaffer commented Apr 22, 2021

legoguy1000 commented Apr 22, 2021

elasticmachine commented Apr 22, 2021

mergify bot commented Apr 22, 2021

jsoriano commented Apr 22, 2021

jsoriano commented Apr 22, 2021

jsoriano left a comment

legoguy1000 commented Apr 22, 2021

legoguy1000 commented Apr 22, 2021

legoguy1000 commented Apr 26, 2021

henrikno left a comment

henrikno Apr 27, 2021

legoguy1000 commented Apr 27, 2021

jsoriano left a comment

jsoriano commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

jsoriano commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

jsoriano commented Apr 28, 2021

jsoriano commented Apr 28, 2021

legoguy1000 commented Apr 28, 2021

legoguy1000 commented Apr 28, 2021

jsoriano commented Apr 28, 2021

[Filebeat] Add ZooKeeper Module #25128

[Filebeat] Add ZooKeeper Module #25128

Conversation

legoguy1000 commented Apr 18, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Apr 18, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

legoguy1000 commented Apr 21, 2021

matschaffer commented Apr 22, 2021

legoguy1000 commented Apr 22, 2021

elasticmachine commented Apr 22, 2021

mergify bot commented Apr 22, 2021

jsoriano commented Apr 22, 2021

jsoriano commented Apr 22, 2021

jsoriano left a comment

Choose a reason for hiding this comment

legoguy1000 commented Apr 22, 2021

legoguy1000 commented Apr 22, 2021

legoguy1000 commented Apr 26, 2021

henrikno left a comment

Choose a reason for hiding this comment

henrikno Apr 27, 2021

Choose a reason for hiding this comment

legoguy1000 commented Apr 27, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

jsoriano commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

legoguy1000 commented Apr 27, 2021

jsoriano commented Apr 28, 2021

jsoriano commented Apr 28, 2021

legoguy1000 commented Apr 28, 2021

legoguy1000 commented Apr 28, 2021

jsoriano commented Apr 28, 2021

legoguy1000 commented Apr 18, 2021 •

edited

Loading

elasticmachine commented Apr 18, 2021 •

edited by jenkins-beats-ci bot

Loading