Nats Filebeat

[ChrsMark]

Nats Filebeat implementation

Co-Authored-By: Stamatis Katsaounis [email protected]
Co-Authored-By: Michael Katsoulis [email protected]

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[ruflin]

@ChrsMark Thanks for opening this WIP PR. Let me know if you some assistance on how to work with the test log files and how to generate the expected json outputs.

[ruflin]

jenkins, test this

[ruflin]

Left a few minor comments, almost there.

[MichaelKatsoulis]

@ruflin I believe we have addressed the open issues in this PR. In another upcoming PR we are planning to add more fields and also the dashboard.

[ruflin]

jenkins, test this

[ChrsMark]

@ruflin made an attempt to address the last comments, feel free!

[ruflin]

jenkins, test this

[ruflin]

@ChrsMark PR looks good to me but something is odd with CI. Could you rebase on master?

[skatsaounis]

@ruflin done

[ChrsMark]

Part of #10071

Completion Status

• Filebeat Implementation
• Tests
• Kibana Dashboard
• ML

Kibana dashboard will be included along with the more fields PR. (we need to enrich the grok patterns so as to extract more info from the messages).
ML needs some investigation and good amount of production data.

[ChrsMark]

Is the CI failure relevant? 🤔

[ruflin]

Hm, lots of stuff failed that I have never seen failing before. Seems like something odd happened here. Can you rebase on master? And perhaps wait until #10723 is merged as otherwise the build would fail.

[ChrsMark]

Even though, I rebased after #10723 was merged nothing changed. Seeing the travis' details I notice two filebeat jobs to be failing. Diving into the first one (filebeat testsuite) I see that not only nats module is failing but also some others too. I'm wondering if removing the fields as discussed previously may cause the problem but this does not explain the other modules' failures. In addition, @skatsaounis informed me that he had trouble in running locally the nose tests cause of some incompatibility errors with elasticsearch. 🤔

Anyway, if we could check something on our end @ruflin feel free to mention!

[ruflin]

Checked out the branch locally and I think I found a fix (see comment above).

@skatsaounis Could you share a bit more on the problems you run into? Which version of ES did you use? In testing/environment you can just run make start to get a full setup (which should have the right version).

[skatsaounis]

@ruflin I would like to shed some light into how I am testing locally. I have a docker compose file which uses 6.6.0 elasticsearch and kibana. Then, I am running the tests with this deployment up and running.

Good to know that make start is the correct way to go, so I am going to use it from now on.

The problem I face with my setup is that the filebeat-test-modules index is not created. Thus, I am receiving a timeout from the python test when waiting for the index. This is happening with every module and fileset. Weird fact: When I initially produced the expected file (couple of commits before) I was using the same way to run the tests and I was able to run them without problem.

Irrelevant (or not): I tried to use 7.0.0-SNAPSHOT version like Travis, but I failed to configure it correctly, so I cannot work with it.

[ruflin]

Different versions of the Elastic Stack might output a bit different data structures. Because of this it's important to use the exact same version for testing. This does not explain on why it was hanging. Did you have geoip and user_agent installed which is still needed in 6.6?

Having said all this, you should be save with make start. And use make stop to stop and clean the containers, because otherwise you have left over volumes and networks which can cause issue in future tests. It's always good to start with a clean env.

[ruflin]

jenkins, test this

[ChrsMark]

@ruflin, mergable? ☺️

[ruflin]

@ChrsMark YES. Thanks.