Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30164

tetianakravchenko · 2022-02-02T15:14:24Z

Signed-off-by: Tetiana Kravchenko [email protected]

What does this PR do?

drop log message in elasticsearch/audit ingest pipeline if it is not an audit log.

in ingest pipelines there are defined drop as a first step - for server, slowlog, and deprecation

there is no drop defined for audit logs - https://github.com/elastic/beats/blob/master/filebeat/module/elasticsearch/audit/ingest/pipeline-json.yml

as a result there might be logs duplications - as described in #16540
note: it is not related to the kubernetes itself, in case audit and other log will be written to the same file - there will be the same problem.

Why is it important?

Avoid logs duplication in different filesets.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Related issues

Closes [Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

Use cases

Screenshots

Before adjustments:

After:

Logs

mergify · 2022-02-02T15:14:58Z

This pull request does not have a backport label. Could you fix it @tetianakravchenko? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-02-02T15:15:01Z

Pinging @elastic/integrations (Team:Integrations)

tetianakravchenko · 2022-02-02T15:25:03Z

filebeat/module/elasticsearch/audit/ingest/pipeline-json.yml

+      field: event.type
+      path: elasticsearch.audit
+  - drop:
+        if: '!["rest", "transport", "ip_filter", "security_config_change"].contains(ctx.elasticsearch?.audit?.event?.type)'


it would be much easier to use ctx.elasticsearch.audit.type != 'audit', but as I see this field was introduced only in version 7.3 - elastic/elasticsearch#42887 (docker - https://github.com/elastic/elasticsearch/blob/7.3/distribution/docker/docker-build-context/src/docker/config/log4j2.properties and in core - https://github.com/elastic/elasticsearch/blob/7.3/x-pack/plugin/core/src/main/config/log4j2.properties#L6), in version 7.1 it was not yet available https://github.com/elastic/elasticsearch/blob/7.1/x-pack/plugin/core/src/main/config/log4j2.properties

would it maybe be file to drop support for < 7.3 ?

from the doc it looks like event.type should be a good filter for audit logs, or am I missing smth?

elasticmachine · 2022-02-02T15:35:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-07T12:35:19.133+0000
Duration: 40 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	980
Skipped	192
Total	1172

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2022-02-02T15:38:27Z

Pinging @elastic/stack-monitoring (Stack monitoring)

matschaffer · 2022-02-03T06:58:08Z

Makes some sense to me. Thinking it might be good to have @pgomulka review this, especially since he recently helped pull log samples for ES 8.0

pgomulka

I left few questions

filebeat/module/elasticsearch/audit/test/test-audit-docker.log

filebeat/module/elasticsearch/audit/test/test-audit-docker.log-expected.json

filebeat/module/elasticsearch/audit/ingest/pipeline-json.yml

pgomulka

LGTM

tetianakravchenko · 2022-02-04T14:40:24Z

/test

Signed-off-by: Tetiana Kravchenko <[email protected]>

…/audit Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko · 2022-02-07T12:34:43Z

/test

…30164) * adjust elasticsearch/audit fileset Signed-off-by: Tetiana Kravchenko <[email protected]> * fix formatting Signed-off-by: Tetiana Kravchenko <[email protected]> * adjust drop condition; revert test case for docker Signed-off-by: Tetiana Kravchenko <[email protected]> * fix typos Signed-off-by: Tetiana Kravchenko <[email protected]> * remove 'remove' of user.name field in pipeline-json for elasticsearch/audit Signed-off-by: Tetiana Kravchenko <[email protected]> (cherry picked from commit 1e72124)

…30164) (#30239) * adjust elasticsearch/audit fileset Signed-off-by: Tetiana Kravchenko <[email protected]> * fix formatting Signed-off-by: Tetiana Kravchenko <[email protected]> * adjust drop condition; revert test case for docker Signed-off-by: Tetiana Kravchenko <[email protected]> * fix typos Signed-off-by: Tetiana Kravchenko <[email protected]> * remove 'remove' of user.name field in pipeline-json for elasticsearch/audit Signed-off-by: Tetiana Kravchenko <[email protected]> (cherry picked from commit 1e72124) Co-authored-by: Tetiana Kravchenko <[email protected]>

test-audit-docker.log also contains a case but it was overlooked in the expected file until elastic#30164 added the appropriate drop statements.

…30164) (#30239) * adjust elasticsearch/audit fileset Signed-off-by: Tetiana Kravchenko <[email protected]> * fix formatting Signed-off-by: Tetiana Kravchenko <[email protected]> * adjust drop condition; revert test case for docker Signed-off-by: Tetiana Kravchenko <[email protected]> * fix typos Signed-off-by: Tetiana Kravchenko <[email protected]> * remove 'remove' of user.name field in pipeline-json for elasticsearch/audit Signed-off-by: Tetiana Kravchenko <[email protected]> (cherry picked from commit 1e72124) Co-authored-by: Tetiana Kravchenko <[email protected]> (cherry picked from commit 32ecd20)

…h logs (#30440) * Ensure we drop server logs that show up in deprecation pipeline * Add note about deprecation dataset normalization * Add test for mixed es server logs This pipeline already contained a drop to avoid duplicate logging. * Ensure we drop server logs that show up in slowlog pipeline This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging. * Add "mixed" test for elasticsearch audit logs test-audit-docker.log also contains a case but it was overlooked in the expected file until #30164 added the appropriate drop statements. * Changelog entry * Remove duplicatd filebeat header

…h logs (#30440) * Ensure we drop server logs that show up in deprecation pipeline * Add note about deprecation dataset normalization * Add test for mixed es server logs This pipeline already contained a drop to avoid duplicate logging. * Ensure we drop server logs that show up in slowlog pipeline This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging. * Add "mixed" test for elasticsearch audit logs test-audit-docker.log also contains a case but it was overlooked in the expected file until #30164 added the appropriate drop statements. * Changelog entry * Remove duplicatd filebeat header (cherry picked from commit 7b67384)

…30164) (#30239) (#30447) * adjust elasticsearch/audit fileset Signed-off-by: Tetiana Kravchenko <[email protected]> * fix formatting Signed-off-by: Tetiana Kravchenko <[email protected]> * adjust drop condition; revert test case for docker Signed-off-by: Tetiana Kravchenko <[email protected]> * fix typos Signed-off-by: Tetiana Kravchenko <[email protected]> * remove 'remove' of user.name field in pipeline-json for elasticsearch/audit Signed-off-by: Tetiana Kravchenko <[email protected]> (cherry picked from commit 1e72124) Co-authored-by: Tetiana Kravchenko <[email protected]> (cherry picked from commit 32ecd20) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…h logs (#30440) (#30488) * Ensure we drop server logs that show up in deprecation pipeline * Add note about deprecation dataset normalization * Add test for mixed es server logs This pipeline already contained a drop to avoid duplicate logging. * Ensure we drop server logs that show up in slowlog pipeline This was partially guarded against in testing due to the grok on `elasticsearch.slowlog` but probably better to explicitly drop and avoid duplicate logging. * Add "mixed" test for elasticsearch audit logs test-audit-docker.log also contains a case but it was overlooked in the expected file until #30164 added the appropriate drop statements. * Changelog entry * Remove duplicatd filebeat header (cherry picked from commit 7b67384) Co-authored-by: Mat Schaffer <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 2, 2022

tetianakravchenko added the Team:Integrations Label for the Integrations team label Feb 2, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Feb 2, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 2, 2022

tetianakravchenko commented Feb 2, 2022

View reviewed changes

jsoriano added the Feature:Stack Monitoring label Feb 2, 2022

sayden self-requested a review February 2, 2022 16:56

matschaffer requested a review from pgomulka February 3, 2022 06:57

pgomulka reviewed Feb 3, 2022

View reviewed changes

pgomulka approved these changes Feb 3, 2022

View reviewed changes

tetianakravchenko added 3 commits February 4, 2022 16:29

adjust elasticsearch/audit fileset

9184b4e

Signed-off-by: Tetiana Kravchenko <[email protected]>

fix formatting

005cdbe

Signed-off-by: Tetiana Kravchenko <[email protected]>

adjust drop condition; revert test case for docker

264c8d1

Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko force-pushed the es-module branch from e42c474 to 264c8d1 Compare February 4, 2022 15:29

tetianakravchenko added 2 commits February 7, 2022 09:27

fix typos

b0d5bc6

Signed-off-by: Tetiana Kravchenko <[email protected]>

remove 'remove' of user.name field in pipeline-json for elasticsearch…

94da9c4

…/audit Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko merged commit 1e72124 into elastic:main Feb 7, 2022

tetianakravchenko added the backport-v8.0.0 Automated backport with mergify label Feb 7, 2022

mergify bot mentioned this pull request Feb 7, 2022

[8.0](backport #30164) Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30239

Merged

matschaffer added a commit to matschaffer/beats that referenced this pull request Feb 17, 2022

Add "mixed" test for elasticsearch audit logs

537d51a

test-audit-docker.log also contains a case but it was overlooked in the expected file until elastic#30164 added the appropriate drop statements.

matschaffer mentioned this pull request Feb 17, 2022

Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs #30440

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30164

Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30164

tetianakravchenko commented Feb 2, 2022

mergify bot commented Feb 2, 2022

elasticmachine commented Feb 2, 2022

tetianakravchenko Feb 2, 2022 •

edited

Loading

elasticmachine commented Feb 2, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Feb 2, 2022

matschaffer commented Feb 3, 2022

pgomulka left a comment

pgomulka left a comment

tetianakravchenko commented Feb 4, 2022

tetianakravchenko commented Feb 7, 2022

Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30164

Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30164

Conversation

tetianakravchenko commented Feb 2, 2022

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Feb 2, 2022

elasticmachine commented Feb 2, 2022

tetianakravchenko Feb 2, 2022 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Feb 2, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Feb 2, 2022

matschaffer commented Feb 3, 2022

pgomulka left a comment

Choose a reason for hiding this comment

pgomulka left a comment

Choose a reason for hiding this comment

tetianakravchenko commented Feb 4, 2022

tetianakravchenko commented Feb 7, 2022

tetianakravchenko Feb 2, 2022 •

edited

Loading

elasticmachine commented Feb 2, 2022 •

edited by jenkins-beats-ci bot

Loading