Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adopt parsers in Filebeat's journald input #29070

Merged
merged 6 commits into from
Nov 30, 2021
Merged

Adopt parsers in Filebeat's journald input #29070

merged 6 commits into from
Nov 30, 2021

Conversation

belimawr
Copy link
Contributor

@belimawr belimawr commented Nov 22, 2021 •
edited
Loading

Splits from #26130

What does this PR do?

Adopt parsers in Filebeat's journald input

Why is it important?

It enables Filebeat to parse more types of log from journald input

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Integration tests using a real journald

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@belimawr belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Nov 22, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 22, 2021
@mergify
Copy link
Contributor

mergify bot commented Nov 22, 2021

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 22, 2021
@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Nov 22, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 22, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-11-30T11:50:20.572+0000

  • Duration: 165 min 19 sec

  • Commit: 67b5ba7

Test stats 🧪

Test Results
Failed 0
Passed 48630
Skipped 4286
Total 52916

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@belimawr belimawr force-pushed the filebeat-journald-parsers branch 3 times, most recently from 0cd3225 to 5a94bcf Compare November 24, 2021 13:32
@belimawr belimawr force-pushed the filebeat-journald-parsers branch from 5a94bcf to 18dabb7 Compare November 24, 2021 13:42
@mergify
Copy link
Contributor

mergify bot commented Nov 24, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-journald-parsers upstream/filebeat-journald-parsers
git merge upstream/master
git push upstream filebeat-journald-parsers

@belimawr belimawr force-pushed the filebeat-journald-parsers branch from 18dabb7 to 4a52f86 Compare November 24, 2021 14:01
@belimawr belimawr marked this pull request as ready for review November 24, 2021 14:01
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@belimawr belimawr requested a review from kvch November 24, 2021 14:01
kvch
kvch suggested changes Nov 25, 2021
View reviewed changes
Copy link
Contributor

@kvch kvch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove the e2e test and add a simpler integration test in a follow up PR.

@belimawr belimawr force-pushed the filebeat-journald-parsers branch 3 times, most recently from 2a3dc19 to 48c919c Compare November 29, 2021 11:06
@kvch kvch added the backport-v8.0.0 Automated backport with mergify label Nov 29, 2021
@mergify mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Nov 29, 2021
@belimawr belimawr force-pushed the filebeat-journald-parsers branch from 48c919c to dc7eff3 Compare November 29, 2021 17:10
@belimawr belimawr force-pushed the filebeat-journald-parsers branch from c0ad34d to 5a5928d Compare November 29, 2021 17:22
@belimawr
Copy link
Contributor Author

Let's remove the e2e test and add a simpler integration test in a follow up PR.

I removed it and added a unit test on: filebeat/input/journald/input_parsers_test.go

@belimawr belimawr requested a review from kvch November 29, 2021 17:24
@belimawr belimawr requested a review from kvch November 30, 2021 09:19
@mergify
Copy link
Contributor

mergify bot commented Nov 30, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-journald-parsers upstream/filebeat-journald-parsers
git merge upstream/master
git push upstream filebeat-journald-parsers

@belimawr
Adopt parsers in Filebeat's journald input
3c13a10 
One test is added that runs Filebeat reading from a journald file, it
only tests one parser, however that should be enough to ensure parsers
are supported on journald input.

Splits from elastic#26130
@belimawr
re-writing tests
3ffef04
@belimawr
make mage check update
5b91c84
@belimawr
renaming test file
fb2e84b
@belimawr
mage
99c444c
@belimawr
clean up
67b5ba7
@belimawr belimawr force-pushed the filebeat-journald-parsers branch from 76bd9af to 67b5ba7 Compare November 30, 2021 11:49
@belimawr
Copy link
Contributor Author

rebase onto master, force push

@belimawr belimawr merged commit 8fcad13 into elastic:master Nov 30, 2021
@belimawr belimawr deleted the filebeat-journald-parsers branch November 30, 2021 16:54
mergify bot pushed a commit that referenced this pull request Nov 30, 2021
@belimawr
Adopt parsers in Filebeat's journald input (#29070)
4d9ec89 
One test is added that runs Filebeat reading from a journald file, it
only tests one parser, however that should be enough to ensure parsers
are supported on journald input.

Splits from #26130

(cherry picked from commit 8fcad13)
@mergify mergify bot mentioned this pull request Nov 30, 2021
Merged
elasticmachine pushed a commit to nxei/beats that referenced this pull request Dec 1, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into github-actions-build
99c50fb 
* upstream/master:
  [libbeat] Fix add_labels flattening of arrays values (elastic#29211)
  Change elastic-agent pprof default to false (elastic#29155)
  elastic#28472 fix flaky tests in libbeat fmtstr to use time.UTC instead of time.Local (elastic#28473)
  Adopt `parsers` in Filebeat's journald input (elastic#29070)
  [Elastic Agent] Add process error handling guidelines (elastic#29152)
  winlogbeat/sys/winevent: use reflect IsZero method (elastic#29190)
  Remove Journalbeat (elastic#29131)
  Add note that there is no warranty or support for generator code (elastic#28797)
  packetbeat: preparation for npcap addition (elastic#29017)
  Use the generic helper for opening file to read in filestream (elastic#29180)
  Workflow for macos (elastic#29174)
  Fix `decode_json_fields` processor to always add error key (elastic#29107)
belimawr added a commit that referenced this pull request Dec 6, 2021
@mergify @belimawr
Adopt parsers in Filebeat's journald input (#29070) (#29203)
96a2a2a 
One test is added that runs Filebeat reading from a journald file, it
only tests one parser, however that should be enough to ensure parsers
are supported on journald input.

Splits from #26130

(cherry picked from commit 8fcad13)

Co-authored-by: Tiago Queiroz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kvch kvch kvch approved these changes

Assignees
No one assigned
Labels
backport-v8.0.0 Automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@belimawr @elasticmachine @kvch @jlind23