Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #22134 to 7.x: [Packetbeat] Refactor packetbeat for use with Elastic Agent #22546

Merged
merged 3 commits into from
Nov 11, 2020
Merged

Cherry-pick #22134 to 7.x: [Packetbeat] Refactor packetbeat for use with Elastic Agent #22546

merged 3 commits into from
Nov 11, 2020

Conversation

andrewstucki
Copy link

Cherry-pick of PR #22134 to 7.x branch. Original message:

What does this PR do?

This PR refactors packetbeat to support for agent-based configuration. In order to facilitate control by agent, it does a few things:

  1. Gets rid of any global variables that are initialized based on the configuration state
  2. Adds handling that overrides any configuration if packetbeat.inputs are provided.
  3. Makes the index that packetbeat pushes events to overrideable on each input (flows or protocols)
  4. Uses centralized configuration management reloading to restart the sniffer and all worker goroutines when configuration changes

Follow ups

We'll likely want to do a few things in the future, I can create issues if we want:

  1. Right now, due to the way a single sniffer gets initialized and then delegates all of the protocol parsing to internally registered protocol handlers, each configuration reload, if even a single input changes, the sniffer needs to be stopped and started--so effectively all inputs are tied together. Not sure if it would make sense to move to some sort of worker routine per protocol or not, but it would get rid of some of the custom reload logic I had to add.
  2. Once again, due to the single sniffer, the sniffing interface can really only be configured once. Right now I just choose the "default" interface based off of the OS, but ideally this would be customizable. That said, what it might mean is that we need to make all packetbeat integrations a single package or change packetbeat so we run a sniffer per input.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

[Packetbeat] Refactor packetbeat for use with Elastic Agent (elastic#…
fd0e75c 
…22134)

* Refactor packetbeat to support agent-based configuration

* Add documentation changes and a Changelog entry

* Update reference template

* Fix funny merge

* Incorporate feedback

* use streams instead of inputs

* support multiple sniffers

* fix shutdown_timeout behavior

(cherry picked from commit 8c05a41)
@andrewstucki andrewstucki requested a review from a team as a code owner November 11, 2020 16:03
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 11, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 11, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 11, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #22546 updated]

  • Start Time: 2020-11-11T22:19:33.011+0000

  • Duration: 57 min 8 sec

Test stats 🧪

Test Results
Failed 0
Passed 16370
Skipped 1368
Total 17738

Steps errors 2

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws

  • Took 0 min 15 sec . View more details on here

Terraform Apply on x-pack/metricbeat/module/aws

  • Took 0 min 16 sec . View more details on here

@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 11, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 16370
Skipped 1368
Total 17738

@andrewstucki andrewstucki merged commit 17bee6a into elastic:7.x Nov 11, 2020
@andrewstucki andrewstucki deleted the backport_22134_7.x branch November 11, 2020 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leehinman leehinman leehinman approved these changes

Assignees
No one assigned
Labels
backport [zube]: In Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewstucki @elasticmachine @leehinman