Start to split filebeat/channel up #17655
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
urso
added
in progress
|
Pinging @elastic/integrations-services (Team:Services) |
urso
commented
Apr 9, 2020
filebeat/beater/filebeat.go
Outdated
| @@ -235,8 +237,11 @@ func (fb *Filebeat) Run(b *beat.Beat) error { | |||
| return err | |||
| } | |||
|
|
|||
| fb.pipeline = pipetool.WithPipelineEventCounter(b.Publisher, wgEvents) | |||
| fb.pipeline = pipetool.WithDefaultGuarantees(fb.pipeline, beat.GuaranteedSend) | |||
There was a problem hiding this comment.
When doing this change I noticed wgEvents was propably not hooked up correctly. This PR might fix an issue or flaky tests with shutdown timeout not working correctly.
urso
added
review
[zube]: In Review
and removed
in progress
urso
commented
Apr 15, 2020
| @@ -62,7 +62,7 @@ func newCrawler( | |||
|
|
|||
| // Start starts the crawler with all inputs | |||
| func (c *crawler) Start( | |||
| pipeline beat.Pipeline, | |||
| pipeline beat.PipelineConnector, | |||
There was a problem hiding this comment.
PipelineConnector is the 'smaller' interface. I hope to merge Pipeline and PipelineConnector interfaces in the future, such that Pipeline will not have any extra methods anymore.
kvch
approved these changes
Apr 15, 2020
ycombinator
reviewed
Apr 15, 2020
filebeat/channel/runner.go
Outdated
|
|
||
| type onCreateFactory struct { | ||
| factory cfgfile.RunnerFactory | ||
| edit onCreateWrapper |
There was a problem hiding this comment.
Nit: call this create instead of edit?
|
Jenkins, run tests |
|
jenkins run the tests please |
added 2 commits
April 29, 2020 14:26
The Factory in filebeat/channel package wraps the publisher pipeline, in order to add some shutdown propagation support, but also to modify the beat.ClientConfig when an input is configured. The channel.Connector reads and applies common settings for all inputs, configures Guaranteed sending, and hooks up a global counter with each beat.Client, in order to track live events. This change reduces the responsibilities of the factory, by splitting them up. A package libbeat/publisher/pipetool is introduced to provide helper functions for modifying a beat.Pipeline before and after Connect. This centers support for more functionality on the beat.PipelineConnector and beat.Client interfaces, instead of incompatible types wrapping the pipeline. The Factory in filebeat/channel will not modify the input configuration or the beat.Client anymore. It is only required to add shutdown signaling to the pipeline (which we will remove in the future as well). The filebeat/channel provides a helper function for decorating a cfgfile.RunnerFactory, such that common configurations are still applied as before. Again, by centering around a small set of common interfaces, reduce the effort to integrate future input refactorings. All in all, the filebeat/channel factory still provides the same functionality as before (for now), but splits the different functionalities into 3 separate interfaces.
added 2 commits
April 29, 2020 14:26
The way the counter support is setup is very specific for filebeat and how Filebeat hooks up a global ACKer, and the Registrar for storing file.State. Moving it to filebeat to not encourage someone to use a it.
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
3 tasks
urso
added
v7.9.0
and removed
needs_backport
urso
pushed a commit
to urso/beats
that referenced
this pull request
Jul 3, 2020
(cherry picked from commit bc8123a)
urso
pushed a commit
that referenced
this pull request
Jul 7, 2020
(cherry picked from commit bc8123a)
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The Factory in filebeat/channel package wraps the publisher pipeline, in order to add some shutdown propagation support, but also to modify the beat.ClientConfig when an input is configured. The channel.Connector reads and applies common settings for all inputs, configures Guaranteed sending, and hooks up a global counter with each beat.Client, in order to track live events.
This change reduces the responsibilities of the factory, by splitting them up. A package libbeat/publisher/pipetool is introduced to provide helper functions for modifying a beat.Pipeline before and after Connect. This centers support for more functionality on the beat.PipelineConnector and beat.Client interfaces, instead of incompatible types wrapping the pipeline.
The Factory in filebeat/channel will not modify the input configuration or the beat.Client anymore. It is only required to add shutdown signaling to the pipeline (which we will remove in the future as well). The filebeat/channel provides a helper function for decorating a cfgfile.RunnerFactory, such that common configurations are still applied as before. Again, by centering around a small set of common interfaces, reduce the effort to integrate future input refactorings.
All in all, the filebeat/channel factory still provides the same functionality as before (for now), but splits the different functionalities into 3 separate interfaces.
Why is it important?
Support ease of integration of new input API, based on RunnerFactory only.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added tests that prove my fix is effective or that my feature worksCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues