Move parsers outside of filestream input so others can use them as well #26541
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
|
Pinging @elastic/agent (Team:Agent) |
botelastic
bot
removed
the
needs_team
💔 Build Failed
Expand to view the summary
Build stats
Trends 🧪Test errors
Expand to view the tests failures
|
urso
reviewed
Jun 28, 2021
urso
reviewed
Jun 28, 2021
libbeat/parser/parser.go
Outdated
| type Config struct { | ||
| pCfg CommonConfig | ||
| parsers []common.ConfigNamespace | ||
| } |
There was a problem hiding this comment.
Maybe consider to implement Unpack on the config. Then it can be used directly with go-ucfg unpack like this (we have a similar pattern for TLS and I just introduced one for http):
type myConfig struct {
Parser parser.Config `config:",inline"`
}
with
func (c *Config) Unpack(cfg *common.Config) error {
tmp := struct {
Common CommonConfig `config:",inline"`
Parsers []common.ConfigNamespace `config:"parser"`
}{}
if err := cfg.Unpack(&tmp); err != nil {
return err
}
newC, err := NewConfig(tmp.Common, tmp.Parsers)
if err != nil {
return err
}
*c = *newC
return nil
}
That way if we have an error, the full name of the setting that failed to parse will be pointed out.
Plus users can use it like:
var settings myConfig
if err := cfg.Unpack(&settings); err != nil {
return err
}
reader := ...
contentsReader := settings.Parser.Create(reader)
urso
reviewed
Jun 28, 2021
| @@ -71,7 +72,7 @@ type readerConfig struct { | |||
| MaxBytes int `config:"message_max_bytes" validate:"min=0,nonzero"` | |||
| Tail bool `config:"seek_to_tail"` | |||
|
|
|||
| Parsers []common.ConfigNamespace `config:"parsers"` | |||
| Parsers parser.Config `config:",inline"` | |||
There was a problem hiding this comment.
This would clash with MaxBytes and LineTerminator, right?
Would it make sense to move Exclude/Include lines or encoding to the parser as well?
kvch
force-pushed
the
feature-libbeat-unified-parser-init
branch
from
3befb94 to
f01f797
Compare
urso
approved these changes
Jun 29, 2021
5 tasks
mergify bot
pushed a commit
that referenced
this pull request
Jun 29, 2021
…ll (#26541) ## What does this PR do? The object has its own `Unpack` function, so it is enough for you to add it as an attribute to your configuration. ```golang config parser.Config ``` Then create the parser based on the configuration ```golang p = inp.parserConfig.Create(r) ``` Example configuration accepted by the code above ```yaml parsers: - multiline: type: count lines_count: 3 ``` (cherry picked from commit 2a56cd7)
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jun 29, 2021
…arwin-arm64 * upstream/master: (295 commits) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599) Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438) [Elastic Agent] Improper casting of int64 (elastic#26520) [Elastic Agent] Enable configuring monitoring namespace (elastic#26439) [Heartbeat] configure permissions for synthetics config (elastic#26393) Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545) [Heartbeat] add screenshots config to synthetics (elastic#26455) [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474) Remove all docs about Beats central management (elastic#26399) update data.json for gcp billing (elastic#26506) Skip x-pack metricbeat tests (elastic#26537) [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529) Add changelog entry for elastic#26224 (elastic#26531) ...
kvch
added a commit
that referenced
this pull request
Jun 29, 2021
…ll (#26541) (#26571) ## What does this PR do? The object has its own `Unpack` function, so it is enough for you to add it as an attribute to your configuration. ```golang config parser.Config ``` Then create the parser based on the configuration ```golang p = inp.parserConfig.Create(r) ``` Example configuration accepted by the code above ```yaml parsers: - multiline: type: count lines_count: 3 ``` (cherry picked from commit 2a56cd7) Co-authored-by: Noémi Ványi <[email protected]>
mdelapenya
added a commit
to mdelapenya/beats
that referenced
this pull request
Jun 30, 2021
* master: (25 commits) fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415) macos for metricbeat to run in the extended meta-stage (elastic#26573) Packaging: add arm7 platform in the main pipeline (elastic#26575) [Heartbeat] Skip flakey timer queue test (elastic#26592) Update to "read_pipeline" permission (elastic#26465) (elastic#26580) API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582) Add Fleet agent.id to Agent monitoring data (elastic#26548) Add kinesis metricset (elastic#25989) Refactor of system/memory metricset (elastic#26334) Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219) [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873) docs: Hint for the error "Error extracting container id" (elastic#25824) [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The object has its own
Unpackfunction, so it is enough for you to add it as an attribute to your configuration.Then create the parser based on the configuration
Example configuration accepted by the code above
Why is it important?
This lets all inputs use the generic parsers of libbeat. The available parsers can be changed by passing a list of allowed parsers for the input.
Checklist
- [ ] I have commented my code, particularly in hard-to-understand areas- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.