Add marshal processor #1830
Merged
Add marshal processor #1830
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kuiperda
requested review from
dpaasman00,
BinaryFissionGames and
a team
as code owners
kuiperda
force-pushed
the
add-marshal-processor
branch
from
6b62c13 to
8bb9fb6
Compare
BinaryFissionGames
suggested changes
Aug 30, 2024
BinaryFissionGames
force-pushed
the
release/v1.59.0
branch
from
0a0d942 to
a474c49
Compare
|
@BinaryFissionGames This is ready for re-review when you have some time. Thanks! |
|
Thanks for the thorough review on this, it is much improved from where it started. |
BinaryFissionGames
approved these changes
Sep 12, 2024
dpaasman00
pushed a commit
that referenced
this pull request
Sep 17, 2024
* add factory * add config * add testdata * processor test and other files * processor.go mvp * rename module * add processors.go * add readme * update go.mod replace * run make fmt * lint * lint 2.0 * Skip processing unless body is a map * convert map to kv directly * do not allow configuring xml * Add metadata.yaml * Update go.mod and rest of metadata * fmt and lint * fix factory_test * log instead of erroring when log type is not map * use golden and comparelogs * run go mod tidy at top leve; l * add kv separators to config * rename testdata fileds * cover cases for KV marshaling * rename separator and pair separator properly * update readme with kv separators * go mod tidy + go generate + make fmt * handle nested maps for kv * handle deeply nested logs for kv that also contain problematic characters * go mod tidy + go generate + make fmt * include fuller pipeline for config examples * rename headers * remove XML references in readme * set default config parameters in config * update go mod * add config defaults to factory test * convert string in newMarshalProcessor * use backticks * do not repeatedly fmt.Sprintf(v) * use recursive function * add map kv separators in config options * add processor tests for mapkvseparators * allow specifying map kv separators * go generate + make fmt * lint * sort by keys * update readme * add test to confirm behavior of nested slices with KV marshaling * add array expectation to readme * fix case of config options
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed Change
Add first iteration of Marshal processor to our Agent. (Formerly "Serialize Processor").
The intent is to allow sending data to security platforms that do their own parsing and do not want the body to be in OTLP format. Currently these use cases cannot easily use Bindplane for data reduction on those logs since they must retain the original format.
This processor supports marshaling to JSON and KV.
XML support is going to be through an OTTL function that is being submitted to OTEL by Sam H.
This processor makes assumptions about the format of the body. For now we are going to expect users to parse and flatten body fields correctly before using this processor.
This is an initial draft that I expect to iterate on, but wanted to open it up for feedback.
Checklist