-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
processor: inline re-ingestion implementation #7455
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
This adds the concept of processor unit stage index which is used when a filter such as rewrite_tag needs to re-ingest a chunk. In this case, a call to flb_input_log_append_ex is made with the processor stage index corresponding to the next processor unit in the processor stack. This allows us to overcome the limitation that comes with ingesting the logs using the synthetic emitter input plugin. Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
leonardo-albertovich
requested review from
edsiper,
fujimotos and
koleini
as code owners
May 22, 2023 16:16
edsiper
requested changes
May 23, 2023
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
cosmo0920
reviewed
May 24, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I confirmed that this PR works as expected with my using rewrite_tag configuration:
pipeline:
inputs:
- name: dummy
tag: test-tag.raw
dummy: '{"tool": "fluent", "sub": {"s1": {"s2": "bit"}}}'
processors:
logs:
- name: modify
match: test-tag.*
add: hostname monox
- name: rewrite_tag
match: test-tag.*
rule: $tool ^(fluent)$ from.$TAG.new.$tool.$sub['s1']['s2'].out false
emitter_name: k8s_meta_into_tag_partial_all
- name: lua
match: from.test-tag.raw.*
call: append_tag
code: |
function append_tag(tag, timestamp, record)
new_record = record
new_record["tag"] = tag
return 1, timestamp, new_record
end
outputs:
- name: stdout
match: '*'
processors:
logs:
- name: lua
call: add_field
code: |
function add_field(tag, timestamp, record)
new_record = record
new_record["output"] = "new data"
return 1, timestamp, new_record
end
I commented a some of nitpicks for naming function which work with processor stage.
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Leonardo Alminana <[email protected]>
…name Signed-off-by: Leonardo Alminana <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the concept of processor stages which are just an index that's used to skip the stages that the record has already gone through when re-ingesting them (ie. filter_rewrite_tag) which allows us to resume the process and invoke any remaining processors as expected.