Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processor: inline re-ingestion implementation #7455

Merged
merged 15 commits into from
May 25, 2023

Conversation

leonardo-albertovich
Copy link
Collaborator

This PR adds the concept of processor stages which are just an index that's used to skip the stages that the record has already gone through when re-ingesting them (ie. filter_rewrite_tag) which allows us to resume the process and invoke any remaining processors as expected.

This adds the concept of processor unit stage index which is used when
a filter such as rewrite_tag needs to re-ingest a chunk.

In this case, a call to flb_input_log_append_ex is made with the
processor stage index corresponding to the next processor unit in
the processor stack.

This allows us to overcome the limitation that comes with ingesting
the logs using the synthetic emitter input plugin.

Signed-off-by: Leonardo Alminana <[email protected]>
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 22, 2023 16:17 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 22, 2023 16:17 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 22, 2023 16:17 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 22, 2023 16:41 — with GitHub Actions Inactive
include/fluent-bit/flb_filter.h Outdated Show resolved Hide resolved
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 23, 2023 15:30 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 23, 2023 15:30 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 23, 2023 15:30 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 23, 2023 15:50 — with GitHub Actions Inactive
Copy link
Contributor

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed that this PR works as expected with my using rewrite_tag configuration:

pipeline:
    inputs:
      - name: dummy
        tag: test-tag.raw
        dummy: '{"tool": "fluent", "sub": {"s1": {"s2": "bit"}}}'

        processors:
          logs:
            - name: modify
              match: test-tag.*
              add: hostname monox
            - name: rewrite_tag
              match: test-tag.*
              rule: $tool ^(fluent)$  from.$TAG.new.$tool.$sub['s1']['s2'].out false
              emitter_name: k8s_meta_into_tag_partial_all
            - name: lua
              match: from.test-tag.raw.*
              call: append_tag
              code: |
                function append_tag(tag, timestamp, record)
                  new_record = record
                  new_record["tag"] = tag
                  return 1, timestamp, new_record
                end

    outputs:
      - name: stdout
        match: '*'

        processors:
          logs:
            - name: lua
              call: add_field
              code: |
                  function add_field(tag, timestamp, record)
                     new_record = record
                     new_record["output"] = "new data"
                     return 1, timestamp, new_record
                  end

I commented a some of nitpicks for naming function which work with processor stage.

include/fluent-bit/flb_input_log.h Outdated Show resolved Hide resolved
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 24, 2023 09:35 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 24, 2023 09:35 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 24, 2023 09:35 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich temporarily deployed to pr May 24, 2023 09:56 — with GitHub Actions Inactive
@edsiper edsiper merged commit 3446ace into master May 25, 2023
@edsiper edsiper deleted the leonardo-master-processor-inline-ingestion branch May 25, 2023 05:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants