Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature-request] Add event based processing for input plugins #59

Open
ccampo133 opened this issue Feb 9, 2023 · 0 comments
Open

[feature-request] Add event based processing for input plugins #59

ccampo133 opened this issue Feb 9, 2023 · 0 comments

Comments

@ccampo133
Copy link

ccampo133 commented Feb 9, 2023

Currently input plugins run on a schedule and are called once per second. This is not ideal for dealing with streaming / event driven data, as that data may reach the input plugin at an uneven rate. Events should ideally be collected as soon as they are available (I believe this is how other plugins work, like the tail plugin).

With the current implementation, the fastest rate data chunks could be consumed would be every second.

I believe this would require a wrapper of flb_input_set_collector_event in the plugin API, however from what I can tell, Fluent Bit core would probably need to be updated as well since the proxy plugin which is used to wrap Go plugins is responsible for setting this timer, i.e. by calling flb_input_set_collector_time (here: https://github.com/fluent/fluent-bit/blob/master/src/flb_plugin_proxy.c#L151-L155).

@ccampo133 ccampo133 changed the title [feature-request] Add stream based processing for input plugins [feature-request] Add event based processing for input plugins Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant