You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently input plugins run on a schedule and are called once per second. This is not ideal for dealing with streaming / event driven data, as that data may reach the input plugin at an uneven rate. Events should ideally be collected as soon as they are available (I believe this is how other plugins work, like the tail plugin).
With the current implementation, the fastest rate data chunks could be consumed would be every second.
I believe this would require a wrapper of flb_input_set_collector_event in the plugin API, however from what I can tell, Fluent Bit core would probably need to be updated as well since the proxy plugin which is used to wrap Go plugins is responsible for setting this timer, i.e. by calling flb_input_set_collector_time (here: https://github.com/fluent/fluent-bit/blob/master/src/flb_plugin_proxy.c#L151-L155).
The text was updated successfully, but these errors were encountered:
ccampo133
changed the title
[feature-request] Add stream based processing for input plugins
[feature-request] Add event based processing for input plugins
Feb 9, 2023
Currently input plugins run on a schedule and are called once per second. This is not ideal for dealing with streaming / event driven data, as that data may reach the input plugin at an uneven rate. Events should ideally be collected as soon as they are available (I believe this is how other plugins work, like the
tail
plugin).With the current implementation, the fastest rate data chunks could be consumed would be every second.
I believe this would require a wrapper of
flb_input_set_collector_event
in the plugin API, however from what I can tell, Fluent Bit core would probably need to be updated as well since the proxy plugin which is used to wrap Go plugins is responsible for setting this timer, i.e. by callingflb_input_set_collector_time
(here: https://github.com/fluent/fluent-bit/blob/master/src/flb_plugin_proxy.c#L151-L155).The text was updated successfully, but these errors were encountered: