Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kinesis_streams plugin: add KPL aggregation and gzip compression support #9451

Open
reegnz opened this issue Oct 1, 2024 · 0 comments
Open

Comments

@reegnz
Copy link

reegnz commented Oct 1, 2024

Is your feature request related to a problem? Please describe.
The old kinesis plugin still has some features like gzip compression and KPL aggregation that the kinesis_streams plugin written in C does not have.

This makes migrating from the 3rd party kinesis plugin to the kinesis_streams plugin somewhat difficult.

Describe the solution you'd like
The kinesis_streams plugin should support aggregation of records as well as compression (at least gzip).

Describe alternatives you've considered

Sticking with the kinesis go plugin until it ultimately becomes unmaintained (it likely already is, no new commits or releases since 18 months and they point to the upstream kinesis_streams plugin as the alternative).

Additional context

We are using fluent-bit to collect application logs. We use both KPL aggregation and gzip in our log consumption pipeline to improve log throughput. In our current solution we had to monkey-patch the official fluent-bit container image to include the kinesis go plugin. Ideally the KPL and aggregation features should be supported natively by fluent-bit without any customization of the container image however.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant