Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove exponential backoff code from all plugins #23

Closed
PettitWesley opened this issue Mar 9, 2020 · 0 comments · Fixed by aws/aws-for-fluent-bit#42
Closed

Remove exponential backoff code from all plugins #23

PettitWesley opened this issue Mar 9, 2020 · 0 comments · Fixed by aws/aws-for-fluent-bit#42
Labels
bug Something isn't working

Comments

@PettitWesley
Copy link
Contributor

PettitWesley commented Mar 9, 2020

When I built the Firehose and CloudWatch plugins, I thought it would be a good idea for them to implement backoff in case of API throttling. That instinct was good, but the execution shows my lack of understanding of Fluent Bit at the time. Fluent Bit has it's own retry mechanisms; a plugin can return FLB_RETRY and let Fluent Bit handle it. More importantly, the backoff mechanism in use currently causes the plugin to sleep- since Fluent Bit is single-threaded, but concurrent, this sleep stops all other plugins, and thus impacts its overall performance.

In practice I suspect few users trigger throttling; however, this is a very easy thing to fix, we just need to remove a bit of code from all the plugins.

@PettitWesley PettitWesley added the bug Something isn't working label Mar 9, 2020
PettitWesley added a commit to PettitWesley/amazon-kinesis-firehose-for-fluent-bit that referenced this issue May 12, 2020
PettitWesley added a commit to PettitWesley/amazon-kinesis-firehose-for-fluent-bit that referenced this issue May 15, 2020
PettitWesley added a commit that referenced this issue Jun 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant