Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop KPL in enrich-kinesis and sink chunks #633

Closed
benjben opened this issue Jun 21, 2022 · 0 comments
Closed

Drop KPL in enrich-kinesis and sink chunks #633

benjben opened this issue Jun 21, 2022 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@benjben
Copy link
Contributor

benjben commented Jun 21, 2022

KPL is designed to have records constantly sent to the native binary and let it take care of the batching for us, whereas enrich works with chunks already : a chunk (possibly several thousands of events) gets enriched (while the previous chunk is getting sunk) and then gets sunk. With this design our benchmarking has revealed that under high load this is hard to fine-tune KPL so that it's as performant as building PutRecords requests manually (and take advantage of the fact that the records are already grouped).

This requires some refactoring of enrich to have the sinking function defined for a chunk rather than for individual records

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant