You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
KPL is designed to have records constantly sent to the native binary and let it take care of the batching for us, whereas enrich works with chunks already : a chunk (possibly several thousands of events) gets enriched (while the previous chunk is getting sunk) and then gets sunk. With this design our benchmarking has revealed that under high load this is hard to fine-tune KPL so that it's as performant as building PutRecords requests manually (and take advantage of the fact that the records are already grouped).
This requires some refactoring of enrich to have the sinking function defined for a chunk rather than for individual records
The text was updated successfully, but these errors were encountered:
KPL is designed to have records constantly sent to the native binary and let it take care of the batching for us, whereas enrich works with chunks already : a chunk (possibly several thousands of events) gets enriched (while the previous chunk is getting sunk) and then gets sunk. With this design our benchmarking has revealed that under high load this is hard to fine-tune KPL so that it's as performant as building
PutRecords
requests manually (and take advantage of the fact that the records are already grouped).This requires some refactoring of enrich to have the sinking function defined for a chunk rather than for individual records
The text was updated successfully, but these errors were encountered: