Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sink shouldn't ignore destination table's ingestion batching policy #217

Closed
yihezkel opened this issue Jan 16, 2022 · 2 comments · Fixed by #232
Closed

Sink shouldn't ignore destination table's ingestion batching policy #217

yihezkel opened this issue Jan 16, 2022 · 2 comments · Fixed by #232

Comments

@yihezkel
Copy link
Member

Is your feature request related to a problem? Please describe.
If the table ingesting to has an ingestion policy, it is ignored because the Connector creates and first ingests to a temp table. This table serves as the bottleneck in this regard because it is not created with the same batching policy as the target table, which it should.
If there was a database-level batching policy, then the temp table would inherit it. But this isn't always feasible.
Additionally, the Connector provides a config KUSTO_CLIENT_BATCHING_LIMIT, but this only impact how often data is fed to the service, and not the service's ingestion policy.

Describe the solution you'd like
When the temp table is created, it should inherit the target table's ingestion policy.

@yihezkel yihezkel changed the title Allow custom time-based ingestion batching policy in Sink Sink shouldn't ignore destination table's ingestion batching policy Jan 16, 2022
@epa095
Copy link

epa095 commented Feb 17, 2022

This tripped me up as well! Let me add that if somehow fixing this is hard then at least add it to the documentation, and explain that setting a ingestionpolicy on the database is a workaround (when feasable).

@yihezkel
Copy link
Member Author

We should just make the temp table's ingestion batching policy very low, like 5s, regardless of the destination table's batching policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants