You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
If the table ingesting to has an ingestion policy, it is ignored because the Connector creates and first ingests to a temp table. This table serves as the bottleneck in this regard because it is not created with the same batching policy as the target table, which it should.
If there was a database-level batching policy, then the temp table would inherit it. But this isn't always feasible.
Additionally, the Connector provides a config KUSTO_CLIENT_BATCHING_LIMIT, but this only impact how often data is fed to the service, and not the service's ingestion policy.
Describe the solution you'd like
When the temp table is created, it should inherit the target table's ingestion policy.
The text was updated successfully, but these errors were encountered:
yihezkel
changed the title
Allow custom time-based ingestion batching policy in Sink
Sink shouldn't ignore destination table's ingestion batching policy
Jan 16, 2022
This tripped me up as well! Let me add that if somehow fixing this is hard then at least add it to the documentation, and explain that setting a ingestionpolicy on the database is a workaround (when feasable).
Is your feature request related to a problem? Please describe.
If the table ingesting to has an ingestion policy, it is ignored because the Connector creates and first ingests to a temp table. This table serves as the bottleneck in this regard because it is not created with the same batching policy as the target table, which it should.
If there was a database-level batching policy, then the temp table would inherit it. But this isn't always feasible.
Additionally, the Connector provides a config
KUSTO_CLIENT_BATCHING_LIMIT
, but this only impact how often data is fed to the service, and not the service's ingestion policy.Describe the solution you'd like
When the temp table is created, it should inherit the target table's ingestion policy.
The text was updated successfully, but these errors were encountered: