changefeedccl: Flush frequency negates batching configurations #69161
Labels
A-cdc
Change Data Capture
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-cdc
Changefeeds running without flush frequency default to 5s flush period.
This behavior is undesirable: it completely negates batching configurations in the sink.
In order to "fix" above behavior, the users resort to specifying larger time
periods for
resolved=X
option.Ability to somewhat control file sizes, by changing "time" is both confusing
and suboptimal.
We have to fix this issue on the master as well as on 21.1 branch.
On the master, we currently flush every time frontier advances. However, it's possible that
the frontier advances too rapidly. In those cases, we may want to honor MinHighWaterMarkCheckpointAdvance cluster
setting; or perhaps even add a new option.
On 21.1, we have to come up with light, backportable fix.
The text was updated successfully, but these errors were encountered: