Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changefeedccl: Flush frequency negates batching configurations #69161

Closed
miretskiy opened this issue Aug 19, 2021 · 2 comments · Fixed by #71436
Closed

changefeedccl: Flush frequency negates batching configurations #69161

miretskiy opened this issue Aug 19, 2021 · 2 comments · Fixed by #71436
Assignees
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-cdc

Comments

@miretskiy
Copy link
Contributor

miretskiy commented Aug 19, 2021

Changefeeds running without flush frequency default to 5s flush period.
This behavior is undesirable: it completely negates batching configurations in the sink.

In order to "fix" above behavior, the users resort to specifying larger time
periods for resolved=X option.

Ability to somewhat control file sizes, by changing "time" is both confusing
and suboptimal.

We have to fix this issue on the master as well as on 21.1 branch.
On the master, we currently flush every time frontier advances. However, it's possible that
the frontier advances too rapidly. In those cases, we may want to honor MinHighWaterMarkCheckpointAdvance cluster
setting; or perhaps even add a new option.

On 21.1, we have to come up with light, backportable fix.

@miretskiy miretskiy added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-cdc Change Data Capture labels Aug 19, 2021
@blathers-crl blathers-crl bot added the T-cdc label Aug 19, 2021
@stevendanna
Copy link
Collaborator

Related: #61700

@miretskiy
Copy link
Contributor Author

No idea why this was reopened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-cdc
Projects
None yet
4 participants