changefeedccl: Support non-retryable errors in sinks #77549
Assignees
Labels
A-cdc
Change Data Capture
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-cdc
Comments
miretskiy
added
C-enhancement
|
cc @cockroachdb/cdc |
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this issue
Oct 27, 2022
Prior to this PR, changefeeds would rely on a white list approach in order to determine which errors were retryable. All other errors would be deemed terminal, causing changefeed to fail. The above approach is brittle, and causes unwanted changefeed termination. This PR changes this approach to treat all errors as retryable, unless otherwise indicated. Errors that are known by changefeed to be fatal are handled explicitly, by marking such errors as terminal. For example, changefeeds would exit if the targetted table is dropped. On the other hand, inability to read this table for any reason would not be treated as terminal. Fixes cockroachdb#90320 Fixes cockroachdb#77549 Fixes cockroachdb#63317 Fixes cockroachdb#71341 Fixes cockroachdb#73016 Informs CRDB-6788 Release note (enterprise change): Changefeed will now treat all errors, unless otherwise indicated, as retryable errors.
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this issue
Nov 2, 2022
Prior to this PR, changefeeds would rely on a white list approach in order to determine which errors were retryable. All other errors would be deemed terminal, causing changefeed to fail. The above approach is brittle, and causes unwanted changefeed termination. This PR changes this approach to treat all errors as retryable, unless otherwise indicated. Errors that are known by changefeed to be fatal are handled explicitly, by marking such errors as terminal. For example, changefeeds would exit if the targetted table is dropped. On the other hand, inability to read this table for any reason would not be treated as terminal. Fixes cockroachdb#90320 Fixes cockroachdb#77549 Fixes cockroachdb#63317 Fixes cockroachdb#71341 Fixes cockroachdb#73016 Informs CRDB-6788 Informs CRDB-7581 Release note (enterprise change): Changefeed will now treat all errors, unless otherwise indicated, as retryable errors.
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this issue
Nov 3, 2022
Prior to this PR, changefeeds would rely on a white list approach in order to determine which errors were retryable. All other errors would be deemed terminal, causing changefeed to fail. The above approach is brittle, and causes unwanted changefeed termination. This PR changes this approach to treat all errors as retryable, unless otherwise indicated. Errors that are known by changefeed to be fatal are handled explicitly, by marking such errors as terminal. For example, changefeeds would exit if the targetted table is dropped. On the other hand, inability to read this table for any reason would not be treated as terminal. Fixes cockroachdb#90320 Fixes cockroachdb#77549 Fixes cockroachdb#63317 Fixes cockroachdb#71341 Fixes cockroachdb#73016 Informs CRDB-6788 Informs CRDB-7581 Release note (enterprise change): Changefeed will now treat all errors, unless otherwise indicated, as retryable errors.
HonoreDB
pushed a commit
to HonoreDB/cockroach
that referenced
this issue
Nov 4, 2022
Prior to this PR, changefeeds would rely on a white list approach in order to determine which errors were retryable. All other errors would be deemed terminal, causing changefeed to fail. The above approach is brittle, and causes unwanted changefeed termination. This PR changes this approach to treat all errors as retryable, unless otherwise indicated. Errors that are known by changefeed to be fatal are handled explicitly, by marking such errors as terminal. For example, changefeeds would exit if the targetted table is dropped. On the other hand, inability to read this table for any reason would not be treated as terminal. Fixes cockroachdb#90320 Fixes cockroachdb#77549 Fixes cockroachdb#63317 Fixes cockroachdb#71341 Fixes cockroachdb#73016 Informs CRDB-6788 Informs CRDB-7581 Release note (enterprise change): Changefeed will now treat all errors, unless otherwise indicated, as retryable errors.
craig bot
pushed a commit
that referenced
this issue
Nov 6, 2022
90810: changefeedccl: Rework error handling r=miretskiy a=miretskiy Prior to this PR, changefeeds would rely on a white list approach in order to determine which errors were retryable. All other errors would be deemed terminal, causing changefeed to fail. The above approach is brittle, and causes unwanted changefeed termination. This PR changes this approach to treat all errors as retryable, unless otherwise indicated. Errors that are known by changefeed to be fatal are handled explicitly, by marking such errors as terminal. For example, changefeeds would exit if the targeted table is dropped. On the other hand, inability to read this table for any reason would not be treated as terminal. Fixes #90320 Fixes #77549 Fixes #63317 Fixes #71341 Fixes #73016 Informs CRDB-6788 Informs CRDB-7581 Release Note (enterprise change): Changefeed will now treat all errors, unless otherwise indicated, as retryable errors. Co-authored-by: Yevgeniy Miretskiy <[email protected]>
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this issue
Dec 6, 2022
Prior to this PR, changefeeds would rely on a white list approach in order to determine which errors were retryable. All other errors would be deemed terminal, causing changefeed to fail. The above approach is brittle, and causes unwanted changefeed termination. This PR changes this approach to treat all errors as retryable, unless otherwise indicated. Errors that are known by changefeed to be fatal are handled explicitly, by marking such errors as terminal. For example, changefeeds would exit if the targetted table is dropped. On the other hand, inability to read this table for any reason would not be treated as terminal. Fixes cockroachdb#90320 Fixes cockroachdb#77549 Fixes cockroachdb#63317 Fixes cockroachdb#71341 Fixes cockroachdb#73016 Informs CRDB-6788 Informs CRDB-7581 Release note (enterprise change): Changefeed will now treat all errors, unless otherwise indicated, as retryable errors.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, any error returned form the sink is treated as a retryable error
because we wrap the sink with "error sink".
This may not be desirable at all times. For example, if we somehow managed
to generate 2GB size message in kafka, there is simply no way we will be able
to send such a message because it will be rejected by sarama. There are other
cases like this, I suppose.
Jira issue: CRDB-13648
The text was updated successfully, but these errors were encountered: