Propagate stream expiry & GC error from Virtual WAL to walsender #23260
Assignees
Labels
2024.1.1_blocker
2024.1.2_blocker
area/cdcsdk
CDC SDK
jira-originated
kind/enhancement
This is an enhancement of an existing feature
priority/high
High Priority
Comments
yugabyte-ci
added
area/cdcsdk
siddharth2411
added a commit
that referenced
this issue
Jul 24, 2024
Summary: GetConsistentChanges RPC, at the moment, does not propagate any error to the walsender. But, with this diff, we will now propagate the following errors to the walsender as we cannot recover from these errors and the only solution is to create a new CDC stream. - Stream has expired - Intents have been GC'ed The consequence of sending these errors to walsender is it will lead to a crash in walsender, thereby, propagating the error further to the actual CDC client (eg: YugabyteDB connector). Such errors are not to be retried by CDC clients as these are non-recoverable errors as mentioned above. Jira: DB-12188 Test Plan: Jenkins: urgent, test regex: .*CDC.* ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestIntentGC ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestStreamExpiry Reviewers: asrinivasan, skumar, stiwary Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36756
jasonyb
pushed a commit
that referenced
this issue
Jul 24, 2024
Summary: Excluded: 7414711 [#20972] YSQL: Fix new local variable names in PG code 3484013 [#20766] YSQL: Remove time stuff from the GetTableKeyRanges functions. 8b23629 [#18360] YSQL: fix yb_hash_code compared to an out of range constant 4ee3511 [PLAT-13438]: Add support for preview gflags fe5e675 [#23218] YSQL: Closing the backend connection on receiving the error "Database might have been dropped by another user" 0901f6c [doc] Default databases (#23261) afaf353 [PLAT-14722]Upgrade YBC client and server versio to 2.2.0.0-b3 fdffc33 [#13358] YSQL: DDL Atomicity Stress Test f0b3c46 [ #21499] YSQL: Skipping the test with connection manager as they are unsuitable 5ed864d [#23260] CDCSDK: Propagate stream expiry & GC error to walsender e37d791 [PLAT-14627][PLAT-14729][PLAT-14726]Process group-based specific gflags comprehensively across multiple callsites in YBA 0e17c97 [PLAT-14737]Missing/incorrect universe status loader positions on dashboard page 6477bad [PLAT-12434] YNP fix XDG_RUNTIME_DIR to be set in .bashrc for non login shells f3ffb18 [docs] Tablet metadata and ASH updates (#23221) Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Subscribers: yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36814
siddharth2411
added a commit
that referenced
this issue
Jul 25, 2024
…r to walsender Summary: **Backport Description:** Minor conflict due to missing test **Original Description:** Original commit: 5ed864d / D36756 GetConsistentChanges RPC, at the moment, does not propagate any error to the walsender. But, with this diff, we will now propagate the following errors to the walsender as we cannot recover from these errors and the only solution is to create a new CDC stream. - Stream has expired - Intents have been GC'ed The consequence of sending these errors to walsender is it will lead to a crash in walsender, thereby, propagating the error further to the actual CDC client (eg: YugabyteDB connector). Such errors are not to be retried by CDC clients as these are non-recoverable errors as mentioned above. Jira: DB-12188 Test Plan: Jenkins: urgent, test regex: .*CDC.* ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestIntentGC ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestStreamExpiry Reviewers: asrinivasan, skumar, stiwary Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36758
siddharth2411
added a commit
that referenced
this issue
Aug 7, 2024
…to walsender Summary: **Backport description:** No merge conflicts **Original description:** Original commit: 5ed864d / D36756 GetConsistentChanges RPC, at the moment, does not propagate any error to the walsender. But, with this diff, we will now propagate the following errors to the walsender as we cannot recover from these errors and the only solution is to create a new CDC stream. - Stream has expired - Intents have been GC'ed The consequence of sending these errors to walsender is it will lead to a crash in walsender, thereby, propagating the error further to the actual CDC client (eg: YugabyteDB connector). Such errors are not to be retried by CDC clients as these are non-recoverable errors as mentioned above. Jira: DB-12188 Test Plan: Jenkins: urgent, test regex: .*CDC.* ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestIntentGC ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestStreamExpiry Reviewers: asrinivasan, skumar, stiwary Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D37092
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-12188
The text was updated successfully, but these errors were encountered: