Logical Replication Support for Table Rewrite DDLs Before Slot Creation #23768
Assignees
Labels
area/ysql
Yugabyte SQL (YSQL)
jira-originated
kind/bug
This issue is a bug
priority/high
High Priority
Comments
yugabyte-ci
added
jira-originated
kind/bug
utkarsh-um-yb
added a commit
that referenced
this issue
Sep 20, 2024
Summary: Previously, when a table rewrite DDL operation (such as `ALTER COLUMN TYPE`, `ADD/DROP PRIMARY KEY` or `TRUNCATE TABLE`) was performed before creating a replication slot, the changes from that table were not streamed during logical replication. The issue arose from the inconsistent handling of DocDB's UUID for a table's ID and PG’s `table_oid`. Normally, the table’s UUID in DocDB matches the `table_oid` in PG. However, in the case of table rewrites, while PG’s `table_oid` remains the same, the DocDB UUID changes. After such rewrites, the `relfilenode` in PG becomes the one-to-one mapping to the new UUID in DocDB (commit: bb77fb2). For clarity, the following terminology will be used throughout this summary: - `table_id`: Refers to the UUID of the underlying DocDB table. - `pg_table_id`: Refers to the original UUID of the DocDB table before the table rewrite. - `table_oid`: Represents PG’s table OID. - `relfilenode`: Corresponds to the relfilenode of the PG table, which maps to DocDB’s UUID. It’s important to note that without any table rewrites, `table_oid` is equal to `relfilenode`. However, with table rewrites, `relfilenode` reflects the new underlying table, while `table_oid` remains unchanged. The following key changes were made to address the issue in this revision: 1. In `YBCInitVirtualWalForCDC`, `table_id` is now computed using `relfilenode` instead of `table_oid` to ensure the correct `table_id` is passed to CDC Service. 2. The `GetConsistentChanges` response now treats the `table_id` as `relfilenode` and accurately retrieves `table_oid` to be used within PG. Previously, `table_id` was directly treated as `table_oid`, causing inconsistencies. 3. Similarly, when fetching the `replica_identity_map` (which maps table_id to its replica identity), the `table_id` was previously used directly to obtain the `table_oid`. Now, it is treated as `relfilenode`, and the `table_oid` is retrieved accordingly. JIRA: DB-12672 Test Plan: == Automated Testing == ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithAlterType ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithDropAddPk ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithTruncate Reviewers: skumar, sumukh.phalgaonkar, stiwary, siddharth.shah, asrinivasan, xCluster, hsunder Reviewed By: skumar, stiwary Subscribers: yql, ybase, ycdcxcluster Differential Revision: https://phorge.dev.yugabyte.com/D37721
utkarsh-um-yb
added a commit
that referenced
this issue
Sep 23, 2024
…ation Summary: Backport Description: No merge conflicts Original Description: Original commit: 5e86515 / D37721 Previously, when a table rewrite DDL operation (such as `ALTER COLUMN TYPE`, `ADD/DROP PRIMARY KEY` or `TRUNCATE TABLE`) was performed before creating a replication slot, the changes from that table were not streamed during logical replication. The issue arose from the inconsistent handling of DocDB's UUID for a table's ID and PG’s `table_oid`. Normally, the table’s UUID in DocDB matches the `table_oid` in PG. However, in the case of table rewrites, while PG’s `table_oid` remains the same, the DocDB UUID changes. After such rewrites, the `relfilenode` in PG becomes the one-to-one mapping to the new UUID in DocDB (commit: bb77fb2). For clarity, the following terminology will be used throughout this summary: - `table_id`: Refers to the UUID of the underlying DocDB table. - `pg_table_id`: Refers to the original UUID of the DocDB table before the table rewrite. - `table_oid`: Represents PG’s table OID. - `relfilenode`: Corresponds to the relfilenode of the PG table, which maps to DocDB’s UUID. It’s important to note that without any table rewrites, `table_oid` is equal to `relfilenode`. However, with table rewrites, `relfilenode` reflects the new underlying table, while `table_oid` remains unchanged. The following key changes were made to address the issue in this revision: 1. In `YBCInitVirtualWalForCDC`, `table_id` is now computed using `relfilenode` instead of `table_oid` to ensure the correct `table_id` is passed to CDC Service. 2. The `GetConsistentChanges` response now treats the `table_id` as `relfilenode` and accurately retrieves `table_oid` to be used within PG. Previously, `table_id` was directly treated as `table_oid`, causing inconsistencies. 3. Similarly, when fetching the `replica_identity_map` (which maps table_id to its replica identity), the `table_id` was previously used directly to obtain the `table_oid`. Now, it is treated as `relfilenode`, and the `table_oid` is retrieved accordingly. JIRA: DB-12672 Test Plan: == Automated Testing == ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithAlterType ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithDropAddPk ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithTruncate Reviewers: skumar, sumukh.phalgaonkar, stiwary, siddharth.shah, asrinivasan Reviewed By: stiwary Subscribers: ycdcxcluster, ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D38306
foucher
pushed a commit
that referenced
this issue
Sep 24, 2024
Summary: 5d3e83e [PLAT-15199] Change TP API URLs according to latest refactoring a50a730 [doc][yba] YBDB compatibility (#23984) 0c84dbe [#24029] Update the callhome diagnostics not to send gflags details. b53ed3a [PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule f0eab8f [PLAT-15278]: Fix DB Scoped XCluster replication restart 344bc76 Revert "[PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule" 3628ba7 [PLAT-14459] Swagger fix bb93ebe [#24021] YSQL: Add --TEST_check_catalog_version_overflow 9ab7806 [#23927] docdb: Add gflag for minimum thread stack size Excluded: 8c8adc0 [#18822] YSQL: Gate update optimizations behind preview flag 5e86515 [#23768] YSQL: Fix table rewrite DDL before slot creation 123d496 [PLAT-14682] Universe task should only unlock itself and make unlock aware of the lock config de9d4ad [doc][yba] CIS hardened OS support (#23789) e131b20 [#23998] DocDB: Update usearch and other header-only third-party dependencies 1665662 Automatic commit by thirdparty_tool: update usearch to commit 240fe9c298100f9e37a2d7377b1595be6ba1f412. 3adbdae Automatic commit by thirdparty_tool: update fp16 to commit 98b0a46bce017382a6351a19577ec43a715b6835. 9a819f7 Automatic commit by thirdparty_tool: update hnswlib to commit 2142dc6f4dd08e64ab727a7bbd93be7f732e80b0. 2dc58f4 Automatic commit by thirdparty_tool: update simsimd to tag v5.1.0. 9a03432 [doc][ybm] Azure private link host (#24086) 039c9a2 [#17378] YSQL: Testing for histogram_bounds in pg_stats 09f7a0f [#24085] DocDB: Refactor HNSW wrappers 555af7d [#24000] DocDB: Shutting down shared exchange could cause TServer to hang 5743a03 [PLAT-15317]Alert emails are not in the correct format. 8642555 [PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule 253ab07 [PLAT-15400][PLAT-15401][PLAT-13051] - Connection pooling ui issues and other ui issues 57576ae [#16487] YSQL: Fix flakey TestPostgresPid test bc8ae45 Update ports for CIS hardened (#24098) 6fa33e6 [#18152, #18729] Docdb: Fix test TestPgIndexSelectiveUpdate cc6d2d1 [docs] added and updated cves (#24046) Excluded: ed153dc [#24055] YSQL: fix pg_hint_plan regression with executing prepared statement Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, jenkins-bot Differential Revision: https://phorge.dev.yugabyte.com/D38322
utkarsh-um-yb
added a commit
that referenced
this issue
Oct 8, 2024
…ation Summary: Backport Description: No merge conflicts Original Description: Original commit: 5e86515 / D37721 Previously, when a table rewrite DDL operation (such as `ALTER COLUMN TYPE`, `ADD/DROP PRIMARY KEY` or `TRUNCATE TABLE`) was performed before creating a replication slot, the changes from that table were not streamed during logical replication. The issue arose from the inconsistent handling of DocDB's UUID for a table's ID and PG’s `table_oid`. Normally, the table’s UUID in DocDB matches the `table_oid` in PG. However, in the case of table rewrites, while PG’s `table_oid` remains the same, the DocDB UUID changes. After such rewrites, the `relfilenode` in PG becomes the one-to-one mapping to the new UUID in DocDB (commit: bb77fb2). For clarity, the following terminology will be used throughout this summary: - `table_id`: Refers to the UUID of the underlying DocDB table. - `pg_table_id`: Refers to the original UUID of the DocDB table before the table rewrite. - `table_oid`: Represents PG’s table OID. - `relfilenode`: Corresponds to the relfilenode of the PG table, which maps to DocDB’s UUID. It’s important to note that without any table rewrites, `table_oid` is equal to `relfilenode`. However, with table rewrites, `relfilenode` reflects the new underlying table, while `table_oid` remains unchanged. The following key changes were made to address the issue in this revision: 1. In `YBCInitVirtualWalForCDC`, `table_id` is now computed using `relfilenode` instead of `table_oid` to ensure the correct `table_id` is passed to CDC Service. 2. The `GetConsistentChanges` response now treats the `table_id` as `relfilenode` and accurately retrieves `table_oid` to be used within PG. Previously, `table_id` was directly treated as `table_oid`, causing inconsistencies. 3. Similarly, when fetching the `replica_identity_map` (which maps table_id to its replica identity), the `table_id` was previously used directly to obtain the `table_oid`. Now, it is treated as `relfilenode`, and the `table_oid` is retrieved accordingly. JIRA: DB-12672 Test Plan: == Automated Testing == ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithAlterType ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithDropAddPk ./yb_build.sh --java-test org.yb.pgsql.TestPgReplicationSlot#testWithTruncate Reviewers: skumar, stiwary Reviewed By: stiwary Subscribers: ycdcxcluster, ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D38737
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-12672
The text was updated successfully, but these errors were encountered: