Backup and Restore support for Colocated tables with Tablespaces #23312
Assignees
Labels
jira-originated
kind/enhancement
This is an enhancement of an existing feature
priority/high
High Priority
Comments
yugabyte-ci
added
jira-originated
kind/enhancement
utkarsh-um-yb
added a commit
that referenced
this issue
Sep 18, 2024
…pace information. Summary: Previously, backup and restore operations for colocated tables failed when multiple tablespaces were present, and the `--use_tablespaces` option was not used. The issue arose because each tablespace could only contain one tablegroup, leading to conflicts during the restore process. Without tablespace information, we could only restore tables to a single tablegroup (corresponding to the default tablespace). However, since a tablegroup represents a (parent) colocated tablet, restoring multiple colocated tablets required the same number of tablegroups. The revision resolves this issue by enabling multiple tablegroups in the default tablespace during restoration. This is achieved through the following steps: * Originally, the tablegroup’s name is `colocation_<tablespace_oid>`, but since the concept of tablespaces doesn’t apply after restoration, we rename the tablegroup as `colocation_restore_<tablegroup_oid>`. The default tablegroup's name remains unchanged after the restore process. * To retain the original name of the default tablegroup, we track which tables are associated with it during the backup. This allows us to correctly restore the tablegroup's properties. **Implementation** This update enhances the backup and restore functionality for colocated tables with tablespaces by addressing the issue as follows: - **Backup Phase**: The backup process now tags each table as either belonging to the default tablegroup or not, along with the tablegroup ID in YSQLDump. The tablegroup name, originally formatted as `colocation_<tablespace_oid>`, will be renamed during restoration as described above. - **Restore Phase**: During restoration, the tablegroup name and its association with the default tablegroup are retrieved from YSQLDump. If the corresponding tablegroup doesn’t already exist, it is created in the default tablespace, allowing multiple tablegroups to coexist without conflicts. - **ALTER Table**: To move a tablegroup to another tablespace after restoration, the syntax: ```ALTER TABLE ALL IN TABLESPACE pg_default COLOCATED WITH <table_name> SET TABLESPACE <new_tablespace_name> CASCADE;``` should be used, ensuring proper management of tablegroups and tablespaces. DB-12237 Test Plan: == Automated == ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithTablespaces ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithoutTablespaces ./yb_build.sh --java-test org.yb.pgsql.TestYsqlDump#ysqlDumpColocatedTablesWithTablespaces 1. **YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithoutTablespaces**: This test case increases the number of tablespaces (and consequently the number of tablegroups) to replicate the scenario where multiple tablespaces exist. It validates that backup and restore work correctly when performed without the `--use_tablespaces` flag. This is the primary case that this revision addresses. 2. **YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithTablespaces**: To ensure consistency, this test also increases the number of tablespaces but verifies that backup and restore function correctly when using the `--use_tablespaces` flag. 3. **org.yb.pgsql.TestYsqlDump#ysqlDumpColocatedTablesWithTablespaces**: This test checks that the YsqlDump output matches the expected format. Specifically, in the case of the fix, the dump file should include the line: ```SELECT pg_catalog.binary_upgrade_set_next_tablegroup_default(true);``` Reviewers: skumar, yguan, aagrawal Reviewed By: skumar Subscribers: svc_phabricator, mlillibridge, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D36859
foucher
added a commit
that referenced
this issue
Sep 20, 2024
Summary: 6b34652 [doc] Update badge colors (#23985) 6dcfce6 [PLAT-15297][dr] GET /configs fails with java.lang.IllegalStateException: DrConfig iTest-dr(uuid) does not have any corresponding xCluster config 7995311 [#23695] docdb: Improve help strings for tablet replica limit flags. 7659151 [DOC-479] Voyager 1.8.1 updates (#23983) 56c9cc9 [PLAT-15282]: fix NPE on k8s operator backup/restore status update c8776b5 [docs] SSL/TLS configuration with hashicorp vault YSQL plugin (#23979) Excluded: d3dd4f7 Bumping version to 2.25.0.0 on branch master Excluded: 0dbe7d6 [#23312] YSQL: Fix backup/restore for colocated tables without tablespace information. 2beb872 [#23786] YSQL: add yb_make_next_ddl_statement_nonincrementing to YbDbAdminVariables 28d0d8e [#24011] build: fix ./yb_build.sh --gcc13 9feb6e8 [#24012] YSQL: Replace deprecated function shared_ptr::unique to fix macos compiler error 2cc1480 [PLAT-15349] Fix update_lb_config automated NLB updates af4c46d [#23741] docdb: Add parameter to run PITR tests with clone instead 0103dfc [PLAT-14459] Continuous backups for s3 e6bb890 [PLAT-15192] [PLAT-15191] Document node-agent-provision.yaml 1552321 [PLAT-15205] YBA callhome improvements ba6acbd [PLAT-15342] Install Node Agent as root level unit for ynp 2478a50 [PLAT-13058] Remove references to public key filepath, since we don't need ssh keys for node agent based provisioning aa14f76 [#23975] YSQL, ASH: Fix TestYbAsh#testPgAuxInfo 885b72b [#23960] DocDB: Fix WriteBuffer crash in move ctor Excluded: f4a002a [#23314] YSQL: Fix variable name to yb_ for ATPrepSetTableSpace 29c56ed [PLAT-15369][YBA CLI]Backup list-increments doesn't support pretty/json outputs 784160c [PLAT-15370][YBA CLI]backup list table output contains base_backup_uuid and backup_uuid b32222d [PLAT-15280][PLAT-14753] - Pg parity improvements and other ui fixes c6521cf [PLAT-15368][YBA CLI]Backup list print "No more backup present" after printing all the backups in -o pretty/json - pg_catalog_version-test.cc - NonIncrementingDDLMode: YB master commit 2beb872 sets the role to yb_db_admin, which due to new behavior in upstream PG commit b073c3ccd06e4cb845e121387a43faa8c68a7b62 (documented in https://www.postgresql.org/docs/15/ddl-schemas.html#DDL-SCHEMAS-PATTERNS) causes the following unintended test failure: ERROR: permission denied for schema public. Add a workaround to grant create permission to the role now set. Test Plan: Jenkins: urgent, rebase: pg15-cherrypicks Reviewers: jason, jenkins-bot Differential Revision: https://phorge.dev.yugabyte.com/D38220
utkarsh-um-yb
added a commit
that referenced
this issue
Sep 21, 2024
…es without tablespace information. Summary: Backport Description: No merge conflicts Original Description: Original commit: 0dbe7d6 / D36859 Previously, backup and restore operations for colocated tables failed when multiple tablespaces were present, and the `--use_tablespaces` option was not used. The issue arose because each tablespace could only contain one tablegroup, leading to conflicts during the restore process. Without tablespace information, we could only restore tables to a single tablegroup (corresponding to the default tablespace). However, since a tablegroup represents a (parent) colocated tablet, restoring multiple colocated tablets required the same number of tablegroups. The revision resolves this issue by enabling multiple tablegroups in the default tablespace during restoration. This is achieved through the following steps: * Originally, the tablegroup’s name is `colocation_<tablespace_oid>`, but since the concept of tablespaces doesn’t apply after restoration, we rename the tablegroup as `colocation_restore_<tablegroup_oid>`. The default tablegroup's name remains unchanged after the restore process. * To retain the original name of the default tablegroup, we track which tables are associated with it during the backup. This allows us to correctly restore the tablegroup's properties. **Implementation** This update enhances the backup and restore functionality for colocated tables with tablespaces by addressing the issue as follows: - **Backup Phase**: The backup process now tags each table as either belonging to the default tablegroup or not, along with the tablegroup ID in YSQLDump. The tablegroup name, originally formatted as `colocation_<tablespace_oid>`, will be renamed during restoration as described above. - **Restore Phase**: During restoration, the tablegroup name and its association with the default tablegroup are retrieved from YSQLDump. If the corresponding tablegroup doesn’t already exist, it is created in the default tablespace, allowing multiple tablegroups to coexist without conflicts. - **ALTER Table**: To move a tablegroup to another tablespace after restoration, the syntax: ```ALTER TABLE ALL IN TABLESPACE pg_default COLOCATED WITH <table_name> SET TABLESPACE <new_tablespace_name> CASCADE;``` should be used, ensuring proper management of tablegroups and tablespaces. DB-12237 Test Plan: == Automated == ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithTablespaces ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithoutTablespaces ./yb_build.sh --java-test org.yb.pgsql.TestYsqlDump#ysqlDumpColocatedTablesWithTablespaces 1. **YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithoutTablespaces**: This test case increases the number of tablespaces (and consequently the number of tablegroups) to replicate the scenario where multiple tablespaces exist. It validates that backup and restore work correctly when performed without the `--use_tablespaces` flag. This is the primary case that this revision addresses. 2. **YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithTablespaces**: To ensure consistency, this test also increases the number of tablespaces but verifies that backup and restore function correctly when using the `--use_tablespaces` flag. 3. **org.yb.pgsql.TestYsqlDump#ysqlDumpColocatedTablesWithTablespaces**: This test checks that the YsqlDump output matches the expected format. Specifically, in the case of the fix, the dump file should include the line: ```SELECT pg_catalog.binary_upgrade_set_next_tablegroup_default(true);``` Reviewers: skumar, aagrawal, yguan Reviewed By: skumar Subscribers: ybase, yql, mlillibridge, svc_phabricator Differential Revision: https://phorge.dev.yugabyte.com/D38205
utkarsh-um-yb
added a commit
that referenced
this issue
Sep 24, 2024
…cated tables without tablespace information. Summary: Previously, backup and restore operations for colocated tables failed when multiple tablespaces were present, and the `--use_tablespaces` option was not used. The issue arose because each tablespace could only contain one tablegroup, leading to conflicts during the restore process. Without tablespace information, we could only restore tables to a single tablegroup (corresponding to the default tablespace). However, since a tablegroup represents a (parent) colocated tablet, restoring multiple colocated tablets required the same number of tablegroups. The revision resolves this issue by enabling multiple tablegroups in the default tablespace during restoration. This is achieved through the following steps: * Originally, the tablegroup’s name is `colocation_<tablespace_oid>`, but since the concept of tablespaces doesn’t apply after restoration, we rename the tablegroup as `colocation_restore_<tablegroup_oid>`. The default tablegroup's name remains unchanged after the restore process. * To retain the original name of the default tablegroup, we track which tables are associated with it during the backup. This allows us to correctly restore the tablegroup's properties. **Implementation** This update enhances the backup and restore functionality for colocated tables with tablespaces by addressing the issue as follows: - **Backup Phase**: The backup process now tags each table as either belonging to the default tablegroup or not, along with the tablegroup ID in YSQLDump. The tablegroup name, originally formatted as `colocation_<tablespace_oid>`, will be renamed during restoration as described above. - **Restore Phase**: During restoration, the tablegroup name and its association with the default tablegroup are retrieved from YSQLDump. If the corresponding tablegroup doesn’t already exist, it is created in the default tablespace, allowing multiple tablegroups to coexist without conflicts. - **ALTER Table**: To move a tablegroup to another tablespace after restoration, the syntax: ```ALTER TABLE ALL IN TABLESPACE pg_default COLOCATED WITH <table_name> SET TABLESPACE <new_tablespace_name> CASCADE;``` should be used, ensuring proper management of tablegroups and tablespaces. DB-12237 Original commit: 0dbe7d6 / D36859 - src/postgres/src/backend/commands/indexcmds.c - DefineIndex, in if (IsYugabyteEnabled()) block after is_colocated assignment: - This YB master commit adjusts indentation of braces. - pg15 initial merge 55782d5 replaced the block with assignment of tablegroupId. - Take pg15 change. Note some diff tools may resolve this automatically because the master change is whitespace-only. - DefineIndex, after declaration of is_colocated_tables_with_tablespace_enabled: - This YB master commit adds logic, and fixes indentation of braces. - pg15 merge commit 4e9241b indents logic around if (is_colocated_tables_with_tablespace_enabled && is_colocated_via_database && !MyColocatedDatabaseLegacy) to be inside IsYugabyteEnabled(). - Take the if (is_colocated_tables_with_tablespace_enabled && is_colocated_via_database && !MyColocatedDatabaseLegacy) condition first. Then pull the entire block beginning char *tablegroup_name = NULL;, adjusting the indentation, up to the end of the block for the OidIsValid(tablespaceId) condition. Then take the master change of the remaining conflict, but indent it to be correctly indented within the IsYugabyteEnabled(). - src/postgres/src/backend/utils/adt/ruleutils.c - pg_get_indexdef_worker - This YB master commit removed piece of code. - YB Pg15 commit 55782d5 changed heap_open (this was present in code which was removed in this master commit) to table_open. - Resolved by accepting incoming master commit hence completely removing piece of code. - src/postgres/src/bin/pg_dump/pg_dump.c - getYbTablePropertiesAndReloptions: - This YB master commit introduces new piece of code. - YB Pg15 commit 55782d5, changed exit_horribly to fatal and introduced #ifdef YB_TODO and #endif - Resolved conflict by accepting a combination of both the changes, by keeping it fatal with ifdef and then introducing new code snippet - yb_ysql_dump_colocated_tables_with_tablespaces.data.sql - Upstream PG commit 9a974cbcba005256a19991203583a94b4f9a21a9 preserves relfilenodes. - Upstream PG commit 9fbc3f318d039c3e1e8614c38e40843cf8fcffde adds dump of pg_stat_statements_info. YB role-related code added in YB master commit 5659b73. - Other minor PG15 changes. - yb_ysql_dump_describe_colocated_tables_with_tablespaces.out - Change is likely due to changes in the execution plan, similar to schema ordering differences found in YB pg15 ab32fa2 Test Plan: == Automated == ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithTablespaces ./yb_build.sh --cxx-test yb-backup-cross-feature-test --gtest_filter YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithoutTablespaces ./yb_build.sh --java-test org.yb.pgsql.TestYsqlDump#ysqlDumpColocatedTablesWithTablespaces 1. **YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithoutTablespaces**: This test case increases the number of tablespaces (and consequently the number of tablegroups) to replicate the scenario where multiple tablespaces exist. It validates that backup and restore work correctly when performed without the `--use_tablespaces` flag. This is the primary case that this revision addresses. 2. **YBBackupTestColocatedTablesWithTablespaces.TestBackupColocatedTablesWithTablespaces**: To ensure consistency, this test also increases the number of tablespaces but verifies that backup and restore function correctly when using the `--use_tablespaces` flag. 3. **org.yb.pgsql.TestYsqlDump#ysqlDumpColocatedTablesWithTablespaces**: This test checks that the YsqlDump output matches the expected format. Specifically, in the case of the fix, the dump file should include the line: ```SELECT pg_catalog.binary_upgrade_set_next_tablegroup_default(true);``` Reviewers: jason, tfoucher Reviewed By: tfoucher Subscribers: ybase, yql, mlillibridge, svc_phabricator Differential Revision: https://phorge.dev.yugabyte.com/D38269
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-12237
The text was updated successfully, but these errors were encountered: