Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] Read sequences data as of time #24153

Closed
1 task done
yamen-haddad opened this issue Sep 26, 2024 · 0 comments
Closed
1 task done

[DocDB] Read sequences data as of time #24153

yamen-haddad opened this issue Sep 26, 2024 · 0 comments
Assignees
@yamen-haddad
Labels
2024.2_blocker area/docdb YugabyteDB core features kind/bug This issue is a bug priority/high High Priority

Comments

@yamen-haddad
Copy link
Member

yamen-haddad commented Sep 26, 2024 •
edited by jira bot
Loading

Jira Link: DB-13040

Description

We want to support reading the sequences_data table as of a point in time in the past. This is because clone is failing in some cases where reading a deleted sequence is needed. Consider the following example:

  • Create a database db1 and create snapshot_schedule
  • Create a table t with serial column (and thus a sequence is created)
  • Collect time(t1)
  • Drop table t (and thus the sequence is dropped)
  • Clone db1 to db2.
    Clone fails with the following error:

The reason for the failure is that ysql_dump is unable to get the last_value of the dropped sequence as it was at the clone time. This is because the sequence is dropped and we cannot perform read as of time to the system_postgres.sequences_data table.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@yamen-haddad yamen-haddad added area/docdb YugabyteDB core features priority/high High Priority 2024.2_blocker labels Sep 26, 2024
@yamen-haddad yamen-haddad self-assigned this Sep 26, 2024
@yugabyte-ci yugabyte-ci added the kind/bug This issue is a bug label Sep 26, 2024
yamen-haddad added a commit that referenced this issue Oct 1, 2024
@yamen-haddad
[#21467, #24153] Docdb: Add Read sequences as of time - sequences sup…
62a6a32 
…port for clone part 2

Summary:
Currently, if we try to clone to a time before a drop sequence happened on the original table, clone fails with the following error:
```
ERROR:  Unable to find relation for sequence 16384
```

The reason for the failure is that ysql_dump is unable to get the last_value of the dropped sequence. This is because the sequence is dropped and we cannot perform read as of time to the `system_postgres.sequences_data` table.

This diff adds the ability to **read** the sequences_data table as of a point in time in the past using the `yb_read_time` GUC variable. The yb_read_time didn't cover reading sequences in the past, as sequences operations use an independent YBSession and different RPCs than the `Perform` RPC used in most other read/write operations. This diff extends yb_read_time GUC to cover `ReadSequenceTuple` RPC. An example of usage:

```
CREATE SEQUENCE seq_1;
db1=# SELECT * FROM seq_1;
 last_value | log_cnt | is_called
------------+---------+-----------
          1 |       0 | f

db1=# SELECT (EXTRACT (EPOCH FROM CURRENT_TIMESTAMP)*1000000)::decimal(38,0);
     numeric
------------------
 1727315135780060

db1=# SELECT nextval('seq_1');
 nextval
---------
       1

db1=# SELECT * FROM seq_1;
 last_value | log_cnt | is_called
------------+---------+-----------
        100 |       0 | t

db1=# SET yb_read_time TO 1727315135780060;
SET
db1=# SELECT * FROM seq_1;
 last_value | log_cnt | is_called
------------+---------+-----------
          1 |       0 | f
```
**Upgrade/Rollback safety:**
Only adding in a new optional field `read_time` to `PgReadSequenceTupleRequestPB`.
- If an old message is sent to a service with the new format, the read_time field is set to 0 "default" and the read request will be executed as of the latest time (the default behaviour before this diff).
- If a new message is sent to a service with an older message format, the read_time field will be ignored and the read happens as of the current time.
Knowing that this message only affects pure read requests, no undesired inconsistency is introduced.

Jira: DB-10350, DB-13040

Test Plan:
./yb_build.sh --cxx-test integration-tests_sequence_utility-itest --gtest_filter SequencesUtilTest.ReadSequencesAsOfTime

Also Enabled the following PITR tests for clone and all of them passed after this diff:
PgsqlSequenceUndoDropSequence
PgsqlSequenceVerifyPartialRestore
PgsqlSequencePartialCleanupAfterRestore

Example to run one of them:
./yb_build.sh --cxx-test yb-admin-snapshot-schedule-test --gtest_filter ColocationAndRestoreType/YbAdminSnapshotScheduleTestWithYsqlColocationRestoreParam.PgsqlSequenceUndoDropSequence/3

3 and 4 are the parameters for clone (non-colocation and colocation respectively)

Reviewers: hsunder, asrivastava, mlillibridge

Reviewed By: asrivastava

Subscribers: yql, slingam, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D38392
timothy-e pushed a commit that referenced this issue Oct 2, 2024
@yugabyte-ci @timothy-e
[BACKPORT pg15-cherrypicks] all: Bulk port from master - 115
80673da 
Summary:
 79a00fd [PLAT-15307]fix sensitive info leaks via Gflags
 cd26c93 [DOC-487] Voyager 1.8.2 changes (#24177)
 fa91de7 [docs] Apache Hudi integration with YSQL  (#23888)
 586d337 Updating DynamoDB comparison (#24216)
 aad5695 [#18822] YSQL: Promote autoflag to skip redundant update operations
 fa38152 Fix UBI image: Add -y option to install of hostname
 6baf188 [#23998] Update third-party dependencies and enable SimSIMD in Usearch
 d57db29 Automatic commit by thirdparty_tool: update usearch to commit 191d9bb46fe5e2a44d1505ce7563ed51c7e55868.
 aab1a8b Automatic commit by thirdparty_tool: update simsimd to tag v5.4.3-yb-1.
 161c0c8 [PLAT-15279] Adding unix timestamp to the core dump
 17c45ff [#24217] YSQL: fill definition of a shell type requires catalog version increment
 037fac0 [DB-13062] yugabyted: added banner and get started component
 2eedabd [doc] Read replica connection load balancing support in JDBC Smart driver (#24006)
 62a6a32 [#21467, #24153] Docdb: Add Read sequences as of time - sequences support for clone part 2
 12de78e [PLAT-14954] added support for systemd-timesyncd
 4a07eb8 [#23988] YSQL: Skip a table for schema comparison if its schema does not change
 d3fd39f [doc][ybm] Add reasoning behind no access to yugabyte user #21105 (#23930)
 556ba8a [PLAT-15074] Install node agents on nodes for the marked universes for on-prem providers
 9beb6dc [#22710][#22707] yugabyted: Update voyager migrations list landing page. (#22834)
 6128137 [PLAT-15545] Simplify the frozen universe message for end user in YBA
 4e36b78 JDBC Driver version update to 42.3.5-yb-8 (#24241)
 254c979 [PLAT-15519]: Update xCluster sync to remove tables from YBA DB

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: tfoucher, fizaa, telgersma

Differential Revision: https://phorge.dev.yugabyte.com/D38624
yamen-haddad added a commit that referenced this issue Oct 14, 2024
@yamen-haddad
[BACKPORT 2024.2][#21467, #24153] Docdb: Add Read sequences as of tim…
14a3ff8 
…e - sequences support for clone part 2

Summary:
Original commit: 62a6a32 / D38392
Currently, if we try to clone to a time before a drop sequence happened on the original table, clone fails with the following error:
```
ERROR:  Unable to find relation for sequence 16384
```

The reason for the failure is that ysql_dump is unable to get the last_value of the dropped sequence. This is because the sequence is dropped and we cannot perform read as of time to the `system_postgres.sequences_data` table.

This diff adds the ability to **read** the sequences_data table as of a point in time in the past using the `yb_read_time` GUC variable. The yb_read_time didn't cover reading sequences in the past, as sequences operations use an independent YBSession and different RPCs than the `Perform` RPC used in most other read/write operations. This diff extends yb_read_time GUC to cover `ReadSequenceTuple` RPC. An example of usage:

```
CREATE SEQUENCE seq_1;
db1=# SELECT * FROM seq_1;
 last_value | log_cnt | is_called
------------+---------+-----------
          1 |       0 | f

db1=# SELECT (EXTRACT (EPOCH FROM CURRENT_TIMESTAMP)*1000000)::decimal(38,0);
     numeric
------------------
 1727315135780060

db1=# SELECT nextval('seq_1');
 nextval
---------
       1

db1=# SELECT * FROM seq_1;
 last_value | log_cnt | is_called
------------+---------+-----------
        100 |       0 | t

db1=# SET yb_read_time TO 1727315135780060;
SET
db1=# SELECT * FROM seq_1;
 last_value | log_cnt | is_called
------------+---------+-----------
          1 |       0 | f
```
**Upgrade/Rollback safety:**
Only adding in a new optional field `read_time` to `PgReadSequenceTupleRequestPB`.
- If an old message is sent to a service with the new format, the read_time field is set to 0 "default" and the read request will be executed as of the latest time (the default behaviour before this diff).
- If a new message is sent to a service with an older message format, the read_time field will be ignored and the read happens as of the current time.
Knowing that this message only affects pure read requests, no undesired inconsistency is introduced.

Jira: DB-10350, DB-13040

Test Plan:
./yb_build.sh --cxx-test integration-tests_sequence_utility-itest --gtest_filter SequencesUtilTest.ReadSequencesAsOfTime

Also Enabled the following PITR tests for clone and all of them passed after this diff:
PgsqlSequenceUndoDropSequence
PgsqlSequenceVerifyPartialRestore
PgsqlSequencePartialCleanupAfterRestore

Example to run one of them:
./yb_build.sh --cxx-test yb-admin-snapshot-schedule-test --gtest_filter ColocationAndRestoreType/YbAdminSnapshotScheduleTestWithYsqlColocationRestoreParam.PgsqlSequenceUndoDropSequence/3

3 and 4 are the parameters for clone (non-colocation and colocation respectively)

Reviewers: hsunder, asrivastava, mlillibridge

Reviewed By: asrivastava

Subscribers: ybase, slingam, yql

Differential Revision: https://phorge.dev.yugabyte.com/D38617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yamen-haddad yamen-haddad

Labels
2024.2_blocker area/docdb YugabyteDB core features kind/bug This issue is a bug priority/high High Priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yugabyte-ci @yamen-haddad