-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[#23669] YSQL: Fix DDL atomicity assertion failure
Summary: From the ticket, the following error is noticed: ``` E0824 04:05:51.544625 262184 pg_client_session.cc:1109] Sending ReportYsqlDdlTxnStatus call failed: Illegal state (yb/master/ysql_ddl_handler.cc:221): Mismatch in txn_state for transaction 6c3e8bad-d77d-4fc3-b05f-cde9ba7cc985: kAborted vs kCommitted ``` I found two logs that showed contradicting message: ``` ./yb-master.ip-172-151-28-119.us-west-2.compute.internal.yugabyte.log.INFO.20240824-035800.38216.gz:I0824 04:05:51.536473 291955 ysql_ddl_handler.cc:188] YsqlDdlTxnCompleteCallback for transaction 6c3e8bad-d77d-4fc3-b05f-cde9ba7cc985 is_committed: true ``` vs ``` ./yb-master.ip-172-151-28-119.us-west-2.compute.internal.yugabyte.log.INFO.20240824-035800.38216.gz:I0824 04:05:51.544212 185834 ysql_ddl_handler.cc:188] YsqlDdlTxnCompleteCallback for transaction 6c3e8bad-d77d-4fc3-b05f-cde9ba7cc985 is_committed: false ``` Both are related to the same txn `6c3e8bad-d77d-4fc3-b05f-cde9ba7cc985`, and their timestamps are close: 04:05:51.536473 vs 04:05:51.54421. However the first log says the txn has committed (is_committed: true), and the second log says the txn has aborted (is_committed: false) So two different code paths deduced different commit/abort status for the same txn. DDL atomicity has two ways for docdb to follow and become eventually consistent with the PG transaction status (commit or abort): (1) PG sends a status report to master tell it whether the txn has committed or not. This is the fastest since PG drives the DDL txn and it knows firsthand. (2) Docdb has a background task, periodically polling the transaction status tablet to find out the status of the give txn. If the transaction status tablet response with the txn status as terminated (not pending), then the same background task will begin to read the current PG metadata from sys catalog table, based upon the nature of the DDL, performs a schema comparison to find out whether the current PG catalog schema indicated a commit or abort: (2.1) If the current PG catalog schema matches the current DocDB schema (the new table schema if the DDL successfully commits), it means the DDL txn has committed. (2.2) If the current PG catalog schema matches the previous DocDB schema (the old table schema prior to the DDL starts), it means the DDL txn has aborted. The bug is that there are some DDL statements that do not change the table's schema. For example, `alter table foo alter column id set not null` only bumps up the table foo's schema version without changing the schema at all. As a result, the current schema is identical to the previous schema. In other words, both (2.1) and (2.2) will compare as equal and because we check (2.1) first, we always conclude that the DDL txn has committed. Had we reversed the order by checking (2.2) first, then we would always conclude that the DDL txn has aborted. I made a change to make the type of is_committed from bool to std::optional<bool>. When is_committed.has_value() is true, it means either committed or aborted so we continue with the current logic. When is_committed is nullopt, it means the DDL does not change the table's schema at all. A new state TxnState::kNoChange is added in addition to the current kUnknown, kCommitted, kAborted. It is not possible to tell whether the DDL txn has committed or aborted by doing schema comparison if the DDL does not change the table schema. However it is fine because we do not need to know whether the DDL txn is committed or aborted, we only need to know the DDL txn has terminated. The end result is that the table's schema does not change. We just need to clear the DDL atomicity related state from the table's DocDB SysTablesEntryPB metadata and the table's schema will be correct. Jira: DB-12580 Test Plan: jenkins Reviewers: hsunder, fizaa Reviewed By: fizaa Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D37755
- Loading branch information
Showing
8 changed files
with
99 additions
and
39 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters