REGIONAL BY ROW data inconsistent after changing node's locality
#84144
Comments
FireMasterK
added
the
C-bug
|
Hello, I am Blathers. I am here to help you get the issue triaged. Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here. I have CC'd a few people who may be able to assist you:
If we have not gotten back to your issue within a few business days, you can try the following:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan. |
blathers-crl
bot
added
A-disaster-recovery
O-community
|
cc @cockroachdb/bulk-io |
dt
removed
T-disaster-recovery
T-sql-schema-deprecated
|
Hi @FireMasterK . Thanks for filing the issue. Also, could you explain to us why you changed the locality, and what you were trying to achieve with this? |
Unfortunately, it looks like some data was lost when I did that. I did Logs from commandThe If I alter the statement to just
I had regions such as My goal was to group and home all Europe-based data together. |
knz
added
the
S-0-corruption-or-data-loss
|
When you changed to the |
|
Yes, I did, I tried that in an attempt to try disabling multi-region capabilities to see if that resolves the issue, but it did not. I added My |
|
Thanks for reporting this @FireMasterK. I still haven't gotten to the bottom of this, but here's what I've found so far. Turns out, the situation you find yourself in has nothing to do with the Running the above repro through a debugger, it looks like we aren't tracking references to the table on the type descriptor. It's also worth noting that this only happens when you alter to a REGIONAL BY ROW table -- things would have worked as expected if we had created a REGIONAL BY ROW table to begin with! It also seems like this is only ever an issue on release-22.1. We tried the above repro out on 21.2 cluster, and things worked (failed!) as expected. |
|
@ZhouXing19 can I take this issue? |
84305: sql: add missing back-reference in region enum type r=postamar a=postamar This commit fixes a long-standing bug in which making an existing multi-region table REGIONAL BY ROW would omit adding a back-reference to the table in the region enum type. This puts the table at risk for corruption in the face of schema changes to the region enum, which is unaware of this dependency. Fixes #84144. Release note: None 84308: Revert "ccl: upgrade by-name sequence reference to by-ID during restore" r=Xiang-Gu a=Xiang-Gu This reverts commit f498598 because this PR introduced some dependencies that violates certain existing dependency assertions. Release note: None Co-authored-by: Marius Posta <[email protected]> Co-authored-by: Xiang Gu <[email protected]>
|
I'm merging and backporting #84305 which fixes the type descriptor corruption that ultimately caused this error, but does nothing to fix clusters which have already been corrupted by an ALTER TABLE ... SET LOCALITY REGIONAL BY ROW: a subsequent DROP REGION like in the repro will happily succeed rather than fail like it should. I'm tracking fixing that in a separate issue: #84322 |
This commit fixes a long-standing bug in which making an existing multi-region table REGIONAL BY ROW would omit adding a back-reference to the table in the region enum type. This puts the table at risk for corruption in the face of schema changes to the region enum, which is unaware of this dependency. Informs cockroachdb#84144. Release note (bug fix): fixes bug where ALTER TABLE ... SET LOCALITY REGIONAL BY ROW could leave the region enum type descriptor unaware of a dependency on the altered table. This would, in turn, wrongly permit a DROP REGION to succeed, rendering the table unusable. Note that this fix does not help existing clusters which have already run such an ALTER TABLE; see cockroachdb#84322 for that.
This commit fixes a long-standing bug in which making an existing multi-region table REGIONAL BY ROW would omit adding a back-reference to the table in the region enum type. This puts the table at risk for corruption in the face of schema changes to the region enum, which is unaware of this dependency. Fixes cockroachdb#84144. This bug seems to have manifested itself only in 22.1 and later after 718222f was merged. We're applying this patch to 21.2 out of prudence. Release note: None
This commit is featured only in the release-22.1 branch and repairs missing table back-references in the region enum type descriptor before performing a DROP REGION or DROP SUPER REGION. Partially addresses cockroachdb#84322. See also cockroachdb#84144. Release note: None
rytaft
added
C-technical-advisory
Describe the problem
Please describe the issue you observed, and any steps we can take to reproduce it:
To Reproduce
REGIONAL BY ROW.--locality=parameter.ALTER TABLE {table} SET LOCALITY REGIONAL BY TABLE IN PRIMARY REGION;Expected behavior
The database to work as expected.
Additional data / screenshots
I can provide any information requested.
Environment:
Additional context
The database's data was inconsistent, backups didn't work, any
REGIONAL BY ROWtable was affected.Jira issue: CRDB-17489
The text was updated successfully, but these errors were encountered: