roachtest: backup-restore/online-restore failed [cluster backup missing sequence] #130778

cockroach-teamcity · 2024-09-16T06:09:04Z

roachtest.backup-restore/online-restore failed with artifacts on master @ 128fcab4c07413513a05aea1d1494943f4bc3092:

(monitor.go:154).Wait: monitor failure: backup 3_round-trip-test-backup_cluster: error in restore statement: pq: table descriptor rewrite failed: rewriting declarative schema changer state: rewriting descriptor ids: missing rewrite for id 255 in SequenceOwner:{DescID: 210, ColumnID: 5, ReferencedDescID: 255}
test artifacts and logs in: /artifacts/backup-restore/online-restore/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: backup-restore/online-restore failed [relation "[239]" does not exist during schemachange workload] #129187 roachtest: backup-restore/online-restore failed [relation "[239]" does not exist during schemachange workload] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2]

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-42230}

The text was updated successfully, but these errors were encountered:

msbutler · 2024-09-17T17:57:52Z

Looks like a full cluster restore failed to restore a full backup with an endtime of 2024-09-16 06:02:27.24754

2024/09/16 06:08:36 test_impl.go:427: test failure #1: full stack retained in failure_1.log: (monitor.go:154).Wait: monitor failure: backup 3_round-trip-test-backup_cluster: error in restore statement: pq: table descriptor rewrite failed:   rewriting declarative schema changer state: rewriting descriptor ids: missing rewrite for id 255 in SequenceOwner:{DescID: 210, ColumnID: 5, ReferencedDescID: 255}

msbutler · 2024-09-18T13:06:51Z

Reproduced this failure with regular restore. we've got a bug in our descriptor rewrite rules somewere. let's figure out what kind of descriptor we're seeking to restore.

RESTORE DATABASE schemachange FROM LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_pGao?AUTH=implicit';
ERROR: table descriptor rewrite failed: rewriting declarative schema changer state: rewriting descriptor ids: missing rewrite for id 255 in SequenceOwner:{DescID: 210, ColumnID: 5, ReferencedDescID: 255}

msbutler · 2024-09-18T13:35:37Z

We can dump all the descriptor ids in the backup via:

roachprod sql $CLUSTER:1 -- -e "SHOW BACKUP FROM LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_pGao?AUTH=implicit' with debug_ids;"

which reveals:

descriptor 210 is table table_w0_59
descriptor 255 is missing from the backup, but it seems related to sequences?

I see we added a sequence to this table in schema change logs:

{
 "workerId": 3,
 "clientTimestamp": "06:02:22.234199",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE SEQUENCE IF NOT EXISTS public.seq_w3_309 OWNED BY schema_w1_70.table_w0_59.col59_w0_64"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}

Looks like the owning column is indeed the 5th column in this table: col59_w0_64

   "sql": "CREATE TABLE schema_w1_70.table_w0_59 (col5͝9_w0_60 FLOAT8 NOT NULL, col59_w0_61 PG_LSN NOT NULL, c͎ol59_w0_62 UUID NOT NULL, col59_w0_63 \"char\" NOT NULL, col59_w0_64 BOX2D, col59_w0_65 INT8 NOT NULL, \"-col'59_w0_66\" FLOAT8 NOT NULL AS (abs(col5͝9_w0_60)) STORED, \"col\"\"59_w0_67\" FLOAT8 NOT NULL AS (abs(col5͝9_w0_60)) STORED, col59_w0_68 STRING AS (lower(CAST(col59_w0_64 AS STRING))) VIRTUAL, col59_w0_69 STRING NOT NULL AS (lower(CAST(col59_w0_61 AS STRING))) STORED, PRIMARY KEY (\"-col'59_w0_66\" ASC, col59_w0_61 DESC, col59_w0_65 ASC), UNIQUE (col59_w0_65, lower(CAST(c͎ol59_w0_62 AS STRING)) DESC, col59_w0_68, col59_w0_69 ASC, col59_w0_61 ASC), UNIQUE (c͎ol59_w0_62, \"-col'59_w0_66\" DESC), UNIQUE (col59_w0_64 DESC, col5͝9_w0_60, col59_w0_63), INDEX (\"col\"\"59_w0_67\"), FAMILY (col59_w0_69), FAMILY (\"-col'59_w0_66\"), FAMILY (\"col\"\"59_w0_67\", c͎ol59_w0_62, col59_w0_64, col5͝9_w0_60, col59_w0_61, col59_w0_65, col59_w0_63))"

In addition we altered this sequence at the client timestamp of 06:02:25.924582, which is slightly less than the AOST used by the full backup 2024-09-16 06:02:27.24754:

"workerId": 2,
 "clientTimestamp": "06:02:25.924582",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE TABLE IF NOT EXISTS schema_w2_103.table_w2_211 (\"co͟l21\\n1_w2_212\" GEOGRAPHY NOT NULL, col211_w2_213 FLOAT8 NOT NULL, c͒ol211_w2_214 DECIMAL NOT NULL, \"col211\"\"_w2_215\" REGCLASS, PRIMARY KEY (c͒ol211_w2_214 ASC), INVERTED INDEX (\"co͟l21\\n1_w2_212\"), INDEX (c͒ol211_w2_214 ASC, \"col211\"\"_w2_215\"), INVERTED INDEX (c͒ol211_w2_214 ASC, col211_w2_213 ASC, \"col211\"\"_w2_215\" DESC, \"co͟l21\\n1_w2_212\" ASC))"
  },
  {
   "sql": "ALTER TABLE public.table_w1_144 ALTER COLUMN col144_w1_152 SET DEFAULT B'0111111111111111111111111'"
  },
  {
   "sql": "CREATE TABLE schema_w4_134.table_w2_216 AS SELECT schema_w2_103.table_w3_233.col233_w3_235, schema_w2_103.table_w3_233.col233_w3_236, schema_w2_103.table_w3_233.\"col\"\"2😞33_w3_238\", schema_w2_103.table_w3_233.\"c\nol233_w3_241\" FROM schema_w2_103.table_w3_233 FETCH FIRST 300000 ROWS ONLY",
   "potentialExecErr": "22003,22P01,22P02"
  },
  {
   "sql": "ALTER SEQUENCE public.seq_w3_309 RENAME TO public.seq_w2_217"
  },
  {
   "sql": "CREATE TABLE IF NOT EXISTS schema_w3_308.table_w2_218 (col218_w2_219 VARCHAR NOT NULL, col218_w2_220 FLOAT8 NOT NULL, col218_w2_221 \"char\", col218_w2_222 STRING COLLATE de_DE NOT NULL, col218_w2_223 JSONB NOT NULL, col218_w2_224 TIMESTAMP, col218_w2_225 FLOAT8[] NOT NULL, 😑col218_w2_226 DECIMAL NOT NULL, col218_w2_227 DECIMAL NOT NULL AS (abs(😑col218_w2_226)) VIRTUAL, PRIMARY KEY (😑col218_w2_226, col218_w2_227 ASC), FAMILY (col218_w2_221), FAMILY (col218_w2_224), FAMILY (col218_w2_222, 😑col218_w2_226, col218_w2_219), FAMILY (col218_w2_223), FAMILY (col218_w2_225, col218_w2_220))"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}

msbutler · 2024-09-18T20:38:16Z

I may to need input from foundations to proceed, but i can explain what i'm seeing:

The following restore failed with:

RESTORE DATABASE schemachange FROM LATEST IN 'gs://cockroachdb-backup-testing-long-ttl/mixed-version/3_round-trip-test-backup_cluster_pGao?AUTH=implicit';
ERROR: table descriptor rewrite failed: rewriting declarative schema changer state: rewriting descriptor ids: missing rewrite for id 255 in SequenceOwner:{DescID: 210, ColumnID: 5, ReferencedDescID: 255}

using the timeline below, i can tell descriptor id 255 is a sequence generated by the schema change workload. When I run SHOW BACKUP on the backup, the sequence is missing. From what I can infer from the schema change workload, the sequence was created a few seconds before AS OF SYSTEM TIME the backup took a snapshot of the cluster. So, the backup is missing a sequence.

On the cluster backup side, we simply grab all cluster descriptors with this helper function:
https://github.com/cockroachdb/cockroach/blob/master/pkg/ccl/backupccl/targets.go#L333
which calls GetAllDescriptors, so it's unclear to me how we missed the grabbing the sequence, Perhaps the workload client timestamp has significant skew with the mvcc timestamp in the db???

Here's a timeline:

We created table table_w0_59, with descriptor id 210
At client timestamp 06:02:22, we created a sequence that depends on a a column (idx 5) of the above table:

{
 "workerId": 3,
 "clientTimestamp": "06:02:22.234199",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE SEQUENCE IF NOT EXISTS public.seq_w3_309 OWNED BY schema_w1_70.table_w0_59.col59_w0_64"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}

At 06:02:27.24754 we took a full cluster backup
At the client timestamp of 06:02:27.621051, we attempted to run a schema change on table table_w0_59:

{
 "workerId": 3,
 "clientTimestamp": "06:02:27.621051",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER TABLE schema_w1_70.table_w0_59 DROP COLUMN col59_w0_69",
   "potentialExecErr": "42P10"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to conflicting locks on /Table/3/1/277/2/1 [reason=wait_policy] - conflicting txn: meta={id=06e42e03 key=/Table/3/1/277/2/1 iso=Serializable pri=0.00036233 epo=0 ts=1726466548.087219193,0 min=1726466548.087219193,0 seq=1}): \"sql txn\" meta={id=72a02a90 key=/Table/3/1/210/2/1 iso=Serializable pri=0.00044680 epo=0 ts=1726466548.275087898,1 min=1726466547.619471946,0 seq=12} lock=true stat=PENDING rts=1726466547.619471946,0 wto=false gul=1726466548.119471946,0 (SQLSTATE 40001)"
}

At some time later, a full cluster restore of the backup failed with: i.e. at the time of the backup, a schema change associated with the sequence failed (id 255).

ERROR: table descriptor rewrite failed: rewriting declarative schema changer state: rewriting descriptor ids: missing rewrite for id 255 in SequenceOwner:{DescID: 210, ColumnID: 5, ReferencedDescID: 255}

i.e. right here:
https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/catalog/rewrite/rewrite.go#L760

So the descriptorRewrites map is missing the sequence. The sequence also doesn't appear in show backup.

fqazi · 2024-09-20T17:48:29Z

@msbutler So, the sequence wasn't backed up because its in a dropped state, so we aren't able to rewrite it. The plan for DROP column with a sequence owner will put the sequence in a dropped state and then clean up the sequence later. The backup process I think skips dropped descriptors, so we can't repair the schema changer state.

What do you think of allowing backup of dropped descriptors if they have the schema changer state set? We don't have to back up their data, but it would fix weird cases like this. The only other alternative is repairing the schema changer state some how to drop these references.

msbutler · 2024-09-20T22:12:57Z

@fqazi thanks for looking into this! The backup process does indeed skip backing up dropped descriptors.

What do you think of allowing backup of dropped descriptors if they have the schema changer state set? We don't have to back up their data, but it would fix weird cases like this.

Would we need to restore these dropped descriptors as well? If so, do you envision any weirdness around restoring a dropped table descriptor, but none of its data?

Also, out of curiosity, how did you infer that the sequence was in a dropped state?

fqazi · 2024-09-23T14:26:54Z

Also, out of curiosity, how did you infer that the sequence was in a dropped state?

I generated a similar plan to a DROP COLUMN when a sequence owner exists. i.e. EXPLAIN ALTER TABLE blah DROP column j. If we look at the plan then we can see that the clean up of the sequence is split across two stages

Would we need to restore these dropped descriptors as well? If so, do you envision any weirdness around restoring a dropped table descriptor, but none of its data?

Its safer to restore the dropped descriptors. The main weirdness will be that the object in question will be empty if the schema change plan fails with a rollback. The user can't access the data anymore if the plan will be completed successfully.

msbutler · 2024-09-23T14:35:23Z

The main weirdness will be that the object in question will be empty if the schema change plan fails with a rollback. The user can't access the data anymore if the plan will be completed successfully.

Oh wait, you're saying that once a table descriptor's state is dropped, it can still be rolled back to some active state if there's still some schema change acting on that descriptor. So yeah, we do need to backup the table's data then.

If a dropped descriptor's schema changer state is set, will all of its child descriptors also have its schema change state set as well? I'm thinking about your proposal

What do you think of allowing backup of dropped descriptors if they have the schema changer state set?

And if we could end up restoring a broken schema graph if the schema change rolls back after the restore.

fqazi · 2024-10-03T19:25:18Z

Explain for this scenario:

  Schema change plan for ALTER TABLE ‹movr›.public.‹t1› DROP COLUMN ‹j›;
   ├── StatementPhase
   │    └── Stage 1 of 1 in StatementPhase
   │         ├── 3 elements transitioning toward PUBLIC
   │         │    ├── ABSENT → BACKFILL_ONLY PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │         │    ├── ABSENT → PUBLIC        IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 2 (t1_pkey+)}
   │         │    └── ABSENT → PUBLIC        IndexData:{DescID: 118 (t1), IndexID: 2 (t1_pkey+)}
   │         ├── 2 elements transitioning toward TRANSIENT_ABSENT
   │         │    ├── ABSENT → DELETE_ONLY   TemporaryIndex:{DescID: 118 (t1), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (t1_pkey-)}
   │         │    └── ABSENT → PUBLIC        IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 3}
   │         ├── 2 elements transitioning toward ABSENT
   │         │    ├── PUBLIC → WRITE_ONLY    Column:{DescID: 118 (t1), ColumnID: 1 (j-)}
   │         │    └── PUBLIC → ABSENT        ColumnName:{DescID: 118 (t1), Name: "j", ColumnID: 1 (j-)}
   │         └── 6 Mutation operations
   │              ├── MakePublicColumnWriteOnly {"ColumnID":1,"TableID":118}
   │              ├── SetColumnName {"ColumnID":1,"Name":"crdb_internal_co...","TableID":118}
   │              ├── MakeAbsentIndexBackfilling {"Index":{"ConstraintID":2,"IndexID":2,"IsUnique":true,"SourceIndexID":1,"TableID":118,"TemporaryIndexID":3}}
   │              ├── AddColumnToIndex {"ColumnID":2,"IndexID":2,"TableID":118}
   │              ├── MakeAbsentTempIndexDeleteOnly {"Index":{"ConstraintID":3,"IndexID":3,"IsUnique":true,"SourceIndexID":1,"TableID":118}}
   │              └── AddColumnToIndex {"ColumnID":2,"IndexID":3,"TableID":118}
   ├── PreCommitPhase
   │    ├── Stage 1 of 2 in PreCommitPhase
   │    │    ├── 3 elements transitioning toward PUBLIC
   │    │    │    ├── BACKFILL_ONLY → ABSENT PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    │    ├── PUBLIC        → ABSENT IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 2 (t1_pkey+)}
   │    │    │    └── PUBLIC        → ABSENT IndexData:{DescID: 118 (t1), IndexID: 2 (t1_pkey+)}
   │    │    ├── 2 elements transitioning toward TRANSIENT_ABSENT
   │    │    │    ├── DELETE_ONLY   → ABSENT TemporaryIndex:{DescID: 118 (t1), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    │    └── PUBLIC        → ABSENT IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 3}
   │    │    ├── 2 elements transitioning toward ABSENT
   │    │    │    ├── WRITE_ONLY    → PUBLIC Column:{DescID: 118 (t1), ColumnID: 1 (j-)}
   │    │    │    └── ABSENT        → PUBLIC ColumnName:{DescID: 118 (t1), Name: "j", ColumnID: 1 (j-)}
   │    │    └── 1 Mutation operation
   │    │         └── UndoAllInTxnImmediateMutationOpSideEffects
   │    └── Stage 2 of 2 in PreCommitPhase
   │         ├── 3 elements transitioning toward PUBLIC
   │         │    ├── ABSENT → BACKFILL_ONLY PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │         │    ├── ABSENT → PUBLIC        IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 2 (t1_pkey+)}
   │         │    └── ABSENT → PUBLIC        IndexData:{DescID: 118 (t1), IndexID: 2 (t1_pkey+)}
   │         ├── 2 elements transitioning toward TRANSIENT_ABSENT
   │         │    ├── ABSENT → DELETE_ONLY   TemporaryIndex:{DescID: 118 (t1), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (t1_pkey-)}
   │         │    └── ABSENT → PUBLIC        IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 3}
   │         ├── 2 elements transitioning toward ABSENT
   │         │    ├── PUBLIC → WRITE_ONLY    Column:{DescID: 118 (t1), ColumnID: 1 (j-)}
   │         │    └── PUBLIC → ABSENT        ColumnName:{DescID: 118 (t1), Name: "j", ColumnID: 1 (j-)}
   │         └── 11 Mutation operations
   │              ├── MakePublicColumnWriteOnly {"ColumnID":1,"TableID":118}
   │              ├── SetColumnName {"ColumnID":1,"Name":"crdb_internal_co...","TableID":118}
   │              ├── MakeAbsentIndexBackfilling {"Index":{"ConstraintID":2,"IndexID":2,"IsUnique":true,"SourceIndexID":1,"TableID":118,"TemporaryIndexID":3}}
   │              ├── MaybeAddSplitForIndex {"IndexID":2,"TableID":118}
   │              ├── AddColumnToIndex {"ColumnID":2,"IndexID":2,"TableID":118}
   │              ├── MakeAbsentTempIndexDeleteOnly {"Index":{"ConstraintID":3,"IndexID":3,"IsUnique":true,"SourceIndexID":1,"TableID":118}}
   │              ├── MaybeAddSplitForIndex {"IndexID":3,"TableID":118}
   │              ├── AddColumnToIndex {"ColumnID":2,"IndexID":3,"TableID":118}
   │              ├── SetJobStateOnDescriptor {"DescriptorID":118,"Initialize":true}
   │              ├── SetJobStateOnDescriptor {"DescriptorID":119,"Initialize":true}
   │              └── CreateSchemaChangerJob {"RunningStatus":"PostCommitPhase ..."}
   ├── PostCommitPhase
   │    ├── Stage 1 of 7 in PostCommitPhase
   │    │    ├── 2 elements transitioning toward TRANSIENT_ABSENT
   │    │    │    ├── DELETE_ONLY → WRITE_ONLY TemporaryIndex:{DescID: 118 (t1), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    │    └── ABSENT      → PUBLIC     IndexData:{DescID: 118 (t1), IndexID: 3}
   │    │    └── 4 Mutation operations
   │    │         ├── MakeDeleteOnlyIndexWriteOnly {"IndexID":3,"TableID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
   │    │         └── UpdateSchemaChangerJob {"RunningStatus":"PostCommitPhase ..."}
   │    ├── Stage 2 of 7 in PostCommitPhase
   │    │    ├── 1 element transitioning toward PUBLIC
   │    │    │    └── BACKFILL_ONLY → BACKFILLED PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    └── 1 Backfill operation
   │    │         └── BackfillIndex {"IndexID":2,"SourceIndexID":1,"TableID":118}
   │    ├── Stage 3 of 7 in PostCommitPhase
   │    │    ├── 1 element transitioning toward PUBLIC
   │    │    │    └── BACKFILLED → DELETE_ONLY PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    └── 4 Mutation operations
   │    │         ├── MakeBackfillingIndexDeleteOnly {"IndexID":2,"TableID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
   │    │         └── UpdateSchemaChangerJob {"RunningStatus":"PostCommitPhase ..."}
   │    ├── Stage 4 of 7 in PostCommitPhase
   │    │    ├── 1 element transitioning toward PUBLIC
   │    │    │    └── DELETE_ONLY → MERGE_ONLY PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    └── 4 Mutation operations
   │    │         ├── MakeBackfilledIndexMerging {"IndexID":2,"TableID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
   │    │         └── UpdateSchemaChangerJob {"RunningStatus":"PostCommitPhase ..."}
   │    ├── Stage 5 of 7 in PostCommitPhase
   │    │    ├── 1 element transitioning toward PUBLIC
   │    │    │    └── MERGE_ONLY → MERGED PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    └── 1 Backfill operation
   │    │         └── MergeIndex {"BackfilledIndexID":2,"TableID":118,"TemporaryIndexID":3}
   │    ├── Stage 6 of 7 in PostCommitPhase
   │    │    ├── 1 element transitioning toward PUBLIC
   │    │    │    └── MERGED     → WRITE_ONLY            PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    ├── 1 element transitioning toward TRANSIENT_ABSENT
   │    │    │    └── WRITE_ONLY → TRANSIENT_DELETE_ONLY TemporaryIndex:{DescID: 118 (t1), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (t1_pkey-)}
   │    │    └── 5 Mutation operations
   │    │         ├── MakeWriteOnlyIndexDeleteOnly {"IndexID":3,"TableID":118}
   │    │         ├── MakeMergedIndexWriteOnly {"IndexID":2,"TableID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
   │    │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
   │    │         └── UpdateSchemaChangerJob {"RunningStatus":"PostCommitPhase ..."}
   │    └── Stage 7 of 7 in PostCommitPhase
   │         ├── 1 element transitioning toward PUBLIC
   │         │    └── WRITE_ONLY → VALIDATED PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
   │         └── 1 Validation operation
   │              └── ValidateIndex {"IndexID":2,"TableID":118}
   └── PostCommitNonRevertiblePhase
        ├── Stage 1 of 4 in PostCommitNonRevertiblePhase
        │    ├── 2 elements transitioning toward PUBLIC
        │    │    ├── VALIDATED             → PUBLIC           PrimaryIndex:{DescID: 118 (t1), IndexID: 2 (t1_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (t1_pkey-)}
        │    │    └── ABSENT                → PUBLIC           IndexName:{DescID: 118 (t1), Name: "t1_pkey", IndexID: 2 (t1_pkey+)}
        │    ├── 2 elements transitioning toward TRANSIENT_ABSENT
        │    │    ├── TRANSIENT_DELETE_ONLY → TRANSIENT_ABSENT TemporaryIndex:{DescID: 118 (t1), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (t1_pkey-)}
        │    │    └── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 3}
        │    ├── 3 elements transitioning toward ABSENT
        │    │    ├── WRITE_ONLY            → DELETE_ONLY      Column:{DescID: 118 (t1), ColumnID: 1 (j-)}
        │    │    ├── PUBLIC                → VALIDATED        PrimaryIndex:{DescID: 118 (t1), IndexID: 1 (t1_pkey-), ConstraintID: 1}
        │    │    └── PUBLIC                → ABSENT           IndexName:{DescID: 118 (t1), Name: "t1_pkey", IndexID: 1 (t1_pkey-)}
        │    └── 10 Mutation operations
        │         ├── MakeWriteOnlyColumnDeleteOnly {"ColumnID":1,"TableID":118}
        │         ├── MakePublicPrimaryIndexWriteOnly {"IndexID":1,"TableID":118}
        │         ├── SetIndexName {"IndexID":1,"Name":"crdb_internal_in...","TableID":118}
        │         ├── SetIndexName {"IndexID":2,"Name":"t1_pkey","TableID":118}
        │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":3,"TableID":118}
        │         ├── MakeValidatedPrimaryIndexPublic {"IndexID":2,"TableID":118}
        │         ├── MakeIndexAbsent {"IndexID":3,"TableID":118}
        │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
        │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
        │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"RunningStatus":"PostCommitNonRev..."}
        ├── Stage 2 of 4 in PostCommitNonRevertiblePhase
        │    ├── 3 elements transitioning toward ABSENT
        │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (t1), ColumnID: 2 (rowid), IndexID: 1 (t1_pkey-)}
        │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (t1), ColumnID: 1 (j-), IndexID: 1 (t1_pkey-)}
        │    │    └── VALIDATED → DELETE_ONLY PrimaryIndex:{DescID: 118 (t1), IndexID: 1 (t1_pkey-), ConstraintID: 1}
        │    └── 6 Mutation operations
        │         ├── MakeWriteOnlyIndexDeleteOnly {"IndexID":1,"TableID":118}
        │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":1,"TableID":118}
        │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":1,"Kind":2,"TableID":118}
        │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
        │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
        │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"RunningStatus":"PostCommitNonRev..."}
        ├── Stage 3 of 4 in PostCommitNonRevertiblePhase
        │    ├── 1 element transitioning toward TRANSIENT_ABSENT
        │    │    └── PUBLIC      → TRANSIENT_ABSENT IndexData:{DescID: 118 (t1), IndexID: 3}
        │    ├── 11 elements transitioning toward ABSENT
        │    │    ├── DELETE_ONLY → ABSENT           Column:{DescID: 118 (t1), ColumnID: 1 (j-)}
        │    │    ├── PUBLIC      → ABSENT           ColumnType:{DescID: 118 (t1), ColumnFamilyID: 0 (primary), ColumnID: 1 (j-)}
        │    │    ├── PUBLIC      → ABSENT           SequenceOwner:{DescID: 118 (t1), ColumnID: 1 (j-), ReferencedDescID: 119 (sq1-)}
        │    │    ├── DELETE_ONLY → ABSENT           PrimaryIndex:{DescID: 118 (t1), IndexID: 1 (t1_pkey-), ConstraintID: 1}
        │    │    ├── PUBLIC      → ABSENT           IndexData:{DescID: 118 (t1), IndexID: 1 (t1_pkey-)}
        │    │    ├── PUBLIC      → ABSENT           Namespace:{DescID: 119 (sq1-), Name: "sq1", ReferencedDescID: 104 (movr)}
        │    │    ├── PUBLIC      → ABSENT           Owner:{DescID: 119 (sq1-)}
        │    │    ├── PUBLIC      → ABSENT           UserPrivileges:{DescID: 119 (sq1-), Name: "admin"}
        │    │    ├── PUBLIC      → ABSENT           UserPrivileges:{DescID: 119 (sq1-), Name: "root"}
        │    │    ├── PUBLIC      → DROPPED          Sequence:{DescID: 119 (sq1-)}
        │    │    └── PUBLIC      → ABSENT           SchemaChild:{DescID: 119 (sq1-), ReferencedDescID: 105 (public)}
        │    └── 15 Mutation operations
        │         ├── MakeIndexAbsent {"IndexID":1,"TableID":118}
        │         ├── CreateGCJobForIndex {"IndexID":1,"TableID":118}
        │         ├── MarkDescriptorAsDropped {"DescriptorID":119}
        │         ├── RemoveObjectParent {"ObjectID":119,"ParentSchemaID":105}
        │         ├── CreateGCJobForIndex {"IndexID":3,"TableID":118}
        │         ├── RemoveSequenceOwner {"ColumnID":1,"OwnedSequenceID":119,"TableID":118}
        │         ├── RemoveOwnerBackReferenceInSequence {"SequenceID":119}
        │         ├── DrainDescriptorName {"Namespace":{"DatabaseID":104,"DescriptorID":119,"Name":"sq1","SchemaID":105}}
        │         ├── NotImplementedForPublicObjects {"DescID":119,"ElementType":"scpb.Owner"}
        │         ├── RemoveUserPrivileges {"DescriptorID":119,"User":"admin"}
        │         ├── RemoveUserPrivileges {"DescriptorID":119,"User":"root"}
        │         ├── MakeDeleteOnlyColumnAbsent {"ColumnID":1,"TableID":118}
        │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
        │         ├── SetJobStateOnDescriptor {"DescriptorID":119}
        │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"RunningStatus":"PostCommitNonRev..."}
        └── Stage 4 of 4 in PostCommitNonRevertiblePhase
             ├── 2 elements transitioning toward ABSENT
             │    ├── DROPPED → ABSENT Sequence:{DescID: 119 (sq1-)}
             │    └── PUBLIC  → ABSENT TableData:{DescID: 119 (sq1-), ReferencedDescID: 104 (movr)}
             └── 4 Mutation operations
                  ├── CreateGCJobForTable {"DatabaseID":104,"TableID":119}
                  ├── RemoveJobStateFromDescriptor {"DescriptorID":118}
                  ├── RemoveJobStateFromDescriptor {"DescriptorID":119}
                  └── UpdateSchemaChangerJob {"IsNonCancelable":true,"RunningStatus":"all stages compl..."}

msbutler · 2024-10-03T20:07:36Z

handing over to foundations, as the fix requires manual changes to reconstructing the drop column schema change on restore.

Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: cockroachdb#130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN where the column was sequence owner. The restore would fail with: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..."

Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: cockroachdb#130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..."

Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: cockroachdb#130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..."

132202: sql/schemachanger: clean up SequenceOwner elements during restore r=fqazi a=fqazi Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: #130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..." Co-authored-by: Faizan Qazi <[email protected]>

132166: rac2,kvserver: do not quiesce if send tokens held r=sumeerbhola a=pav-kv This PR prevents range quiescence if RACv2 holds any send tokens for this range. Quiescence would prevent `MsgApp` pings which ensure that the leader reliably learns about the follower store admitting log entries, and causes it to release tokens accordingly. We do not want to end up holding tokens permanently. Resolves #129581 132202: sql/schemachanger: clean up SequenceOwner elements during restore r=fqazi a=fqazi Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: #130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..." Co-authored-by: Pavel Kalinnikov <[email protected]> Co-authored-by: Faizan Qazi <[email protected]>

132202: sql/schemachanger: clean up SequenceOwner elements during restore r=fqazi a=fqazi Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: #130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..." 132234: stringer: add license header to stringer-generated files r=rickystewart a=jlinder Some string-generated files did not contain license headers. This change adds license headers to these files. Part of RE-658 Release note: None Co-authored-by: Faizan Qazi <[email protected]> Co-authored-by: James H. Linder <[email protected]>

132129: roachtest: add slow disk perturbation test r=kvoli a=andrewbaptist This change adds a new set of perturbation tests perturbation/*/slowDisk which tests slow disks. We have see support cases where slow disks can cause cluster level availability outages. Epic: none Release note: None 132166: rac2,kvserver: do not quiesce if send tokens held r=sumeerbhola a=pav-kv This PR prevents range quiescence if RACv2 holds any send tokens for this range. Quiescence would prevent `MsgApp` pings which ensure that the leader reliably learns about the follower store admitting log entries, and causes it to release tokens accordingly. We do not want to end up holding tokens permanently. Resolves #129581 132202: sql/schemachanger: clean up SequenceOwner elements during restore r=fqazi a=fqazi Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: #130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..." Co-authored-by: Andrew Baptist <[email protected]> Co-authored-by: Pavel Kalinnikov <[email protected]> Co-authored-by: Faizan Qazi <[email protected]>

132166: rac2,kvserver: do not quiesce if send tokens held r=sumeerbhola a=pav-kv This PR prevents range quiescence if RACv2 holds any send tokens for this range. Quiescence would prevent `MsgApp` pings which ensure that the leader reliably learns about the follower store admitting log entries, and causes it to release tokens accordingly. We do not want to end up holding tokens permanently. Resolves #129581 132202: sql/schemachanger: clean up SequenceOwner elements during restore r=fqazi a=fqazi Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: #130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..." Co-authored-by: Pavel Kalinnikov <[email protected]> Co-authored-by: Faizan Qazi <[email protected]>

blathers-crl · 2024-10-10T15:56:49Z

Based on the specified backports for linked PR #132202, I applied the following new label(s) to this issue: branch-release-23.1, branch-release-23.2, branch-release-24.1, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

Previously, when restoring a backup taken in middle of a DROP COLUMN, where a column had a sequence owner assigned, it was possible for the backup to be unrestorable. This would happen because the sequence reference would have been dropped in the plan, but the seqeunce owner element was still within the state. To address this, this test updates the rewrite logic to clean up any SequenceOwner elements which have the referenced sequence already removed. Fixes: #130778 Release note (bug fix): Addressed a rare bug that could prevent backups taken during a DROP COLUMN operation with a sequence owner from restoring with the error: "rewriting descriptor ids: missing rewrite for <id> in SequenceOwner..."

cockroach-teamcity mentioned this issue Sep 17, 2024

roachtest: backup-restore/online-restore failed #130859

Open

exalate-issue-sync bot assigned msbutler Sep 17, 2024

msbutler added the X-noreuse Prevent automatic commenting from CI test failures label Sep 19, 2024

msbutler changed the title ~~roachtest: backup-restore/online-restore failed~~ roachtest: backup-restore/online-restore failed [cluster backup missing sequence] Sep 19, 2024

exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Oct 2, 2024

msbutler assigned fqazi and unassigned msbutler Oct 3, 2024

msbutler added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-disaster-recovery labels Oct 3, 2024

exalate-issue-sync bot unassigned fqazi Oct 3, 2024

rafiss assigned fqazi Oct 4, 2024

rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 4, 2024

fqazi mentioned this issue Oct 8, 2024

sql/schemachanger: clean up SequenceOwner elements during restore #132202

Merged

craig bot closed this as completed in e08e89b Oct 10, 2024

blathers-crl bot mentioned this issue Oct 10, 2024

release-23.2: sql/schemachanger: clean up SequenceOwner elements during restore #132324

Closed

blathers-crl bot mentioned this issue Oct 10, 2024

release-24.1: sql/schemachanger: clean up SequenceOwner elements during restore #132325

Merged

blathers-crl bot mentioned this issue Oct 10, 2024

release-24.2: sql/schemachanger: clean up SequenceOwner elements during restore #132326

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: backup-restore/online-restore failed [cluster backup missing sequence] #130778

roachtest: backup-restore/online-restore failed [cluster backup missing sequence] #130778

cockroach-teamcity commented Sep 16, 2024 •

edited by cockroach-jira-scripts

Loading

msbutler commented Sep 17, 2024 •

edited

Loading

msbutler commented Sep 18, 2024

msbutler commented Sep 18, 2024 •

edited

Loading

msbutler commented Sep 18, 2024 •

edited

Loading

fqazi commented Sep 20, 2024

msbutler commented Sep 20, 2024

fqazi commented Sep 23, 2024

msbutler commented Sep 23, 2024

fqazi commented Oct 3, 2024

msbutler commented Oct 3, 2024

blathers-crl bot commented Oct 10, 2024

roachtest: backup-restore/online-restore failed [cluster backup missing sequence] #130778

roachtest: backup-restore/online-restore failed [cluster backup missing sequence] #130778

Comments

cockroach-teamcity commented Sep 16, 2024 • edited by cockroach-jira-scripts Loading

msbutler commented Sep 17, 2024 • edited Loading

msbutler commented Sep 18, 2024

msbutler commented Sep 18, 2024 • edited Loading

msbutler commented Sep 18, 2024 • edited Loading

fqazi commented Sep 20, 2024

msbutler commented Sep 20, 2024

fqazi commented Sep 23, 2024

msbutler commented Sep 23, 2024

fqazi commented Oct 3, 2024

msbutler commented Oct 3, 2024

blathers-crl bot commented Oct 10, 2024

cockroach-teamcity commented Sep 16, 2024 •

edited by cockroach-jira-scripts

Loading

msbutler commented Sep 17, 2024 •

edited

Loading

msbutler commented Sep 18, 2024 •

edited

Loading

msbutler commented Sep 18, 2024 •

edited

Loading