ccl/schemachangerccl: TestBackupSuccessMixedVersion_base_add_column failed

[github-actions]

ccl/schemachangerccl.TestBackupSuccessMixedVersion_base_add_column failed on master @ b42ad6451071f395758903e547e3ec008642da50:

        	            	      │    │    ├── WRITE_ONLY            → PUBLIC           Column:{DescID: 118 (#118), ColumnID: 3 (j+)}
        	            	      │    │    ├── VALIDATED             → PUBLIC           PrimaryIndex:{DescID: 118 (#118), IndexID: 2 (tbl_pkey+), ConstraintID: 2, TemporaryIndexID: 3, SourceIndexID: 1 (tbl_pkey-)}
        	            	      │    │    ├── ABSENT                → PUBLIC           IndexName:{DescID: 118 (#118), Name: "tbl_pkey", IndexID: 2 (tbl_pkey+)}
        	            	      │    │    └── VALIDATED             → PUBLIC           ColumnNotNull:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 2 (tbl_pkey+)}
        	            	      │    ├── 4 elements transitioning toward TRANSIENT_ABSENT
        	            	      │    │    ├── TRANSIENT_DELETE_ONLY → TRANSIENT_ABSENT TemporaryIndex:{DescID: 118 (#118), IndexID: 3, ConstraintID: 3, SourceIndexID: 1 (tbl_pkey-)}
        	            	      │    │    ├── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 1, IndexID: 3}
        	            	      │    │    ├── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 2, IndexID: 3}
        	            	      │    │    └── PUBLIC                → TRANSIENT_ABSENT IndexColumn:{DescID: 118 (#118), ColumnID: 3 (j+), IndexID: 3}
        	            	      │    ├── 2 elements transitioning toward ABSENT
        	            	      │    │    ├── PUBLIC                → VALIDATED        PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
        	            	      │    │    └── PUBLIC                → ABSENT           IndexName:{DescID: 118 (#118), Name: "tbl_pkey", IndexID: 1 (tbl_pkey-)}
        	            	      │    └── 13 Mutation operations
        	            	      │         ├── MakePublicPrimaryIndexWriteOnly {"IndexID":1,"TableID":118}
        	            	      │         ├── SetIndexName {"IndexID":1,"Name":"crdb_internal_in...","TableID":118}
        	            	      │         ├── SetIndexName {"IndexID":2,"Name":"tbl_pkey","TableID":118}
        	            	      │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":3,"TableID":118}
        	            	      │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":3,"Kind":2,"TableID":118}
        	            	      │         ├── RemoveColumnFromIndex {"ColumnID":3,"IndexID":3,"Kind":2,"Ordinal":1,"TableID":118}
        	            	      │         ├── MakeValidatedColumnNotNullPublic {"ColumnID":3,"TableID":118}
        	            	      │         ├── MakeValidatedPrimaryIndexPublic {"IndexID":2,"TableID":118}
        	            	      │         ├── MakeIndexAbsent {"IndexID":3,"TableID":118}
        	            	      │         ├── MakeWriteOnlyColumnPublic {"ColumnID":3,"TableID":118}
        	            	      │         ├── RefreshStats {"TableID":118}
        	            	      │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
        	            	      │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":996392877633306600,"RunningStatus":"PostCommitNonRev..."}
        	            	      ├── Stage 2 of 3 in PostCommitNonRevertiblePhase
        	            	      │    ├── 3 elements transitioning toward ABSENT
        	            	      │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (#118), ColumnID: 1, IndexID: 1 (tbl_pkey-)}
        	            	      │    │    ├── PUBLIC    → ABSENT      IndexColumn:{DescID: 118 (#118), ColumnID: 2, IndexID: 1 (tbl_pkey-)}
        	            	      │    │    └── VALIDATED → DELETE_ONLY PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
        	            	      │    └── 5 Mutation operations
        	            	      │         ├── MakeWriteOnlyIndexDeleteOnly {"IndexID":1,"TableID":118}
        	            	      │         ├── RemoveColumnFromIndex {"ColumnID":1,"IndexID":1,"TableID":118}
        	            	      │         ├── RemoveColumnFromIndex {"ColumnID":2,"IndexID":1,"Kind":2,"TableID":118}
        	            	      │         ├── SetJobStateOnDescriptor {"DescriptorID":118}
        	            	      │         └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":996392877633306600,"RunningStatus":"PostCommitNonRev..."}
        	            	      └── Stage 3 of 3 in PostCommitNonRevertiblePhase
        	            	           ├── 1 element transitioning toward TRANSIENT_ABSENT
        	            	           │    └── PUBLIC      → TRANSIENT_ABSENT IndexData:{DescID: 118 (#118), IndexID: 3}
        	            	           ├── 2 elements transitioning toward ABSENT
        	            	           │    ├── DELETE_ONLY → ABSENT           PrimaryIndex:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-), ConstraintID: 1}
        	            	           │    └── PUBLIC      → ABSENT           IndexData:{DescID: 118 (#118), IndexID: 1 (tbl_pkey-)}
        	            	           └── 5 Mutation operations
        	            	                ├── MakeIndexAbsent {"IndexID":1,"TableID":118}
        	            	                ├── CreateGCJobForIndex {"IndexID":1,"TableID":118}
        	            	                ├── CreateGCJobForIndex {"IndexID":3,"TableID":118}
        	            	                ├── RemoveJobStateFromDescriptor {"DescriptorID":118,"JobID":996392877633306600}
        	            	                └── UpdateSchemaChangerJob {"IsNonCancelable":true,"JobID":996392877633306600,"RunningStatus":"all stages compl..."}
        --- FAIL: TestBackupSuccessMixedVersion_base_add_column/post_commit_stage_7_of_7/restore_database (0.99s)

Parameters:

• attempt=1
• run=1
• shard=26

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41490

[rafiss]

    backup.go:490: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/backup.go:490
        	Error:      	Not equal: 
        	            	expected: [][]string{[]string{"CREATE SCHEMA public"}, []string{"CREATE SEQUENCE public.sq1 MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 1 START 1"}, []string{"CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, j INT8 NOT NULL DEFAULT 42:::INT8, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"}}
        	            	actual  : [][]string{[]string{"CREATE SCHEMA public"}, []string{"CREATE SEQUENCE public.sq1 MINVALUE 1 MAXVALUE 9223372036854775807 INCREMENT 1 START 1"}, []string{"CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -8,3 +8,3 @@
        	            	  ([]string) (len=1) {
        	            	-  (string) (len=130) "CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, j INT8 NOT NULL DEFAULT 42:::INT8, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"
        	            	+  (string) (len=95) "CREATE TABLE public.tbl (i INT8 NOT NULL, k INT8 NULL, CONSTRAINT tbl_pkey PRIMARY KEY (i ASC))"
        	            	  }

[rafiss]

This is definitely unexpected, let's see if we can repro under stress to decide if we should investigate futher.

[rafiss]

Happened again in TestBackupSuccess_base_add_column #131583

[rafiss]

closing as a dupe of #131583