ccl/schemachangerccl: TestBackup_base_add_column_default_seq failed

[cockroach-teamcity]

ccl/schemachangerccl.TestBackup_base_add_column_default_seq failed with artifacts on release-23.1 @ 5376e479204d6e8243f67c30aea3d031df529afd:

F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x4005612a80, 0x68c7b68?)
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646 +0xd8
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x401189e090, {0x68c7b68, 0x400aa1c240}, {0x68ac2c0, 0x4005612a80}, 0x4000c99200)
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:392 +0x20c
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x68c7b68?, 0x400aa1c240?})
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 +0x48
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x128
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x358
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/9a7cc7a3f88f8f8dddb6619a0383ed86/logTestBackup_base_add_column_default_seq3658397246
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/9a7cc7a3f88f8f8dddb6619a0383ed86/logTestBackup_base_add_column_default_seq3658397246
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !****************************************************************************
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !This node experienced a fatal error (printed above), and as a result the
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !process is terminating.
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !will try to determine the root cause, recommend next steps, and we can
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !improve CockroachDB based on your report.
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !Please submit a crash report by following the instructions here:
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !If you would rather not post publicly, please contact us directly at:
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !    [email protected]
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 06:56:58.454294 826343 3@pebble/db.go:811  [n2,s2,pebble] 1 !The Cockroach Labs team appreciates your feedback.
=== RUN   TestBackup_base_add_column_default_seq/backup/restore_stage_5_of_10/restore_database#01
    cumulative.go:1322: testing backup 1 (rollback=false)
=== RUN   TestBackup_base_add_column_default_seq/backup/restore_stage_8_of_10
    cumulative.go:1157: stage 1/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 2/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 3/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 4/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 5/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 6/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 7/10 in PostCommitPhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 8/10 in PostCommitNonRevertiblePhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 9/10 in PostCommitNonRevertiblePhase (rollback=false) 8 "db" false
    cumulative.go:1157: stage 10/10 in PostCommitNonRevertiblePhase (rollback=false) 8 "db" true
    cumulative.go:1189: reached final stage, waiting for completion
    cumulative.go:1204: finished

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-27738

[rafiss]

dupe of #102945