Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: TestFullClusterBackup failed #68028

Closed
cockroach-teamcity opened this issue Jul 24, 2021 · 3 comments
Closed

ccl/backupccl: TestFullClusterBackup failed #68028

cockroach-teamcity opened this issue Jul 24, 2021 · 3 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

ccl/backupccl.TestFullClusterBackup failed with artifacts on master @ b02d22f9b3d30a0288ad1d8464dd6f2d82c08f0d:

Fatal error:

panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked

Stack:

goroutine 585669 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00337b950, 0xa067828, 0xc005d27bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x62f5c40, 0x9f7e060)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00337b950, 0xa0678d0, 0xc005764f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x62f5c40, 0x9f7e060)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send.func1(0xc001bf7b78, 0xc001bf7c30, 0xc0015d7000, 0xc001bf7c28)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:85 +0x745
panic(0x62f5c40, 0x9f7e060)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).AssertHeld(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync_race.go:49
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).MaybeGossipSystemConfigRaftMuLocked(0xc006fdc000, 0xa0678d0, 0xc005764f60, 0xc0041f2c60, 0x5b1688)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_gossip.go:83 +0x865
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc006fdc000, 0xa0678d0, 0xc005764f60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:711 +0xc09
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc006fdc000, 0xa0678d0, 0xc005764f60, 0xc0065eac60, 0xc0089070e0, 0x0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:118 +0x1718
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc006fdc000, 0xa0678d0, 0xc005764f60, 0xc0065eac60, 0xc0089070e0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:149 +0x84e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc006fdc000, 0xa0678d0, 0xc005764f60, 0xc0065eac60, 0x7155f78, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:376 +0x4a2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc006fdc000, 0xa0678d0, 0xc005764f30, 0x6, 0xc0065eac60, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:107 +0x652
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:43
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0015d7000, 0xa0678d0, 0xc005764f00, 0x1694a304d21f5d8e, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:180 +0x5b3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc008dd0d80, 0xa0678d0, 0xc005764f00, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:191 +0x198
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0xa0678d0, 0xc005764f00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:901 +0x27e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00337b950, 0xa0678d0, 0xc005764f00, 0x6afe280, 0x10, 0xc001bf8248, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324 +0x12a
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0088b6c00, 0xa0678d0, 0xc005764f00, 0xc0065eab00, 0x0, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:889 +0x267
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0088b6c00, 0xa0678d0, 0xc005764ed0, 0xc0065eab00, 0x5b227c, 0xc0050ecf60, 0x4a19ac)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:963 +0x118
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:462
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc00c4c2a80, 0xa0678d0, 0xc005764ed0, 0x1, 0xa0baf48, 0xc002897030, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:195 +0x25d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc00c4c2a80, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:177 +0x25f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext(0xa0b41e8, 0xc00c4c2a80, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x443
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc008ed2400, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1882 +0xdfe
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc008ed2400, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1500 +0x437
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc008ed2400, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1118 +0x1f0a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc008ed2400, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:757 +0x77b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc002c40460, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x1b5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0xc002c40428, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0xc002c403f8, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:200 +0x275
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc002c40358, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:240 +0x115
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc002c40358, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:175 +0x36a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc002c40288, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:285 +0x415
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc002c40268, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105 +0x355
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0xc002c401c8, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:241 +0x275
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc002c40000, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:513 +0x575
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc008dd0c00, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:808 +0x175
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc0065ea790, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:982 +0x228
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit(0xc0065ea790, 0xa0678d0, 0xc0050ecf60, 0x27a5153, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:645 +0x25e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit(0xc0065ea790, 0xa0678d0, 0xc0050ecf60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:680 +0x10e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc0065ea790, 0xa0678d0, 0xc0050ecf60, 0xc001bfef30, 0xc0065ea790, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:876 +0x4d9
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc008dd0c00, 0xa0678d0, 0xc0050ecf60, 0xc001bfef80, 0x0, 0xc006104eb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771 +0x125
github.com/cockroachdb/cockroach/pkg/sql/gcjob.deleteDatabaseZoneConfig(0xa0678d0, 0xc0050ecf60, 0xc008dd0c00, 0xc00a31fd10, 0xc00a31fd10, 0x0, 0xc000000034, 0xa0678d0, 0xc0050ecf60)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/descriptor_utils.go:69 +0xb3
github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC(0xa0678d0, 0xc0050ecf60, 0xc008b33500, 0xc00857c840, 0xc006104e80, 0xc00c4bd780, 0x29)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:74 +0x250
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume(0x96a32030dbb8001, 0xa0678d0, 0xc0050ecf60, 0x67b3de0, 0xc00847e0d8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:167 +0x74f
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc009434680, 0xa024890, 0xc00f3f62f8, 0xa0678d0, 0xc0050ecf60, 0x67b3de0, 0xc00847e0d8, 0xc001bff6e8)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1025 +0x142
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc002cdcd00, 0xa0678d0, 0xc0050eced0, 0x67b3de0, 0xc00847e0d8, 0xa024890, 0xc00f3f62f8, 0xc00b586000, 0xc00f3f61b0, 0x7, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1026 +0xa45
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc002cdcd00, 0xa067828, 0xc0053bdc00, 0xa024890, 0xc00f3f62f8, 0xc00b586000, 0xc00f3f61b0, 0x7, 0xc00bbf40f0, 0x16, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:290 +0x61d
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0xa067828, 0xc005d27bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:222 +0x19b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2(0xc00337b950, 0xa067828, 0xc005d27bc0, 0x0, 0x0, 0xc0014ee640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x162
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:434 +0x28e
Log preceding fatal error 

=== RUN   TestFullClusterBackup
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestFullClusterBackup397437552
    test_log_scope.go:74: use -show-logs to present logs inline

Reproduce

To reproduce, try: 

make stressrace TESTS=TestFullClusterBackup PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-race -parallel=4

/cc @cockroachdb/bulk-io pbardea

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 24, 2021
@cockroach-teamcity
Copy link
Member Author

ccl/backupccl.TestFullClusterBackup failed with artifacts on master @ 9baaa282b3a09977b96bd3e5ae6e2346adfa2c16:

Fatal error:

panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked

Stack:

goroutine 603661 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00736a510, 0xa06ade8, 0xc00ef04bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x62f8c80, 0x9f815e0)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00736a510, 0xa06ae90, 0xc00856c2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x62f8c80, 0x9f815e0)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send.func1(0xc00122bb78, 0xc00122bc30, 0xc005cc1800, 0xc00122bc28)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:85 +0x745
panic(0x62f8c80, 0x9f815e0)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).AssertHeld(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync_race.go:49
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).MaybeGossipSystemConfigRaftMuLocked(0xc002be3500, 0xa06ae90, 0xc00856c360, 0xc007f7b0e0, 0x5b1688)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_gossip.go:83 +0x865
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc002be3500, 0xa06ae90, 0xc00856c360, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:711 +0xc09
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc002be3500, 0xa06ae90, 0xc00856c360, 0xc0047f8f20, 0xc009754c60, 0x0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:118 +0x1718
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc002be3500, 0xa06ae90, 0xc00856c360, 0xc0047f8f20, 0xc009754c60, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:149 +0x84e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc002be3500, 0xa06ae90, 0xc00856c360, 0xc0047f8f20, 0x71591e8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:376 +0x4a2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc002be3500, 0xa06ae90, 0xc00856c330, 0x6, 0xc0047f8f20, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:107 +0x652
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:43
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc005cc1800, 0xa06ae90, 0xc00856c2d0, 0x1694f18df5d2f5c7, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:180 +0x5b3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc00ca5fa80, 0xa06ae90, 0xc00856c2d0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:191 +0x198
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0xa06ae90, 0xc00856c2d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:901 +0x27e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00736a510, 0xa06ae90, 0xc00856c2d0, 0x6b014ee, 0x10, 0xc00122c248, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324 +0x12a
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0078eb800, 0xa06ae90, 0xc00856c2d0, 0xc0047f8e70, 0x0, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:889 +0x267
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0078eb800, 0xa06ae90, 0xc00856c2a0, 0xc0047f8e70, 0x5b227c, 0xc00991fe90, 0x4a19ac)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:963 +0x118
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:462
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc00a922c00, 0xa06ae90, 0xc00856c2a0, 0x1, 0xa0be508, 0xc011294ee0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:195 +0x25d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc00a922c00, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:177 +0x25f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext(0xa0b77a8, 0xc00a922c00, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x443
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc00e369400, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1882 +0xdfe
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc00e369400, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1500 +0x437
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc00e369400, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1118 +0x1f0a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc00e369400, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:757 +0x77b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc00952e960, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x1b5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0xc00952e928, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0xc00952e8f8, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:200 +0x275
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc00952e858, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:240 +0x115
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc00952e858, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:175 +0x36a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc00952e788, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:285 +0x415
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc00952e768, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105 +0x355
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0xc00952e6c8, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:241 +0x275
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc00952e500, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:513 +0x575
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc00ca5f900, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:808 +0x175
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc0047f8630, 0xa06ae90, 0xc00991fe90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:982 +0x228
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit(0xc0047f8630, 0xa06ae90, 0xc00991fe90, 0x27a5793, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:645 +0x25e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit(0xc0047f8630, 0xa06ae90, 0xc00991fe90, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:680 +0x10e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc0047f8630, 0xa06ae90, 0xc00991fe90, 0xc001232f30, 0xc0047f8630, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:876 +0x4d9
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc00ca5f900, 0xa06ae90, 0xc00991fe90, 0xc001232f80, 0x0, 0xc002fa7330)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771 +0x125
github.com/cockroachdb/cockroach/pkg/sql/gcjob.deleteDatabaseZoneConfig(0xa06ae90, 0xc00991fe90, 0xc00ca5f900, 0xc00d8e4000, 0xc00d8e4000, 0x0, 0xc000000034, 0xa06ae90, 0xc00991fe90)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/descriptor_utils.go:69 +0xb3
github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC(0xa06ae90, 0xc00991fe90, 0xc00f3e3180, 0xc0073b51a0, 0xc002fa7300, 0xc000acf980, 0x29)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:74 +0x250
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume(0x96b335b86b80001, 0xa06ae90, 0xc00991fe90, 0x67b6f00, 0xc0023be1c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:167 +0x74f
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc002d5d8c0, 0xa027e50, 0xc01165cf88, 0xa06ae90, 0xc00991fe90, 0x67b6f00, 0xc0023be1c0, 0xc0012336e8)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1025 +0x142
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc00a221ba0, 0xa06ae90, 0xc00991fe00, 0x67b6f00, 0xc0023be1c0, 0xa027e50, 0xc01165cf88, 0xc00cefec80, 0xc01165cde0, 0x7, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1026 +0xa45
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc00a221ba0, 0xa06ade8, 0xc00c95a0c0, 0xa027e50, 0xc01165cf88, 0xc00cefec80, 0xc01165cde0, 0x7, 0xc008748660, 0x16, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:290 +0x61d
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0xa06ade8, 0xc00ef04bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:222 +0x19b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2(0xc00736a510, 0xa06ade8, 0xc00ef04bc0, 0x0, 0x0, 0xc006736050)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x162
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:434 +0x28e
Log preceding fatal error 

=== RUN   TestFullClusterBackup
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestFullClusterBackup101982872
    test_log_scope.go:74: use -show-logs to present logs inline

Reproduce

To reproduce, try: 

make stressrace TESTS=TestFullClusterBackup PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-race -parallel=4

/cc @cockroachdb/bulk-io pbardea

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/backupccl.TestFullClusterBackup failed with artifacts on master @ f7528c59e296ed9acd2a20d590f2a42bbad0dcd0:

Fatal error:

panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked

Stack:

goroutine 602849 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00c1338c0, 0xa06aea8, 0xc00cadf980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x62f8c80, 0x9f81680)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00c1338c0, 0xa06af50, 0xc00644ed20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x62f8c80, 0x9f81680)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send.func1(0xc003371b78, 0xc003371c30, 0xc001dc8000, 0xc003371c28)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:85 +0x745
panic(0x62f8c80, 0x9f81680)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).AssertHeld(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync_race.go:49
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).MaybeGossipSystemConfigRaftMuLocked(0xc0059aea80, 0xa06af50, 0xc00644ed80, 0xc006d2cc60, 0x5b1688)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_gossip.go:83 +0x865
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0059aea80, 0xa06af50, 0xc00644ed80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:711 +0xc09
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc0059aea80, 0xa06af50, 0xc00644ed80, 0xc007a59810, 0xc00bef27e0, 0x0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:118 +0x1718
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc0059aea80, 0xa06af50, 0xc00644ed80, 0xc007a59810, 0xc00bef27e0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:149 +0x84e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0059aea80, 0xa06af50, 0xc00644ed80, 0xc007a59810, 0x7159260, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:376 +0x4a2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc0059aea80, 0xa06af50, 0xc00644ed50, 0x6, 0xc007a59810, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:107 +0x652
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:43
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc001dc8000, 0xa06af50, 0xc00644ed20, 0x169540171f21dbe5, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:180 +0x5b3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc0098b6e00, 0xa06af50, 0xc00644ed20, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:191 +0x198
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0xa06af50, 0xc00644ed20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:901 +0x27e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00c1338c0, 0xa06af50, 0xc00644ed20, 0x6b014ee, 0x10, 0xc003372248, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324 +0x12a
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0062e2000, 0xa06af50, 0xc00644ed20, 0xc007a596b0, 0x0, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:889 +0x267
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0062e2000, 0xa06af50, 0xc00644ecf0, 0xc007a596b0, 0x5b227c, 0xc009df21b0, 0x4a19ac)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:963 +0x118
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:462
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc009d2e380, 0xa06af50, 0xc00644ecf0, 0x1, 0xa0be5c8, 0xc00a209150, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:195 +0x25d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc009d2e380, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:177 +0x25f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext(0xa0b7868, 0xc009d2e380, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x443
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc00d56e000, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1882 +0xdfe
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc00d56e000, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1500 +0x437
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc00d56e000, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1118 +0x1f0a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc00d56e000, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:757 +0x77b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc006f63d60, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x1b5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0xc006f63d28, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0xc006f63cf8, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:200 +0x275
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc006f63c58, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:240 +0x115
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc006f63c58, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:175 +0x36a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc006f63b88, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:285 +0x415
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc006f63b68, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105 +0x355
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0xc006f63ac8, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:241 +0x275
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc006f63900, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:513 +0x575
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc0098b6c80, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:808 +0x175
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc007a59080, 0xa06af50, 0xc009df21b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:982 +0x228
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit(0xc007a59080, 0xa06af50, 0xc009df21b0, 0x27a5ab3, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:645 +0x25e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit(0xc007a59080, 0xa06af50, 0xc009df21b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:680 +0x10e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc007a59080, 0xa06af50, 0xc009df21b0, 0xc003378f30, 0xc007a59080, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:876 +0x4d9
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(0xc0098b6c80, 0xa06af50, 0xc009df21b0, 0xc003378f80, 0x0, 0xc00a465d70)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:771 +0x125
github.com/cockroachdb/cockroach/pkg/sql/gcjob.deleteDatabaseZoneConfig(0xa06af50, 0xc009df21b0, 0xc0098b6c80, 0xc00e2257b8, 0xc00e2257b8, 0x0, 0xc000000034, 0xa06af50, 0xc009df21b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/descriptor_utils.go:69 +0xb3
github.com/cockroachdb/cockroach/pkg/sql/gcjob.performGC(0xa06af50, 0xc009df21b0, 0xc009f6ae00, 0xc0098e6e40, 0xc00a465d40, 0xc003563dc0, 0x29)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:74 +0x250
github.com/cockroachdb/cockroach/pkg/sql/gcjob.schemaChangeGCResumer.Resume(0x96c34b41e880001, 0xa06af50, 0xc009df21b0, 0x67b6f00, 0xc00c7e6630, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/gcjob/gc_job.go:167 +0x74f
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func1(0xc0066d4ac0, 0xa027ed0, 0xc000dd6618, 0xa06af50, 0xc009df21b0, 0x67b6f00, 0xc00c7e6630, 0xc0033796e8)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1025 +0x142
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc001d2e1a0, 0xa06af50, 0xc00160bfb0, 0x67b6f00, 0xc00c7e6630, 0xa027ed0, 0xc000dd6618, 0xc00869b180, 0xc009009e80, 0x7, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1026 +0xa45
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc001d2e1a0, 0xa06aea8, 0xc008a1f640, 0xa027ed0, 0xc000dd6618, 0xc00869b180, 0xc009009e80, 0x7, 0xc007f84c18, 0x16, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:290 +0x61d
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1(0xa06aea8, 0xc00cadf980)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:222 +0x19b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2(0xc00c1338c0, 0xa06aea8, 0xc00cadf980, 0x0, 0x0, 0xc0029a7680)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x162
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:434 +0x28e
Log preceding fatal error 

=== RUN   TestFullClusterBackup
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestFullClusterBackup122686519
    test_log_scope.go:74: use -show-logs to present logs inline

Reproduce

To reproduce, try: 

make stressrace TESTS=TestFullClusterBackup PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-race -parallel=4

/cc @cockroachdb/bulk-io pbardea

This test on roachdash | Improve this report!

@blathers-crl blathers-crl bot added the T-kv KV Team label Jul 26, 2021
@tbg
Copy link
Member

tbg commented Jul 27, 2021

Nathan fixed this in #68045

@tbg tbg closed this as completed Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tbg @cockroach-teamcity