Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/multiregionccl: TestIndexCleanupAfterAlterFromRegionalByRow failed #68046

Closed
cockroach-teamcity opened this issue Jul 26, 2021 · 1 comment
Closed

ccl/multiregionccl: TestIndexCleanupAfterAlterFromRegionalByRow failed #68046

cockroach-teamcity opened this issue Jul 26, 2021 · 1 comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team

Comments

@cockroach-teamcity
Copy link
Member

ccl/multiregionccl.TestIndexCleanupAfterAlterFromRegionalByRow failed with artifacts on master @ f7528c59e296ed9acd2a20d590f2a42bbad0dcd0:

Fatal error:

panic: mutex is not write locked [recovered]
	panic: mutex is not write locked [recovered]
	panic: mutex is not write locked

Stack:

goroutine 154962 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00288b170, 0xad631f0, 0xc003059980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x15d
panic(0x6b6ece0, 0xac56c00)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send.func1(0xc001ea4a30, 0xc001ea4ae8, 0xc0070ef800, 0xc001ea4ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:85 +0x745
panic(0x6b6ece0, 0xac56c00)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).AssertHeld(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync_race.go:49
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).MaybeGossipSystemConfigRaftMuLocked(0xc00348ea80, 0xad631f0, 0xc003059a40, 0xc001ea2080, 0x50d6319)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_gossip.go:83 +0x865
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).MaybeGossipSystemConfigIfHaveFailureRaftMuLocked(0xc00348ea80, 0xad631f0, 0xc003059a40, 0xc005b0efc0, 0x5db2a8)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_gossip.go:141 +0xb0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc00348ea80, 0xad631f0, 0xc003059a40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:718 +0xafa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc00348ea80, 0xad631f0, 0xc003059a40, 0xc0081c3a20, 0xc0042f5a70, 0x169540ccce19cdcb, 0x1, 0x0, 0x300000003, 0x2, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:118 +0x1718
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc00348ea80, 0xad631f0, 0xc003059a40, 0xc0081c3a20, 0xc0042f5a70, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:149 +0x84e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00348ea80, 0xad631f0, 0xc003059a40, 0xc0081c3a20, 0x7ad2130, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:376 +0x4a2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00348ea80, 0xad631f0, 0xc003059a10, 0x6, 0xc0081c3a20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:107 +0x652
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:43
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0070ef800, 0xad631f0, 0xc0030599b0, 0x169540ccc52d34bf, 0x1, 0x300000003, 0x2, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:180 +0x5b3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc00669ea00, 0xad631f0, 0xc0030599b0, 0x0, 0x0, 0x300000003, 0x2, 0x0, 0x6, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:191 +0x198
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0xad631f0, 0xc003059980, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:901 +0x27e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00288b170, 0xad631f0, 0xc003059980, 0x74350a2, 0x10, 0xc001167100, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324 +0x12a
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc006443200, 0xad631f0, 0xc003059980, 0xc0081c3970, 0x0, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:889 +0x267
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc006443200, 0xad631f0, 0xc0030598f0, 0xc0081c3970, 0xc006443200, 0xc004cf43c8, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:963 +0x118
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:7967 +0xa2
github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0xc00796a640, 0xc006953ec0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:126 +0x67d
google.golang.org/grpc.getChainUnaryHandler.func1(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0x70e25c0, 0x1, 0xc005f38ec0, 0xc001167690)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1127 +0x150
github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0xc00796a640, 0xc005f38ec0, 0xc005f38ec0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:173 +0x125
google.golang.org/grpc.getChainUnaryHandler.func1(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0x0, 0xf6cb5b, 0x40, 0x7fa1f77242b0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1127 +0x150
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0xc00796a640, 0xc005f38e80, 0xc001167848, 0xf4c7ac, 0xc005f38eb0, 0xc005f38e80)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/auth.go:71 +0x12f
google.golang.org/grpc.chainUnaryServerInterceptors.func1(0xad631f0, 0xc0030598f0, 0x73d6d60, 0xc0081c3970, 0xc00796a640, 0xc006953ec0, 0xc00aacf948, 0x461d38, 0x30, 0x70fdd80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1113 +0x113
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x73831e0, 0xc006443200, 0xad631f0, 0xc0030598f0, 0xc002e64fc0, 0xc001774900, 0xad631f0, 0xc0030598f0, 0xc0055b1800, 0xae)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:7969 +0x1f2
google.golang.org/grpc.(*Server).processUnaryRPC(0xc004c3b180, 0xadcbe18, 0xc008d2c480, 0xc005b0e7e0, 0xc001bcd6e0, 0xd2a02a0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1286 +0xa66
google.golang.org/grpc.(*Server).handleStream(0xc004c3b180, 0xadcbe18, 0xc008d2c480, 0xc005b0e7e0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1609 +0x142c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc007793970, 0xc004c3b180, 0xadcbe18, 0xc008d2c480, 0xc005b0e7e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:934 +0xe7
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:932 +0x355
Log preceding fatal error 

E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:718
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:118
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:149
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:376
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:107
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:43
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:180
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:191
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:901
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:889
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:963
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:7967
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:126
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | google.golang.org/grpc.getChainUnaryHandler.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1127
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:173
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | google.golang.org/grpc.getChainUnaryHandler.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1127
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/auth.go:71
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | google.golang.org/grpc.chainUnaryServerInterceptors.func1
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1113
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:7969
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | google.golang.org/grpc.(*Server).processUnaryRPC
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1286
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | google.golang.org/grpc.(*Server).handleStream
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1609
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | google.golang.org/grpc.(*Server).serveStreams.func1.2
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:934
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | runtime.goexit
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +Wraps: (2) panic: mutex is not write locked
E210726 05:41:28.850041 154962 1@util/log/logcrash/crash_reporting.go:140  [n3] 14 +Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

To reproduce, try: 

make stressrace TESTS=TestIndexCleanupAfterAlterFromRegionalByRow PKG=./pkg/ccl/multiregionccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-race -parallel=4

Same failure on other branches

/cc @cockroachdb/sql-queries ajstorm

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 26, 2021
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Jul 26, 2021
@otan
Copy link
Contributor

otan commented Jul 26, 2021

resolved by #68045

@otan otan closed this as completed Jul 26, 2021
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@otan @cockroach-teamcity