Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestSchemaChangeAfterCreateInTxn failed #126075

Closed
cockroach-teamcity opened this issue Jun 23, 2024 · 1 comment · Fixed by #126114
Closed

sql: TestSchemaChangeAfterCreateInTxn failed #126075

cockroach-teamcity opened this issue Jun 23, 2024 · 1 comment · Fixed by #126114
Assignees
@rafiss
Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jun 23, 2024 •
edited by cockroach-jira-scripts
Loading

sql.TestSchemaChangeAfterCreateInTxn failed on release-24.1 @ 7cf9592bd59077500801e5827e5b8230e9a112f9:

github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0013b4d88, {0x82a7c88, 0xc02664db30}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:839 +0x86e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc0013b4e88, {0x82a7c88, 0xc02664db30}, {0x82f3468?, 0xc01fb582e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:230 +0x808
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7fbc48922f18, 0xc0013b5268}, 0xc0164245f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x14f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0164250f8, {0x82a7c88, 0xc00ddfb770}, {0x82de7c0, 0xc0013b5208})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1e8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0164250f8, {0x82a7c88, 0xc00ddfb770})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1098 +0x1385
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757 +0x190
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc01c36ca88, 0xfdc0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x125
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc012cdb500, {0x82a7c88, 0xc0240e0c90}, {0x82962e0, 0xc01c36ca88}, 0xc010d46c08)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x82a7c88?, 0xc0240e0c90?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x3f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10143217
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
Other goroutines holding locks:
goroutine 10146737 lock 0xc01560d078
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4638 concurrency.(*lockTableImpl).verify ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4637 concurrency.(*lockTableImpl).verify ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:98 concurrency.verifyingLockTable.UpdateLocks ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:554 concurrency.(*managerImpl).OnLockUpdated ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:837 kvserver.(*Replica).handleReadWriteLocalEvalResult ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:233 kvserver.(*replicaStateMachine).ApplySideEffects ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 apply.mapCheckedCmdIter ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 apply.(*Task).applyOneBatch ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:250 apply.(*Task).ApplyCommittedEntries ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:453 kvserver.(*replicaStateMachine).moveStats ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:756 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:688 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:420 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 55 lock 0xc00e704c98
github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:742 admission.(*GrantCoordinator).CPULoad ??? <<<<<
github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:741 admission.(*GrantCoordinator).CPULoad ???
github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:241 goschedstats.(*schedStatsTicker).getStatsOnTick ???
github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:171 goschedstats.init.0.func1 ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=7
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-39759
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jun 23, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Jun 23, 2024
@rafiss rafiss mentioned this issue Jun 24, 2024
Closed
@rafiss
Copy link
Collaborator

rafiss commented Jun 24, 2024

there is a potential deadlock in kvserver.(*Replica).handleRaftReady. similar to #115541, i think it's just because the test runs slower.

POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 10146737 lock 0xc0013b4e58
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:755 kvserver.(*Replica).handleRaftReady ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:754 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:688 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:420 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 stop.(*Stopper).RunAsyncTaskEx.func2 ???

Have been trying to lock it again for more than 5m0s
goroutine 11799873 lock 0xc0013b4e58
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:335 kvserver.(*Replica).evalAndPropose.func6 ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:334 kvserver.(*Replica).evalAndPropose.func6 ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:348 kvserver.(*Replica).executeWriteBatch ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:511 kvserver.(*Replica).executeBatchWithConcurrencyRetries ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:197 kvserver.(*Replica).SendWithWriteBytes ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:192 kvserver.(*Store).SendWithWriteBytes ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 kvserver.(*Stores).SendWithWriteBytes ???
github.com/cockroachdb/cockroach/pkg/server/node.go:1435 server.(*Node).batchInternal ???
github.com/cockroachdb/cockroach/pkg/server/node.go:1634 server.(*Node).Batch ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 rpc.makeInternalClientAdapter.func1 ???
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96 rpc.NewServerEx.ServerInterceptor.func12 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:168 rpc.NewServerEx.func3 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:94 rpc.kvAuth.unaryInterceptor ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:135 rpc.NewServerEx.func1.1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 stop.(*Stopper).RunTaskWithErr ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:137 rpc.NewServerEx.func1 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:814 rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 rpc.makeInternalClientAdapter.func2 ???
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:227 rpc.NewContext.ClientInterceptor.func8 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:898 rpc.getChainUnaryInvoker.func1 ???
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 rpc.makeInternalClientAdapter.func3 ???
<autogenerated>:0 rpc.(*internalClientAdapter).Batch ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:213 kvcoord.(*grpcTransport).sendBatch ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:191 kvcoord.(*grpcTransport).SendNext ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2749 kvcoord.(*DistSender).sendToReplicas ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2090 kvcoord.(*DistSender).sendPartialBatch ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1615 kvcoord.(*DistSender).divideAndSendBatchToRanges ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1234 kvcoord.(*DistSender).Send ???
github.com/cockroachdb/cockroach/pkg/kv/db.go:228 kv.(*CrossRangeTxnWrapperSender).Send ???
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:333 requestbatcher.(*RequestBatcher).sendBatch.func1.1 ???
github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:28 timeutil.RunWithTimeout ???
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:351 requestbatcher.(*RequestBatcher).sendBatch.func1.2 ???
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:373 requestbatcher.(*RequestBatcher).sendBatch.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 stop.(*Stopper).RunAsyncTaskEx.func2 ???

Here is what goroutine 10146737 doing now
goroutine 10146737 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0xffffffffffffffff?, 0x0?, 0x0?)
	GOROOT/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc4ee5c0)
	GOROOT/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
	GOROOT/src/sync/mutex.go:90
github.com/sasha-s/go-deadlock.(*lockOrder).preLock(0xc4ee5c0, {0xc012e59520, 0x11, 0x32}, {0x5f92280, 0xc00b3756f8})
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:280 +0xb3
github.com/sasha-s/go-deadlock.preLock(...)
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:159
github.com/sasha-s/go-deadlock.lock(0xc016423290, {0x5f92280, 0xc00b3756f8})
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:176 +0x77
github.com/sasha-s/go-deadlock.(*Mutex).Lock(...)
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:85
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify.func1(0xc00b3756c0, 0xc01560d040)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4644 +0x65
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc01560d040)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4647 +0x145
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7fbc488b00e8?, 0xc01560d040?}}, 0x14?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x89
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x0?, {0x82a7c88, 0xc02664db30}, 0x0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:555 +0x30
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0013b4d88, {0x82a7c88, 0xc02664db30}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:839 +0x86e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc0013b4e88, {0x82a7c88, 0xc02664db30}, {0x82f3468?, 0xc01fb582e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:230 +0x808
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7fbc48922f18, 0xc0013b5268}, 0xc0164245f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x14f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0164250f8, {0x82a7c88, 0xc00ddfb770}, {0x82de7c0, 0xc0013b5208})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1e8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0164250f8, {0x82a7c88, 0xc00ddfb770})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1098 +0x1385
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757 +0x190
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc01c36ca88, 0xfdc0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x125
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc012cdb500, {0x82a7c88, 0xc0240e0c90}, {0x82962e0, 0xc01c36ca88}, 0xc010d46c08)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x82a7c88?, 0xc0240e0c90?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x3f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10143217
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe

@rafiss rafiss self-assigned this Jun 24, 2024
@rafiss rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jun 24, 2024
@rafiss rafiss mentioned this issue Jun 24, 2024
Merged
@craig craig bot closed this as completed in 6693f4a Jun 24, 2024
This was referenced Jun 24, 2024
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafiss rafiss

Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
No open projects
SQL Foundations
Status: Done
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

sql: avoid slow lock verification in TestSchemaChangeAfterCreateInTxn rafiss/cockroach
2 participants
@rafiss @cockroach-teamcity