Revert "kv: store span by pointer in latch struct" #114973

stevendanna · 2023-11-22T10:57:06Z

This reverts commit 831c61d.

This commit introduced a data race. I believe the fundamental data race is that we hand out a pointer here:

https://github.com/lyang24/cockroach/blob/831c61ded84ba2d5887736f7046a45c3cac3cd4c/pkg/kv/kvserver/spanlatch/manager.go#L224

But, in release, in some cases we attempt to reuse the underlying slice and zero it out before hand:

https://github.com/lyang24/cockroach/blob/831c61ded84ba2d5887736f7046a45c3cac3cd4c/pkg/kv/kvserver/spanset/spanset.go#L113-L115

Here, I just revert the offending commit rather than fixing it so that we can unblock CI.

Data Race Stack Trace

09:21:44     WARNING: DATA RACE
09:21:44     Read at 0x00c003dca940 by goroutine 642:
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*latch).Key()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:119 +0xfb
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*iterator).constrainMinSearchBounds.func1()
09:21:44           github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/spanlatch/latch_interval_btree.go:1087 +0x28
09:21:44       sort.Search()
09:21:44           GOROOT/src/sort/search.go:65 +0x53
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*iterator).constrainMinSearchBounds()
09:21:44           github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/spanlatch/latch_interval_btree.go:1086 +0xfa
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*iterator).FirstOverlap()
09:21:44           github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/spanlatch/latch_interval_btree.go:1069 +0x1b7
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).iterAndWait()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:551 +0x7a
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).wait()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:527 +0x5b9
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).Acquire()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:253 +0x1c4
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*latchManagerImpl).Acquire()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/latch_manager.go:28 +0x64
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:291 +0x6c7
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239 +0x328
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x514
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x575
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xc91
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x176
09:21:44       github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
09:21:44           github.com/cockroachdb/cockroach/pkg/server/node.go:1333 +0x7c4
09:21:44       github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
09:21:44           github.com/cockroachdb/cockroach/pkg/server/node.go:1464 +0x464
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x76
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x5ab
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf0
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x2f3
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
09:21:44           <autogenerated>:1 +0xdc
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6a
09:21:44       github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0x137
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x144
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x7a
09:21:44     FAIL: //pkg/cloud/userfile:userfile_test (see /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/testlogs/pkg/cloud/userfile/userfile_test/test.log)
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x519
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x199
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x422
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907 +0xed
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
09:21:44           <autogenerated>:1 +0x29
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211 +0x2a2
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189 +0x144
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x3a3
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
09:21:44           <autogenerated>:1 +0x6b
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2430 +0x1cf2
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1931 +0x9c4
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1499 +0x564
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1115 +0xa2f
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:224 +0xf2
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:1090 +0x181
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).send()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:1073 +0x71
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
09:21:44           <autogenerated>:1 +0x1f
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:922 +0x26f
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:945 +0x92
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:438 +0xa93
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:342 +0x165
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:527 +0x6d8
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:435 +0x897
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:348 +0x1059
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239 +0x328
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x514
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x504
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xc91
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x176
09:21:44       github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
09:21:44           github.com/cockroachdb/cockroach/pkg/server/node.go:1333 +0x7c4
09:21:44       github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
09:21:44           github.com/cockroachdb/cockroach/pkg/server/node.go:1464 +0x464
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x76
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x5ab
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf0
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x2f3
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
09:21:44           <autogenerated>:1 +0xdc
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6a
09:21:44       github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0x137
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x144
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x7a
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x519
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x199
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x422
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
09:21:44           github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907 +0xed
09:21:44       github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
09:21:44           <autogenerated>:1 +0x29
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211 +0x2a2
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189 +0x144
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x3a3
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
09:21:44           <autogenerated>:1 +0x6b
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2430 +0x1cf2
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1931 +0x9c4
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1499 +0x564
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1115 +0xa2f
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x214
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:47 +0x1a4
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).tryRefreshTxnSpans()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:627 +0x4d7
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).maybeRefreshPreemptively()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:513 +0x5d8
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:144 +0x154
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:195 +0x3fa
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:295 +0x244
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:114 +0x39d
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:246 +0x793
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:533 +0x9d3
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:1090 +0x181
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/txn.go:1218 +0x2d4
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*Txn).commit()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/txn.go:774 +0x204
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Commit()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/txn.go:784 +0xa4
09:21:44       github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1496 +0x98a
09:21:44       github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).commitTxn()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/internal.go:1267 +0x7c4
09:21:44       github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).newInternalExecutorWithTxn.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/internal.go:1695 +0xda
09:21:44       github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/internal.go:1828 +0x681
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/txn.go:1012 +0x8f
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.runTxn()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:1055 +0x64
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:1018 +0xec
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/db.go:993 +0x5c
09:21:44       github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn-fm()
09:21:44           <autogenerated>:1 +0x1b
09:21:44       github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/internal.go:1800 +0x635
09:21:44     Target //pkg/cloud/userfile:userfile_test up-to-date:
09:21:44       github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/internal.go:1735 +0xec
09:21:44       _bazel/bin/pkg/cloud/userfile/userfile_test_/userfile_test
09:21:44       github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).updateSchedule()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:168 +0x3fa
09:21:44       github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).start.func1()
09:21:44           github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:117 +0x65d
09:21:44       github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1ea
09:21:44     
09:21:44     Previous write at 0x00c003dca940 by goroutine 232:
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).Release()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:114 +0x1b7
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.releaseGuard()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:674 +0x64
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).FinishReq()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:453 +0x1d9
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:1367 +0x36f
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ProposalData).finishApplication()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:242 +0x9d
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicatedCmd).AckOutcomeAndFinish()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_cmd.go:156 +0x11a
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.AppliedCommand.AckOutcomeAndFinish()
09:21:44           <autogenerated>:1 +0x55
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.forEachAppliedCmdIter()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:268 +0x135
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:301 +0x2b9
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0xce
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1b58
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x21b
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x1ef
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x306
09:21:44       github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2()
09:21:44           github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x94
09:21:44       github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
09:21:44           github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1ea

Epic: none

Release note: None

This reverts commit 831c61d.

cockroach-teamcity · 2023-11-22T10:57:16Z

This change is

stevendanna · 2023-11-22T11:12:08Z

bors r=kvoli,arulajmani

stevendanna · 2023-11-22T11:51:31Z

This can also cause a NPE:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4666bf]

goroutine 2007938 [running]:
panic({0x5765ce0?, 0xb19d3c0?})
	GOROOT/src/runtime/panic.go:1017 +0x3ac fp=0xc0034b49e0 sp=0xc0034b4930 pc=0x49c40c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc001110fc0?, {0x7883150, 0xc022624f60})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65 fp=0xc0034b4a28 sp=0xc0034b49e0 pc=0x115fb85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr.func1()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:333 +0x28 fp=0xc0034b4a50 sp=0xc0034b4a28 pc=0x1160748
runtime.deferCallSave(0xc0034b4b08, 0xc0034b78f0?)
	GOROOT/src/runtime/panic.go:798 +0x84 fp=0xc0034b4a60 sp=0xc0034b4a50 pc=0x49bfc4
runtime.runOpenDeferFrame(0xc013ee1130)
	GOROOT/src/runtime/panic.go:771 +0x1b8 fp=0xc0034b4aa0 sp=0xc0034b4a60 pc=0x49bdf8
panic({0x5765ce0?, 0xb19d3c0?})
	GOROOT/src/runtime/panic.go:914 +0x21f fp=0xc0034b4b50 sp=0xc0034b4aa0 pc=0x49c27f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109 +0x14e fp=0xc0034b4b90 sp=0xc0034b4b50 pc=0x3b0a0ce
panic({0x5765ce0?, 0xb19d3c0?})
	GOROOT/src/runtime/panic.go:920 +0x270 fp=0xc0034b4c40 sp=0xc0034b4b90 pc=0x49c2d0
runtime.panicmem(...)
	GOROOT/src/runtime/panic.go:261
runtime.sigpanic()
	GOROOT/src/runtime/signal_unix.go:861 +0x378 fp=0xc0034b4ca0 sp=0xc0034b4c40 pc=0x4b3b18
cmpbody()
	src/internal/bytealg/compare_amd64.s:59 +0x3f fp=0xc0034b4ca8 sp=0xc0034b4ca0 pc=0x4666bf
bytes.Compare(...)
	GOROOT/src/bytes/bytes.go:27
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.keyBound.contains({{0x0?, 0x18?, 0xc0034b4d50?}, 0xcc?}, 0x3959cfb?)
	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/kv/kvserver/spanlatch/latch_interval_btree.go:88 +0x51 fp=0xc0034b4cf0 sp=0xc0034b4ca8 pc=0x39552d1
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*iterator).findNextOverlap(0xc0034b4ed8, 0x0?)
	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/kv/kvserver/spanlatch/latch_interval_btree.go:1145 +0x47e fp=0xc0034b4d60 sp=0xc0034b4cf0 pc=0x395a41e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*iterator).FirstOverlap(0xc0034b4ed8, 0x7883150?)
	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/kv/kvserver/spanlatch/latch_interval_btree.go:1071 +0xf4 fp=0xc0034b4d90 sp=0xc0034b4d60 pc=0x3959bf4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).iterAndWait(0xc0034b4e60?, {0x7883150, 0xc0226250b0}, 0xc0218fd508?, 0xc0034b4ed8, 0x39?, 0x10?, 0xc01f2004c0?, 0xc02457b788, 0x65afb48)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:551 +0x56 fp=0xc0034b4e18 sp=0xc0034b4d90 pc=0x3953e76
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).wait(0xc0036fa700?, {0x7883150, 0xc0226250b0}, 0xc02457b680, {{{{0xc007598360, 0xf}, {0xc026188120, 0x7}}, {{0xc025a60fc0, 0x19}, ...}}})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:527 +0x405 fp=0xc0034b5000 sp=0xc0034b4e18 pc=0x3953c85
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).Acquire(0x0?, {0x7883150, 0xc0226250b0}, 0xc01f63d5c0?, 0x0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:253 +0x18d fp=0xc0034b5148 sp=0xc0034b5000 pc=0x395244d

craig · 2023-11-22T11:53:07Z

Build succeeded:

Bazel Essential CI (Cockroach)

Revert "kv: store span by pointer in latch struct"

f2b7e9f

This reverts commit 831c61d.

stevendanna requested a review from a team as a code owner November 22, 2023 10:57

arulajmani approved these changes Nov 22, 2023

View reviewed changes

kvoli approved these changes Nov 22, 2023

View reviewed changes

craig bot merged commit d23688e into cockroachdb:master Nov 22, 2023
3 checks passed

rafiss mentioned this pull request Nov 22, 2023

sql/catalog/lease: TestKVWriterMatchesIEWriter failed #114954

Closed

erikgrinaker mentioned this pull request Nov 27, 2023

kv/kvclient/rangefeed: TestDBClientScan failed #113750

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert "kv: store span by pointer in latch struct" #114973

Revert "kv: store span by pointer in latch struct" #114973

stevendanna commented Nov 22, 2023 •

edited

Loading

cockroach-teamcity commented Nov 22, 2023

stevendanna commented Nov 22, 2023

stevendanna commented Nov 22, 2023

craig bot commented Nov 22, 2023

Revert "kv: store span by pointer in latch struct" #114973

Revert "kv: store span by pointer in latch struct" #114973

Conversation

stevendanna commented Nov 22, 2023 • edited Loading

cockroach-teamcity commented Nov 22, 2023

stevendanna commented Nov 22, 2023

stevendanna commented Nov 22, 2023

craig bot commented Nov 22, 2023

stevendanna commented Nov 22, 2023 •

edited

Loading