Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1709175563.894281246,2->1709175563.574810504,0 (1) attach... #119761

Closed
cockroach-sentry opened this issue Feb 29, 2024 · 1 comment
Closed

Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1709175563.894281246,2->1709175563.574810504,0 (1) attach... #119761

cockroach-sentry opened this issue Feb 29, 2024 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Feb 29, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5022352518/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1709175563.894281246,2->1709175563.574810504,0
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:385
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:198
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:259
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2)
Wraps: (3) tags: [n5,client=23.79.233.41:42392,hostssl,user=×,f×,distsql.stmt=×,distsql.gateway=5,distsql.txn=×,streamID=×,received-error]
Wraps: (4) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.txn=×,streamID=×,sent-error=]
Wraps: (5)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/cockroach/pkg/sql/colexecerror/*colexecerror.StorageError
Wraps: (6)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:779
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:551
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1206
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1306
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:849
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:284
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:251
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:249
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:960
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:859
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1044
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:929
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1052
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2238
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1741
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1312
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:933
  | github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:224
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1012
Wraps: (7) failed indeterminate commit recovery
Wraps: (8) assertion failure
Wraps: (9)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.RecoverTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go:133
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:526
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:294
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:708
  | github.com...
Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 258 to 260 in b95f222

case parallelUnorderedSynchronizerStateRunning:
if err := colexecerror.CatchVectorizedRuntimeError(s.nextBatch[inputIdx]); err != nil {
if s.getState() == parallelUnorderedSynchronizerStateDraining && s.Ctx.Err() == nil && s.cancelLocalInput[inputIdx] != nil {

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in b95f222

}()
operation()
return retErr

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 197 to 199 in b95f222

return func() {
s.batches[inputIdx] = inputOp.Next()
}

cockroach/pkg/sql/colflow/stats.go

Lines 125 to 127 in b95f222

// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in b95f222

}()
operation()
return retErr

cockroach/pkg/sql/colflow/stats.go

Lines 117 to 119 in b95f222

func (bic *batchInfoCollector) next() {
bic.batch = bic.Input.Next()
}

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 384 to 386 in b95f222

if receivedErr != nil {
colexecerror.ExpectedError(receivedErr)
}

cockroach/pkg/sql/colexecerror/error.go

Lines 202 to 204 in b95f222

func ExpectedError(err error) {
panic(newNotInternalError(err))
}

GOROOT/src/runtime/panic.go#L883-L885

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in b95f222

}
err := logcrash.PanicAsError(0, panicObj)
log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

cockroach/pkg/kv/db.go

Lines 1011 to 1013 in b95f222

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/db.go

Lines 223 to 225 in b95f222

br, pErr := s.wrapped.Send(ctx, ba)
if _, ok := pErr.GetDetail().(*kvpb.OpRequiresTxnError); !ok {

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 932 to 934 in b95f222

} else {
rpl, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, 0 /* batchIdx */)
}

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1311 to 1313 in b95f222

if !ri.NeedAnother(rs) {
resp := ds.sendPartialBatch(
ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(), nil, /* positions */

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1740 to 1742 in b95f222

prevTok = routingTok
reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2237 to 2239 in b95f222

}
br, err = transport.SendNext(ctx, ba)
ds.maybeIncrementErrCounters(br, err)

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in b95f222

}
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 210 to 212 in b95f222

log.VEvent(ctx, 2, "sending batch request")
reply, err := iface.Batch(ctx, ba)
log.VEvent(ctx, 2, "received batch response")

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1051-L1053
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L928-L930
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1043-L1045

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in b95f222

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L858-L860
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in b95f222

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L250-L252
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L283-L285
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in b95f222

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L848-L850

cockroach/pkg/server/node.go

Lines 1305 to 1307 in b95f222

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1205 to 1207 in b95f222

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L550-L552
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L778-L780

cockroach/pkg/kv/kvserver/txnrecovery/manager.go

Lines 108 to 110 in b95f222

log.VEventf(ctx, 2, "recovery error: %v", res.Err)
return nil, errors.Wrap(res.Err, "failed indeterminate commit recovery")
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1051-L1053
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L928-L930
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L1043-L1045

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 227 to 229 in b95f222

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L858-L860
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in b95f222

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L250-L252
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L283-L285
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L959-L961

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 96 to 98 in b95f222

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/rpc/pkg/rpc/context.go#L848-L850

cockroach/pkg/server/node.go

Lines 1305 to 1307 in b95f222

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1205 to 1207 in b95f222

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L494-L496
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L175-L177
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L122-L124
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L843-L845
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L755-L757
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L441-L443
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L667-L669
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L707-L709
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L293-L295
https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L525-L527

cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go

Lines 132 to 134 in b95f222

if was, is := args.Txn.WriteTimestamp, reply.RecoveredTxn.WriteTimestamp; was != is {
return result.Result{}, errors.AssertionFailedf(
"programming error: timestamp change by implicitly committed transaction: %v->%v", was, is, 

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1 at line 259
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.func1.1 at line 198
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 126
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 118
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 385
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 203
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1012
pkg/kv/db.go in pkg/kv.(*CrossRangeTxnWrapperSender).Send at line 224
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 933
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1312
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1741
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2238
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1052
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 929
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1044
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 859
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 849
pkg/server/node.go in pkg/server.(*Node).Batch at line 1306
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1206
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 551
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 779
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1052
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 929
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1044
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 859
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 960
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 849
pkg/server/node.go in pkg/server.(*Node).Batch at line 1306
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1206
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 495
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 176
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 123
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 844
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 756
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 442
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 668
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 708
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 294
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 526
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 133

Tags

Tag Value
Command server
Environment v23.1.14
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.14
Cockroach SHA b95f222
# of CPUs 32
# of Goroutines 2212

Jira issue: CRDB-36298
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Feb 29, 2024
@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1709175563.894281246,2->1709175563.574810504,0 (1) attach... Sentry: cmd_recover_txn.go:133: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1709175563.894281246,2->1709175563.574810504,0 (1) attach... Mar 15, 2024
@yuzefovich
Copy link
Member

fixed by #107658

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry