Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: panic.go:884: panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | gi... #110641

Closed
cockroach-sentry opened this issue Sep 14, 2023 · 1 comment
Closed

Sentry: panic.go:884: panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | gi... #110641

cockroach-sentry opened this issue Sep 14, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Sep 14, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4477661726/?referrer=webhooks_plugin

Panic Message:

panic.go:884: panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:121
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:890
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache.(*intervalSkl).AddRange
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache/interval_skl.go:262
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache.(*sklImpl).Add
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache/skl_impl.go:57
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).addToTSCacheChecked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go:51
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).updateTimestampCache.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go:65
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).updateTimestampCache
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go:239
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:1276
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:173
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:481
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:176
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:205
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1092
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1141
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:724
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:95
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:795
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:263
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:795
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:232
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:341
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:230
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:795
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:734
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:225
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:879
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:764
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:887
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:190
Wraps: (2) panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0
panic.go:884: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 189 to 191 in 07a53a3

ba.Replica = r
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 210 to 212 in 07a53a3

}
reply, err := iface.Batch(ctx, &ba)
// If we queried a remote node, perform extra validation and

https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L886-L888
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L763-L765
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L878-L880

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 224 to 226 in 07a53a3

if grpcutil.IsLocalRequestContext(ctx) {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L733-L735
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L794-L796
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L229-L231

cockroach/pkg/util/stop/stopper.go

Lines 340 to 342 in 07a53a3

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L231-L233
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L794-L796
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L262-L264
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L794-L796

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 94 to 96 in 07a53a3

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L723-L725

cockroach/pkg/server/node.go

Lines 1140 to 1142 in 07a53a3

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1091 to 1093 in 07a53a3

var pErr *roachpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, *args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L204-L206
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L175-L177
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L480-L482
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go#L172-L174
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L1275-L1277
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go#L238-L240
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go#L64-L66
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go#L50-L52

cockroach/pkg/kv/kvserver/tscache/skl_impl.go

Lines 56 to 58 in 07a53a3

} else {
tc.cache.AddRange(nonNil(start), end, excludeTo, val)
}

cockroach/pkg/kv/kvserver/tscache/interval_skl.go

Lines 261 to 263 in 07a53a3

log.Errorf(context.Background(), "%s, [%s,%s)", msg, from, to)
panic(redact.Safe(msg))
case cmp == 0:

GOROOT/src/runtime/panic.go#L889-L891
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L120-L122
GOROOT/src/runtime/panic.go#L883-L885

pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 190
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 887
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 764
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 879
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 225
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 734
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 795
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 230
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 341
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 232
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 795
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 263
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 795
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 95
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 724
pkg/server/node.go in pkg/server.(*Node).Batch at line 1141
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1092
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 205
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 176
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 481
pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 173
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*endCmds).done at line 1276
pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go in pkg/kv/kvserver.(*Replica).updateTimestampCache at line 239
pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go in pkg/kv/kvserver.(*Replica).updateTimestampCache.func1 at line 65
pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go in pkg/kv/kvserver.(*Replica).addToTSCacheChecked at line 51
pkg/kv/kvserver/tscache/skl_impl.go in pkg/kv/kvserver/tscache.(*sklImpl).Add at line 57
pkg/kv/kvserver/tscache/interval_skl.go in pkg/kv/kvserver/tscache.(*intervalSkl).AddRange at line 262
GOROOT/src/runtime/panic.go in runtime.gopanic at line 890
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 121
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag Value
Command start-single-node
Environment v22.2.2
Go Version go1.19.1
Platform linux amd64
Distribution CCL
Cockroach Release v22.2.2
Cockroach SHA 07a53a3
# of CPUs 20
# of Goroutines 482

Jira issue: CRDB-31539
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Sep 14, 2023
@yuzefovich
Copy link
Member

dup of #97824

@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: panic.go:884: panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | gi... Sentry: panic.go:884: panic: inverted range (issue #32149): key lens = [4,15), diff @ index 0 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | gi... Sep 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry