kvserver: v22.1.5: inverted range (issue #32149) #97824

cockroach-teamcity · 2023-03-01T00:01:58Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/3964991526/?referrer=webhooks_plugin

Panic message:

panic.go:1038: panic: inverted range (issue #32149): key lens = [3,48), diff @ index 0
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:251
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:95
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1047
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache.(*intervalSkl).AddRange
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache/interval_skl.go:263
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache.(*sklImpl).Add
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache/skl_impl.go:57
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).addToTSCacheChecked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go:68
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).updateTimestampCache.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go:82
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).updateTimestampCache
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go:218
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:1200
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:149
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:478
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithoutRangeID
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:170
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:100
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:197
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:191
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
| github.com/cockroachdb/cockroach/pkg/server/node.go:1006
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| github.com/cockroachdb/cockroach/pkg/server/node.go:989
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| github.com/cockroachdb/cockroach/pkg/server/node.go:1058
| github.com/cockroachdb/cockroach/pkg/roachpb.Internal_Batch_Handler.func1
| github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_go_proto/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9490
| github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:136
| google.golang.org/grpc.chainUnaryInterceptors.func1.1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1116
| github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func3
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:238
| google.golang.org/grpc.chainUnaryInterceptors.func1.1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1119
| github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:207
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
| github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:205
| google.golang.org/grpc.chainUnaryInterceptors.func1.1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1119
| google.golang.org/grpc.chainUnaryInterceptors.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1121
| github.com/cockroachdb/cockroach/pkg/roachpb.Internal_Batch_Handler
| github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_go_proto/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9492
Wraps: (2) panic: inverted range (issue #32149): key lens = [3,48), diff @ index 0
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: panic: inverted range (issue #32149): key lens = [3,48), diff @ index 0
panic.go:1038: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic

cockroach/pkg/util/stop/stopper.go

Lines 250 to 252 in a30a663

    
           	logcrash.ReportPanicWithGlobalSettings(ctx, r, 1) 
        
           	panic(r) 
        
           }

in pkg/util/stop.(*Stopper).Recover
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L94-L96 in pkg/kv/kvserver.(*Store).Send.func1
GOROOT/src/runtime/panic.go#L1046-L1048 in runtime.gopanic

cockroach/pkg/kv/kvserver/tscache/interval_skl.go

Lines 262 to 264 in a30a663

    
           	log.Errorf(context.Background(), "%s, [%s,%s)", msg, from, to) 
        
           	panic(redact.Safe(msg)) 
        
           case cmp == 0:

in pkg/kv/kvserver/tscache.(*intervalSkl).AddRange

cockroach/pkg/kv/kvserver/tscache/skl_impl.go

Lines 56 to 58 in a30a663

    
           } else { 
        
           	tc.cache.AddRange(nonNil(start), end, excludeTo, val) 
        
           }

cockroach/pkg/server/node.go

Lines 1005 to 1007 in a30a663

    
           var pErr *roachpb.Error 
        
           br, pErr = n.stores.Send(ctx, *args) 
        
           if pErr != nil {

in pkg/server.(*Node).batchInternal.func1

cockroach/pkg/util/stop/stopper.go

Lines 343 to 345 in a30a663

    
           	return f(ctx) 
        
           }

in pkg/util/stop.(*Stopper).RunTaskWithErr

cockroach/pkg/server/node.go

Lines 988 to 990 in a30a663

    
           var br *roachpb.BatchResponse 
        
           if err := n.stopper.RunTaskWithErr(ctx, "node.Node: batch", func(ctx context.Context) error { 
        
           	var reqSp spanForRequest

in pkg/server.(*Node).batchInternal

cockroach/pkg/server/node.go

Lines 1057 to 1059 in a30a663


	br, err := n.batchInternal(ctx, tenantID, args)

in pkg/server.(*Node).Batch
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/roachpb/api.pb.go#L9489-L9491 in pkg/roachpb._Internal_Batch_Handler.func1

cockroach/pkg/util/tracing/grpc_interceptor.go

Lines 135 to 137 in a30a663

    
           if methodExcludedFromTracing(info.FullMethod) { 
        
           	return handler(ctx, req) 
        
           }

in pkg/util/tracing.ServerInterceptor.func1
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1115-L1117 in google.golang.org/grpc.chainUnaryInterceptors.func1.1
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/rpc/pkg/rpc/context.go#L237-L239 in pkg/rpc.NewServer.func3
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1118-L1120 in google.golang.org/grpc.chainUnaryInterceptors.func1.1
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/rpc/pkg/rpc/context.go#L206-L208 in pkg/rpc.NewServer.func1.1

cockroach/pkg/util/stop/stopper.go

Lines 343 to 345 in a30a663

    
           	return f(ctx) 
        
           }

in pkg/util/stop.(*Stopper).RunTaskWithErr
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/rpc/pkg/rpc/context.go#L204-L206 in pkg/rpc.NewServer.func1
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1118-L1120 in google.golang.org/grpc.chainUnaryInterceptors.func1.1
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1120-L1122 in google.golang.org/grpc.chainUnaryInterceptors.func1
https://github.com/cockroachdb/cockroach/blob/a30a663cbd9323d34d50f343dd038af64671e25f/pkg/roachpb/api.pb.go#L9491-L9493 in pkg/roachpb._Internal_Batch_Handler

GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).Recover at line 251
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).Send.func1 at line 95
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1047
pkg/kv/kvserver/tscache/interval_skl.go in pkg/kv/kvserver/tscache.(*intervalSkl).AddRange at line 263
pkg/kv/kvserver/tscache/skl_impl.go in pkg/kv/kvserver/tscache.(*sklImpl).Add at line 57
pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go in pkg/kv/kvserver.(*Replica).addToTSCacheChecked at line 68
pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go in pkg/kv/kvserver.(*Replica).updateTimestampCache.func1 at line 82
pkg/kv/kvserver/pkg/kv/kvserver/replica_tscache.go in pkg/kv/kvserver.(*Replica).updateTimestampCache at line 218
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*endCmds).done at line 1200
pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 149
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 478
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).sendWithoutRangeID at line 170
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).Send at line 100
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).Send at line 197
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).Send at line 191
pkg/server/node.go in pkg/server.(*Node).batchInternal.func1 at line 1006
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 344
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 989
pkg/server/node.go in pkg/server.(*Node).Batch at line 1058
pkg/roachpb/api.pb.go in pkg/roachpb._Internal_Batch_Handler.func1 at line 9490
pkg/util/tracing/grpc_interceptor.go in pkg/util/tracing.ServerInterceptor.func1 at line 136
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1116
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServer.func3 at line 238
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1119
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServer.func1.1 at line 207
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 344
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServer.func1 at line 205
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1119
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1 at line 1121
pkg/roachpb/api.pb.go in pkg/roachpb._Internal_Batch_Handler at line 9492

Tag	Value
Cockroach Release	`v22.1.5`
Cockroach SHA:	`a30a663`
Platform	windows amd64
Distribution	CCL
Environment	v22.1.5
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-24903

The text was updated successfully, but these errors were encountered:

github-actions · 2024-05-02T11:04:12Z

We have marked this issue as stale because it has been inactive for
12 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy.

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Mar 1, 2023

yuzefovich changed the title sentry: panic.go:1038: panic: inverted range (issue #32149): key lens = [3,48), diff @ index 0 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:1038 | github.c... kvserver: v22.1.5: inverted range (issue #32149) Apr 5, 2023

blathers-crl bot added the T-kv KV Team label Apr 5, 2023

yuzefovich mentioned this issue Apr 5, 2023

sentry: panic.go:1038: panic: inverted range (issue #32149): key lens = [3,48), diff @ index 0 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:1038 | github.c... #97825

Closed

github-actions bot added the no-issue-activity label May 2, 2024

github-actions bot added the X-stale label May 13, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 13, 2024

exalate-issue-sync bot closed this as completed May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvserver: v22.1.5: inverted range (issue #32149) #97824

kvserver: v22.1.5: inverted range (issue #32149) #97824

cockroach-teamcity commented Mar 1, 2023 •

edited by cockroach-jira-scripts

Loading

github-actions bot commented May 2, 2024

kvserver: v22.1.5: inverted range (issue #32149) #97824

kvserver: v22.1.5: inverted range (issue #32149) #97824

Comments

cockroach-teamcity commented Mar 1, 2023 • edited by cockroach-jira-scripts Loading

github-actions bot commented May 2, 2024

cockroach-teamcity commented Mar 1, 2023 •

edited by cockroach-jira-scripts

Loading