stability: panic on RangeLookup dispatched to correct range

[mberhault]

build sha: c382dd7
Beta cluster with photos + block_writer load. Restarted from scratch earlier today.

~1h after a restart, encountered the following:

panic: RangeLookup dispatched to correct range, but no matching RangeDescriptor was found: /Meta2/Table/55/1/8752590996753796817/"88e08dd7-d1b6-4aeb-ad46-1880b351e731"/6611/0

1: running [Created by grpc.(*Server).serveStreams.func1 @ server.go:324]
              panic.go:464            panic(#1, #18)
    storage   store.go:1557           (*Store).Send.func1(0xc833bd1258, 0xc833bd12c8, 0x14449b721ca69e59, 0, 0xc833bd12c0, #5)
              panic.go:426            panic(#1, #18)
    storage   replica_command.go:909  (*Replica).RangeLookup(#12, #19, #9, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   replica_command.go:129  (*Replica).executeCmd(#12, #20, #17, 0, 0, 0, #19, #9, 0, #22, ...)
    storage   replica.go:1784         (*Replica).executeBatch(#12, #20, #17, 0, 0, #19, #9, 0, #22, 0, ...)
    storage   replica.go:1071         (*Replica).addReadOnlyCmd(#12, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   replica.go:765          (*Replica).Send(#12, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   store.go:1639           (*Store).Send(#5, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   stores.go:177           (*Stores).Send(#6, #20, 0xc82bb06c30, 0, 0, #4, 0x4, 0x1, 0, 0, ...)
    server    node.go:729             (*Node).Batch.func2()
    stop      stopper.go:166          (*Stopper).RunTask(#8, 0xc833bd1870, 0)
    server    node.go:741             (*Node).Batch(#7, #20, #16, 0xc8271b2150, 0, 0, 0)
    roachpb   api.pb.go:1306          _Internal_Batch_Handler(#2, #7, #20, #16, 0xc833082a00, 0, 0, 0, 0)
    grpc      server.go:497           (*Server).processUnaryRPC(#11, #21, #14, #15, #10, #3, 0, 0, 0)
    grpc      server.go:646           (*Server).handleStream(#11, #21, #14, #15, 0)
    grpc      server.go:323           (*Server).serveStreams.func1.1(#13, #11, #21, #14, #15)

Full log (from ec2-52-91-234-29.compute-1.amazonaws.com):
node4.log.parse.txt

[tbg]

Could you save the data? Only that node is ok. Please don't restart (you
can after taking the dump, but it'll likely happen again).

On Tue, Apr 12, 2016 at 9:45 AM marc [email protected] wrote:

build sha: c382dd7
c382dd7
Beta cluster with photos + block_writer load. Restarted from scratch
earlier today.

~1h after a restart, encountered the following:

panic: RangeLookup dispatched to correct range, but no matching RangeDescriptor was found: /Meta2/Table/55/1/8752590996753796817/"88e08dd7-d1b6-4aeb-ad46-1880b351e731"/6611/0

1: running [Created by grpc.(_Server).serveStreams.func1 @ server.go:324]
panic.go:464 panic(#1, #18)
storage store.go:1557 (_Store).Send.func1(0xc833bd1258, 0xc833bd12c8, 0x14449b721ca69e59, 0, 0xc833bd12c0, #5)
panic.go:426 panic(#1, #18)
storage replica_command.go:909 (_Replica).RangeLookup(#12, #19, #9, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
storage replica_command.go:129 (_Replica).executeCmd(#12, #20, #17, 0, 0, 0, #19, #9, 0, #22, ...)
storage replica.go:1784 (_Replica).executeBatch(#12, #20, #17, 0, 0, #19, #9, 0, #22, 0, ...)
storage replica.go:1071 (_Replica).addReadOnlyCmd(#12, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
storage replica.go:765 (_Replica).Send(#12, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
storage store.go:1639 (_Store).Send(#5, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
storage stores.go:177 (_Stores).Send(#6, #20, 0xc82bb06c30, 0, 0, #4, 0x4, 0x1, 0, 0, ...)
server node.go:729 (_Node).Batch.func2()
stop stopper.go:166 (_Stopper).RunTask(#8, 0xc833bd1870, 0)
server node.go:741 (_Node).Batch(#7, #20, #16, 0xc8271b2150, 0, 0, 0)
roachpb api.pb.go:1306 _Internal_Batch_Handler(#2, #7, #20, #16, 0xc833082a00, 0, 0, 0, 0)
grpc server.go:497 (_Server).processUnaryRPC(#11, #21, #14, #15, #10, #3, 0, 0, 0)
grpc server.go:646 (_Server).handleStream(#11, #21, #14, #15, 0)
grpc server.go:323 (*Server).serveStreams.func1.1(#13, #11, #21, #14, #15)

Full log (from ec2-52-91-234-29.compute-1.amazonaws.com):
node4.log.parse.txt
https://github.com/cockroachdb/cockroach/files/215109/node4.log.parse.txt

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#6000

-- Tobias

[mberhault]

yeah, it's still down, but I'm backing up with suffix .6000 on ec2-52-91-234-29.compute-1.amazonaws.com

[tamird]

Is this data still available somewhere?

[tamird]

This looks like a meta2 gap? Closing as dupe of #9265.