sentry: stopper.go:182: runtime.errorString: runtime error: invalid memory address or nil pointer dereference

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/961519443/?project=164528&referrer=webhooks_plugin

Panic message:

stopper.go:182: runtime.errorString: runtime error: invalid memory address or nil pointer dereference

Stacktrace (expand for inline code snippets):

/usr/local/go/src/runtime/asm_amd64.s#L572-L574 in runtime.call32
/usr/local/go/src/runtime/panic.go#L501-L503 in runtime.gopanic
/usr/local/go/src/runtime/panic.go#L62-L64 in runtime.panicmem
/usr/local/go/src/runtime/signal_unix.go#L387-L389 in runtime.sigpanic

cockroach/pkg/storage/store.go

Lines 3474 to 3476 in 1634c6b

	// count them.
	s.metrics.raftRcvdMessages[req.Message.Type].Inc(1)

in pkg/storage.(*Store).HandleRaftUncoalescedRequest

cockroach/pkg/storage/store.go

Lines 3440 to 3442 in 1634c6b

	if err := s.HandleRaftUncoalescedRequest(ctx, &beatReqs[i], respStream); err != nil {
	log.Errorf(ctx, "could not handle uncoalesced heartbeat %s", err)

in pkg/storage.(*Store).uncoalesceBeats

cockroach/pkg/storage/store.go

Lines 3455 to 3457 in 1634c6b

	}
	s.uncoalesceBeats(ctx, req.Heartbeats, req.FromReplica, req.ToReplica, raftpb.MsgHeartbeat, respStream)
	s.uncoalesceBeats(ctx, req.HeartbeatResps, req.FromReplica, req.ToReplica, raftpb.MsgHeartbeatResp, respStream)

in pkg/storage.(*Store).HandleRaftRequest

cockroach/pkg/storage/raft_transport.go

Lines 286 to 288 in 1634c6b

	return handler.HandleRaftRequest(ctx, req, respStream)
	}

in pkg/storage.(*RaftTransport).handleRaftRequest

cockroach/pkg/storage/raft_transport.go

Lines 341 to 343 in 1634c6b

	atomic.AddInt64(&stats.serverRecv, 1)
	if pErr := t.handleRaftRequest(ctx, req, stream); pErr != nil {
	atomic.AddInt64(&stats.serverSent, 1)

in pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1

cockroach/pkg/storage/raft_transport.go

Lines 349 to 351 in 1634c6b

	}
	}()
	})

in pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1

cockroach/pkg/util/stop/stopper.go

Lines 198 to 200 in 1634c6b

	defer s.stop.Done()
	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunWorker.func1

/usr/local/go/src/runtime/asm_amd64.s in runtime.call32 at line 573
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 502
/usr/local/go/src/runtime/panic.go in runtime.panicmem at line 63
/usr/local/go/src/runtime/signal_unix.go in runtime.sigpanic at line 388
pkg/storage/store.go in pkg/storage.(*Store).HandleRaftUncoalescedRequest at line 3475
pkg/storage/store.go in pkg/storage.(*Store).uncoalesceBeats at line 3441
pkg/storage/store.go in pkg/storage.(*Store).HandleRaftRequest at line 3456
pkg/storage/raft_transport.go in pkg/storage.(*RaftTransport).handleRaftRequest at line 287
pkg/storage/raft_transport.go in pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1 at line 342
pkg/storage/raft_transport.go in pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1 at line 350
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunWorker.func1 at line 199

Tag	Value
Cockroach Release	v2.1.5
Cockroach SHA:	1634c6b
Platform	linux amd64
Distribution	CCL
Environment	v2.1.5
Command	server
Go Version	go1.10.7
# of CPUs	12
# of Goroutines	302

[tbg]

cockroach/pkg/storage/store.go

Lines 3474 to 3476 in 1634c6b

	// count them.
	s.metrics.raftRcvdMessages[req.Message.Type].Inc(1)

I assume a message was received for which there wasn't a handler. The map is populated here

cockroach/pkg/storage/metrics.go

Lines 1379 to 1390 in 0b6580e

	sm.raftRcvdMessages[raftpb.MsgProp] = sm.RaftRcvdMsgProp
	sm.raftRcvdMessages[raftpb.MsgApp] = sm.RaftRcvdMsgApp
	sm.raftRcvdMessages[raftpb.MsgAppResp] = sm.RaftRcvdMsgAppResp
	sm.raftRcvdMessages[raftpb.MsgVote] = sm.RaftRcvdMsgVote
	sm.raftRcvdMessages[raftpb.MsgVoteResp] = sm.RaftRcvdMsgVoteResp
	sm.raftRcvdMessages[raftpb.MsgPreVote] = sm.RaftRcvdMsgPreVote
	sm.raftRcvdMessages[raftpb.MsgPreVoteResp] = sm.RaftRcvdMsgPreVoteResp
	sm.raftRcvdMessages[raftpb.MsgSnap] = sm.RaftRcvdMsgSnap
	sm.raftRcvdMessages[raftpb.MsgHeartbeat] = sm.RaftRcvdMsgHeartbeat
	sm.raftRcvdMessages[raftpb.MsgHeartbeatResp] = sm.RaftRcvdMsgHeartbeatResp
	sm.raftRcvdMessages[raftpb.MsgTransferLeader] = sm.RaftRcvdMsgTransferLeader
	sm.raftRcvdMessages[raftpb.MsgTimeoutNow] = sm.RaftRcvdMsgTimeoutNow

It looks the same on master, so not sure what that could've been (of course it could've been a bit flip, but that's too convenient an answer).

[tbg]

Closing since it apparently only happened once in three months on a single node. We'll need more data to track this down.