storage: applied(%d) is out of range [prevApplied(%d), committed(%d)]

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/974738775/?project=164528&referrer=webhooks_plugin

Panic message:

stopper.go:182: string

Stacktrace (expand for inline code snippets):

/usr/local/go/src/runtime/asm_amd64.s#L572-L574 in runtime.call32
/usr/local/go/src/runtime/panic.go#L501-L503 in runtime.gopanic

cockroach/pkg/storage/raft.go

Lines 105 to 107 in 4523f98

	log.ErrorfDepth(r.ctx, 1, format, v...)
	panic(fmt.Sprintf(format, v...))
	}

in pkg/storage.(*raftLogger).Panicf
https://github.com/cockroachdb/cockroach/blob/4523f9819d2dde9b054fc8faf1568e107f130986/vendor/go.etcd.io/etcd/raft/log.go#L211-L213 in vendor/go.etcd.io/etcd/raft.(*raftLog).appliedTo
https://github.com/cockroachdb/cockroach/blob/4523f9819d2dde9b054fc8faf1568e107f130986/vendor/go.etcd.io/etcd/raft/raft.go#L355-L357 in vendor/go.etcd.io/etcd/raft.newRaft
https://github.com/cockroachdb/cockroach/blob/4523f9819d2dde9b054fc8faf1568e107f130986/vendor/go.etcd.io/etcd/raft/rawnode.go#L78-L80 in vendor/go.etcd.io/etcd/raft.NewRawNode

cockroach/pkg/storage/replica.go

Lines 544 to 546 in 4523f98

	ctx := r.AnnotateCtx(context.TODO())
	raftGroup, err := raft.NewRawNode(newRaftConfig(
	raft.Storage((*replicaRaftStorage)(r)),

in pkg/storage.(*Replica).withRaftGroupLocked

cockroach/pkg/storage/replica.go

Lines 1955 to 1957 in 4523f98

	if err := r.withRaftGroupLocked(true, func(raftGroup *raft.RawNode) (bool, error) {
	return true, nil

in pkg/storage.(*Replica).maybeInitializeRaftGroup

cockroach/pkg/storage/replica.go

Lines 1996 to 1998 in 4523f98

	// If the internal Raft group is not initialized, create it and wake the leader.
	r.maybeInitializeRaftGroup(ctx)

in pkg/storage.(*Replica).sendWithRangeID

cockroach/pkg/storage/replica.go

Lines 1968 to 1970 in 4523f98

	) (*roachpb.BatchResponse, *roachpb.Error) {
	return r.sendWithRangeID(ctx, r.RangeID, ba)
	}

in pkg/storage.(*Replica).Send

cockroach/pkg/storage/replica_range_lease.go

Lines 316 to 318 in 4523f98

	ba.Add(leaseReq)
	_, pErr = p.repl.Send(ctx, ba)
	}

in pkg/storage.(*pendingLeaseRequest).requestLeaseAsync.func2

cockroach/pkg/util/stop/stopper.go

Lines 323 to 325 in 4523f98

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTask.func1

/usr/local/go/src/runtime/asm_amd64.s in runtime.call32 at line 573
/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 502
pkg/storage/raft.go in pkg/storage.(*raftLogger).Panicf at line 106
vendor/go.etcd.io/etcd/raft/log.go in vendor/go.etcd.io/etcd/raft.(*raftLog).appliedTo at line 212
vendor/go.etcd.io/etcd/raft/raft.go in vendor/go.etcd.io/etcd/raft.newRaft at line 356
vendor/go.etcd.io/etcd/raft/rawnode.go in vendor/go.etcd.io/etcd/raft.NewRawNode at line 79
pkg/storage/replica.go in pkg/storage.(*Replica).withRaftGroupLocked at line 545
pkg/storage/replica.go in pkg/storage.(*Replica).maybeInitializeRaftGroup at line 1956
pkg/storage/replica.go in pkg/storage.(*Replica).sendWithRangeID at line 1997
pkg/storage/replica.go in pkg/storage.(*Replica).Send at line 1969
pkg/storage/replica_range_lease.go in pkg/storage.(*pendingLeaseRequest).requestLeaseAsync.func2 at line 317
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTask.func1 at line 324

Tag	Value
Cockroach Release	v2.1.1
Cockroach SHA:	4523f98
Platform	linux amd64
Distribution	CCL
Environment	v2.1.1
Command	server
Go Version	go1.10.3
# of CPUs	4
# of Goroutines	152

[tbg]

applied(%d) is out of range [prevApplied(%d), committed(%d)]

[nvanbenschoten]

Merging into #41144, as this still looks possible as of v19.1.