Skip to content

Commit

Permalink
storage: expose quota release queue and base index
Browse files Browse the repository at this point in the history
This exposes the information on `_status/range/N`.

I've looked at a few ranges locally and I'm already noticing that
weirdly often the last index will be significantly ahead of the
quota release base index (though the release queue is always empty).

This might be something that only happens on single-node instances (the
release mechanism is based on Ready handling, which is different in
single instances because nothing ever needs to get acked externally),
but it's unexpected to me.

Motivated by cockroachdb#39022, where we see a proposal quota leak: quota is queued
up for release, but the base index is way ahead of the last index, so it
will never be released.

Release note: None
  • Loading branch information
tbg committed Jul 26, 2019
1 parent 28bb1e9 commit d944e94
Show file tree
Hide file tree
Showing 3 changed files with 227 additions and 71 deletions.
5 changes: 5 additions & 0 deletions pkg/storage/replica.go
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,9 @@ type Replica struct {
// replicas have persisted the corresponding entry into their logs.
proposalQuota *quotaPool

// The base index is the index up to (including) which quota was already
// released. That is, the first element in quotaReleaseQueue below is
// released as the base index moves up by one, etc.
proposalQuotaBaseIndex uint64

// Once the leader observes a proposal come 'out of Raft', we add the
Expand Down Expand Up @@ -908,6 +911,8 @@ func (r *Replica) State() storagepb.RangeInfo {
ri.NumDropped = uint64(r.mu.droppedMessages)
if r.mu.proposalQuota != nil {
ri.ApproximateProposalQuota = r.mu.proposalQuota.approximateQuota()
ri.ProposalQuotaBaseIndex = int64(r.mu.proposalQuotaBaseIndex)
ri.ProposalQuotaReleaseQueue = append([]int64(nil), r.mu.quotaReleaseQueue...)
}
ri.RangeMaxBytes = *r.mu.zone.RangeMaxBytes
if desc := ri.ReplicaState.Desc; desc != nil {
Expand Down
Loading

0 comments on commit d944e94

Please sign in to comment.