Skip to content

Commit

Permalink
Merge #76609
Browse files Browse the repository at this point in the history 
76609: kvserver: Add replica load metrics r=kvoli a=kvoli

Previously, replica stats was oriented for collecting statistics on
batch requests per second - QPS. To facilitate more general use of
additional replica load statistics, this patch introduces more generic
naming in addition to allowing easy opt out of locality specific
elements.

The max, min, count and sum are now recorded per time window as
aggregate, running statistic calculated at write time in constant time.

Previously, writes-per-second representing the count of raft key
mutations applied and queries-per-second, representing the count of
batch requests processed by a replica were tracked on the replica level
by the leaseholder. This patch introduces: requests-per-second, tracking
the number of requests received; reads-per-second, tracking the number
of keys read; write-bytes-per-second and read-bytes-per-second, tracking
the number of bytes that have been read and written to this range per
second.

Only reads-per-second and requests-per-second are exposed in aggregation
over a store as a metric, which is in addition to queries-per-second and
writes-per-second, being currently exported.


Previously, HotRanges would report the replica descriptor and queries
per second (QPS) of the hottest ranges per store; ranked by QPS. This
patch introduces the additional range load information: requests per
second `RQPS`, writes per second `WPS`, reads per second `RPS`, write
bytes per second `WBPS` and read bytes per second `RBPS`. These
additional load metrics are useful for hot range inspection, as they
provide a more granular view of range load, than QPS alone.


resolves #72053
resolves #77280



Jira issue: CRDB-14773

Co-authored-by: Austen McClernon <[email protected]>
  • Loading branch information
@kvoli
craig[bot] and kvoli committed Apr 15, 2022
2 parents b55c877 + 790c95a commit 0d07f9c
Show file tree
Hide file tree
Showing 26 changed files with 946 additions and 95 deletions.
37 changes: 29 additions & 8 deletions docs/generated/http/full.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1323,8 +1323,12 @@ only.

| Field | Type | Label | Description | Support status |
| ----- | ---- | ----- | ----------- | -------------- |
| queries_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Queries per second served by this range.<br><br>Note that queries per second will only be known by the leaseholder. All other replicas will report it as 0. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Writes per second served by this range. | [reserved](#support-status) |
| queries_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Queries per second (batch requests) served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Writes per second served is the number of keys written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| requests_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Requests per second is the number of requests served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| reads_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Reads per second served is the number of keys read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| write_bytes_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Writes (bytes) per second is the number of bytes written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| read_bytes_per_second | [double](#cockroach.server.serverpb.RaftDebugResponse-double) | | Reads (bytes) per second is the number of bytes read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |



Expand Down Expand Up @@ -1564,8 +1568,12 @@ only.

| Field | Type | Label | Description | Support status |
| ----- | ---- | ----- | ----------- | -------------- |
| queries_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Queries per second served by this range.<br><br>Note that queries per second will only be known by the leaseholder. All other replicas will report it as 0. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Writes per second served by this range. | [reserved](#support-status) |
| queries_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Queries per second (batch requests) served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Writes per second served is the number of keys written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| requests_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Requests per second is the number of requests served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| reads_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Reads per second served is the number of keys read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| write_bytes_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Writes (bytes) per second is the number of bytes written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| read_bytes_per_second | [double](#cockroach.server.serverpb.RangesResponse-double) | | Reads (bytes) per second is the number of bytes read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |



Expand Down Expand Up @@ -1769,8 +1777,12 @@ only.

| Field | Type | Label | Description | Support status |
| ----- | ---- | ----- | ----------- | -------------- |
| queries_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Queries per second served by this range.<br><br>Note that queries per second will only be known by the leaseholder. All other replicas will report it as 0. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Writes per second served by this range. | [reserved](#support-status) |
| queries_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Queries per second (batch requests) served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Writes per second served is the number of keys written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| requests_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Requests per second is the number of requests served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| reads_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Reads per second served is the number of keys read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| write_bytes_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Writes (bytes) per second is the number of bytes written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| read_bytes_per_second | [double](#cockroach.server.serverpb.TenantRangesResponse-double) | | Reads (bytes) per second is the number of bytes read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |



Expand Down Expand Up @@ -3447,6 +3459,11 @@ target node(s) selected in a HotRangesRequest.
| desc | [cockroach.roachpb.RangeDescriptor](#cockroach.server.serverpb.HotRangesResponse-cockroach.roachpb.RangeDescriptor) | | Desc is the descriptor of the range for which the report was produced.<br><br>TODO(knz): This field should be removed. See: https://github.com/cockroachdb/cockroach/issues/53212 | [reserved](#support-status) |
| queries_per_second | [double](#cockroach.server.serverpb.HotRangesResponse-double) | | QueriesPerSecond is the recent number of queries per second on this range. | [alpha](#support-status) |
| leaseholder_node_id | [int32](#cockroach.server.serverpb.HotRangesResponse-int32) | | LeaseholderNodeID indicates the Node ID that is the current leaseholder for the given range. | [reserved](#support-status) |
| requests_per_second | [double](#cockroach.server.serverpb.HotRangesResponse-double) | | Requests per second is the recent number of requests received per second on this range. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.HotRangesResponse-double) | | Writes per second is the recent number of keys written per second on this range. | [reserved](#support-status) |
| reads_per_second | [double](#cockroach.server.serverpb.HotRangesResponse-double) | | Reads per second is the recent number of keys read per second on this range. | [reserved](#support-status) |
| write_bytes_per_second | [double](#cockroach.server.serverpb.HotRangesResponse-double) | | Write bytes per second is the recent number of bytes written per second on this range. | [reserved](#support-status) |
| read_bytes_per_second | [double](#cockroach.server.serverpb.HotRangesResponse-double) | | Read bytes per second is the recent number of bytes read per second on this range. | [reserved](#support-status) |



Expand Down Expand Up @@ -3739,8 +3756,12 @@ only.

| Field | Type | Label | Description | Support status |
| ----- | ---- | ----- | ----------- | -------------- |
| queries_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Queries per second served by this range.<br><br>Note that queries per second will only be known by the leaseholder. All other replicas will report it as 0. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Writes per second served by this range. | [reserved](#support-status) |
| queries_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Queries per second (batch requests) served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| writes_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Writes per second served is the number of keys written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| requests_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Requests per second is the number of requests served by this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| reads_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Reads per second served is the number of keys read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| write_bytes_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Writes (bytes) per second is the number of bytes written to this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |
| read_bytes_per_second | [double](#cockroach.server.serverpb.RangeResponse-double) | | Reads (bytes) per second is the number of bytes read from this range per second, averaged over the last 30 minute period. | [reserved](#support-status) |



Expand Down
5 changes: 5 additions & 0 deletions docs/generated/http/hotranges-other.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,5 +63,10 @@ Support status: [alpha](#support-status)
| desc | [cockroach.roachpb.RangeDescriptor](#cockroach.roachpb.RangeDescriptor) | | Desc is the descriptor of the range for which the report was produced.<br><br>TODO(knz): This field should be removed. See: https://github.com/cockroachdb/cockroach/issues/53212 | [reserved](#support-status) |
| queries_per_second | [double](#double) | | QueriesPerSecond is the recent number of queries per second on this range. | [alpha](#support-status) |
| leaseholder_node_id | [int32](#int32) | | LeaseholderNodeID indicates the Node ID that is the current leaseholder for the given range. | [reserved](#support-status) |
| requests_per_second | [double](#double) | | Requests per second is the recent number of requests received per second on this range. | [reserved](#support-status) |
| writes_per_second | [double](#double) | | Writes per second is the recent number of keys written per second on this range. | [reserved](#support-status) |
| reads_per_second | [double](#double) | | Reads per second is the recent number of keys read per second on this range. | [reserved](#support-status) |
| write_bytes_per_second | [double](#double) | | Write bytes per second is the recent number of bytes written per second on this range. | [reserved](#support-status) |
| read_bytes_per_second | [double](#double) | | Read bytes per second is the recent number of bytes read per second on this range. | [reserved](#support-status) |


28 changes: 26 additions & 2 deletions docs/generated/swagger/spec.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1079,13 +1079,37 @@
"type": "object",
"properties": {
"queries_per_second": {
"description": "Queries per second served by this range.\n\nNote that queries per second will only be known by the leaseholder.\nAll other replicas will report it as 0.",
"description": "Queries per second (batch requests) served by this range per second,\naveraged over the last 30 minute period.",
"type": "number",
"format": "double",
"x-go-name": "QueriesPerSecond"
},
"read_bytes_per_second": {
"description": "Reads (bytes) per second is the number of bytes read from this range per\nsecond, averaged over the last 30 minute period.",
"type": "number",
"format": "double",
"x-go-name": "ReadBytesPerSecond"
},
"reads_per_second": {
"description": "Reads per second served is the number of keys read from this range per\nsecond, averaged over the last 30 minute period.",
"type": "number",
"format": "double",
"x-go-name": "ReadsPerSecond"
},
"requests_per_second": {
"description": "Requests per second is the number of requests served by this range per\nsecond, averaged over the last 30 minute period.",
"type": "number",
"format": "double",
"x-go-name": "RequestsPerSecond"
},
"write_bytes_per_second": {
"description": "Writes (bytes) per second is the number of bytes written to this range per\nsecond, averaged over the last 30 minute period.",
"type": "number",
"format": "double",
"x-go-name": "WriteBytesPerSecond"
},
"writes_per_second": {
"description": "Writes per second served by this range.",
"description": "Writes per second served is the number of keys written to this range per\nsecond, averaged over the last 30 minute period.",
"type": "number",
"format": "double",
"x-go-name": "WritesPerSecond"
Expand Down
1 change: 1 addition & 0 deletions pkg/kv/kvserver/BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ go_library(
"replica_gc_queue.go",
"replica_gossip.go",
"replica_init.go",
"replica_load.go",
"replica_metrics.go",
"replica_placeholder.go",
"replica_proposal.go",
Expand Down
8 changes: 4 additions & 4 deletions pkg/kv/kvserver/allocator.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -354,10 +354,10 @@ func rangeUsageInfoForRepl(repl *Replica) RangeUsageInfo {
info := RangeUsageInfo{
LogicalBytes: repl.GetMVCCStats().Total(),
}
if queriesPerSecond, dur := repl.leaseholderStats.avgQPS(); dur >= MinStatsDuration {
if queriesPerSecond, dur := repl.leaseholderStats.averageRatePerSecond(); dur >= MinStatsDuration {
info.QueriesPerSecond = queriesPerSecond
}
if writesPerSecond, dur := repl.writeStats.avgQPS(); dur >= MinStatsDuration {
if writesPerSecond, dur := repl.writeStats.averageRatePerSecond(); dur >= MinStatsDuration {
info.WritesPerSecond = writesPerSecond
}
return info
Expand Down Expand Up @@ -1737,7 +1737,7 @@ func (a *Allocator) TransferLeaseTarget(
return candidates[a.randGen.Intn(len(candidates))]

case qpsConvergence:
leaseReplQPS, _ := stats.avgQPS()
leaseReplQPS, _ := stats.averageRatePerSecond()
candidates := make([]roachpb.StoreID, 0, len(existing)-1)
for _, repl := range existing {
if repl.StoreID != leaseRepl.StoreID() {
Expand Down Expand Up @@ -1970,7 +1970,7 @@ func (a Allocator) shouldTransferLeaseForAccessLocality(
}
}

qpsStats, qpsStatsDur := stats.perLocalityDecayingQPS()
qpsStats, qpsStatsDur := stats.perLocalityDecayingRate()

// If we haven't yet accumulated enough data, avoid transferring for now,
// unless we've been explicitly asked otherwise. Do not fall back to the
Expand Down
22 changes: 19 additions & 3 deletions pkg/kv/kvserver/metrics.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,18 @@ var (
Measurement: "Storage",
Unit: metric.Unit_COUNT,
}
metaAverageRequestsPerSecond = metric.Metadata{
Name: "rebalancing.requestspersecond",
Help: "Average number of requests received recently per second.",
Measurement: "Requests/Sec",
Unit: metric.Unit_COUNT,
}
metaAverageReadsPerSecond = metric.Metadata{
Name: "rebalancing.readspersecond",
Help: "Average number of keys read recently per second.",
Measurement: "Keys/Sec",
Unit: metric.Unit_COUNT,
}

// Metric for tracking follower reads.
metaFollowerReadsCount = metric.Metadata{
Expand Down Expand Up @@ -1367,9 +1379,11 @@ type StoreMetrics struct {
Reserved *metric.Gauge

// Rebalancing metrics.
AverageQueriesPerSecond *metric.GaugeFloat64
AverageWritesPerSecond *metric.GaugeFloat64
L0SubLevelsHistogram *metric.Histogram
L0SubLevelsHistogram *metric.Histogram
AverageQueriesPerSecond *metric.GaugeFloat64
AverageWritesPerSecond *metric.GaugeFloat64
AverageReadsPerSecond *metric.GaugeFloat64
AverageRequestsPerSecond *metric.GaugeFloat64

// Follower read metrics.
FollowerReadsCount *metric.Counter
Expand Down Expand Up @@ -1822,6 +1836,8 @@ func newStoreMetrics(histogramWindow time.Duration) *StoreMetrics {
l0SublevelMaxSampled,
1, /* sig figures (integer) */
),
AverageRequestsPerSecond: metric.NewGaugeFloat64(metaAverageRequestsPerSecond),
AverageReadsPerSecond: metric.NewGaugeFloat64(metaAverageReadsPerSecond),

// Follower reads metrics.
FollowerReadsCount: metric.NewCounter(metaFollowerReadsCount),
Expand Down
3 changes: 3 additions & 0 deletions pkg/kv/kvserver/replica.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,9 @@ type Replica struct {
// [1]: https://github.com/cockroachdb/cockroach/pull/16664
writeStats *replicaStats

// loadStats tracks a sliding window of throughput on this replica.
loadStats *ReplicaLoad

// creatingReplica is set when a replica is created as uninitialized
// via a raft message.
creatingReplica *roachpb.ReplicaDescriptor
Expand Down
4 changes: 4 additions & 0 deletions pkg/kv/kvserver/replica_application_state_machine.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -656,6 +656,7 @@ func (b *replicaAppBatch) runPreApplyTriggersAfterStagingWriteBatch(
}
if added := res.Delta.KeyCount; added > 0 {
b.r.writeStats.recordCount(float64(added), 0)
b.r.loadStats.writeKeys.recordCount(float64(added), 0)
}
if res.AddSSTable.AtWriteTimestamp {
b.r.handleSSTableRaftMuLocked(
Expand Down Expand Up @@ -983,6 +984,9 @@ func (b *replicaAppBatch) ApplyToStateMachine(ctx context.Context) error {
// Record the write activity, passing a 0 nodeID because replica.writeStats
// intentionally doesn't track the origin of the writes.
b.r.writeStats.recordCount(float64(b.mutations), 0 /* nodeID */)
if b.r.loadStats != nil {
b.r.loadStats.writeKeys.recordCount(float64(b.mutations), 0)
}

now := timeutil.Now()
if needsSplitBySize && r.splitQueueThrottle.ShouldProcess(now) {
Expand Down
1 change: 1 addition & 0 deletions pkg/kv/kvserver/replica_init.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ func newUnloadedReplica(
}
if store.cfg.StorePool != nil {
r.leaseholderStats = newReplicaStats(store.Clock(), store.cfg.StorePool.getNodeLocalityString)
r.loadStats = newReplicaLoad(store.Clock(), store.cfg.StorePool.getNodeLocalityString)
}
// Pass nil for the localityOracle because we intentionally don't track the
// origin locality of write load.
Expand Down
Loading

0 comments on commit 0d07f9c

Please sign in to comment.