ui: surface PausedReplicas in DB Console

[erikgrinaker]

In #83851, we added a concept of "paused replicas". If a Raft follower's Pebble store is overloaded, then the leaseholder will temporarily pause replication to that follower until Pebble recovers (assuming we can still maintain a quorum).

The number of currently paused replicas is surfaced via the status server's range info as RangeInfo.paused_replicas:

cockroach/pkg/kv/kvserver/kvserverpb/state.proto

Line 187 in 43a37d5

repeated int32 paused_replicas = 21 [(gogoproto.casttype) = "github.com/cockroachdb/cockroach/pkg/roachpb.ReplicaID"];

And via the metric admission.raft.paused_replicas:

cockroach/pkg/ts/catalog/chart_catalog.go

Lines 567 to 572 in 43a37d5

	{
	Title: "Paused Followers",
	Metrics: []string{
	"admission.raft.paused_replicas",
	},
	},

We should surface this in the DB Console, e.g.:

• Display it in the range report.

• Display it in the problem ranges report, which requires extending the ProblemRanges API call with a new response field PausedFollowers and populating it.
  

  cockroach/pkg/server/serverpb/status.proto

  Line 1176 in 8b07764

  message ProblemRangesResponse {

  
  

  cockroach/pkg/server/problem_ranges.go

  Line 93 in 0ffc720

  for _, info := range resp.resp.Ranges {

• Display it in the replication metrics, either on an existing chart or a new one.

• Other places?

Jira issue: CRDB-17690

[erikgrinaker]

CC @cockroachdb/replication.

[tbg]

Just wanted to point out that the paused follower metric is populated on the leaseholders. So if s3 is overloaded, we expect other stores to register nonzero values on the paused follower metric, since they are doing the pausing.

We could add a metric that basically just indicates whether a store is pausable (reported by the store itself). This is basically just a metric that reports int(iothreshold.Score > configured_pausable_threshold_cluster_setting).

On a related note, I think something like this is also lacking for IO admission control (which triggers at iothreshold score 1).

[Santamaura]

What is the simplest way to simulate pausing replicas?

[tbg]

@Santamaura one reliably way is to rebase on top of #81516 and then invoke the roachtest like this:

export GCE_PROJECT=andrei-jepsen &&./pkg/cmd/roachtest/roachstress.sh -c 1 -u admission/follower-overload/presplit-with-leases -- --debug

The roachtest will "fail" after a few minutes but it will leave the cluster running with load against it. You can then navigate to the fourth node, port 3000, log in with admin:admin and then dashboards -> browse to open the one dashboard that exists. You will find a graph of pausable replicas there. As the "L0 threshold" graph crosses 0.8, you will see paused replicas. (This might take an hour or two, but once it's there it'll periodically be the case).