base: increase RaftTickInterval from 200 ms to 500 ms

Tick costs for unquiesced ranges can use a large amount of CPU on nodes
with many replicas. Increasing the tick interval from 200 ms to 500 ms
reduces this CPU cost by 60%. On a 3-node cluster with 50.000 unquiesced
ranges, this reduced the total CPU usage when idle from 54% to 32%.

All derived intervals and timeouts have been adjusted such that they
remain the same in wall time.

This increases the latency (from 200 to 500 ms) for tick-driven actions:

* Transfers of Raft leadership to leaseholders.
* Follower overload pausing.
* Updating the node liveness map.
* Updating the IO thresholds map.

Furthermore, because it reduces the resolution of the randomized Raft
election timeout interval from [10-20) ticks to [4-8) ticks, it
increases the chance of collisions and thus the chance of unsuccessful
elections.

Environment variables have been added to adjust this and any
tick-dependant values at runtime in case problems arise.

Epic: none
Release note (performance improvement): The Raft tick interval has been
increased from 200 ms to 500 ms in order to reduce per-replica CPU
costs, and can now be adjusted via `COCKROACH_RAFT_TICK_INTERVAL`.
Dependant parameters such as the Raft election timeout
(`COCKROACH_RAFT_ELECTION_TIMEOUT_TICKS`), reproposal timeout
(`COCKROACH_RAFT_REPROPOSAL_TIMEOUT_TICKS`), and heartbeat interval
(`COCKROACH_RAFT_HEARTBEAT_INTERVAL_TICKS`) have been adjusted such that
their wall-time value remains the same.

pkg/base/config.go

@@ -53,18 +53,10 @@ const (
 	defaultSQLAddr  = ":" + DefaultPort
 	defaultHTTPAddr = ":" + DefaultHTTPPort
 
-	// defaultRaftTickInterval is the default resolution of the Raft timer.
-	defaultRaftTickInterval = 200 * time.Millisecond
-
 	// NB: this can't easily become a variable as the UI hard-codes it to 10s.
 	// See https://github.com/cockroachdb/cockroach/issues/20310.
 	DefaultMetricsSampleInterval = 10 * time.Second
 
-	// defaultRaftHeartbeatIntervalTicks is the default value for
-	// RaftHeartbeatIntervalTicks, which determines the number of ticks between
-	// each heartbeat.
-	defaultRaftHeartbeatIntervalTicks = 5
-
 	// defaultRangeLeaseRenewalFraction specifies what fraction the range lease
 	// renewal duration should be of the range lease active time. For example,
 	// with a value of 0.2 and a lease duration of 10 seconds, leases would be
@@ -222,6 +214,16 @@ var (
 	// https://github.com/cockroachdb/cockroach/issues/93397.
 	defaultRPCHeartbeatTimeout = 3 * NetworkTimeout
 
+	// defaultRaftTickInterval is the default resolution of the Raft timer.
+	defaultRaftTickInterval = envutil.EnvOrDefaultDuration(
+		"COCKROACH_RAFT_TICK_INTERVAL", 500*time.Millisecond)
+
+	// defaultRaftHeartbeatIntervalTicks is the default value for
+	// RaftHeartbeatIntervalTicks, which determines the number of ticks between
+	// each heartbeat.
+	defaultRaftHeartbeatIntervalTicks = envutil.EnvOrDefaultInt(
+		"COCKROACH_RAFT_HEARTBEAT_INTERVAL_TICKS", 2)
+
 	// defaultRaftElectionTimeoutTicks specifies the minimum number of Raft ticks
 	// before holding an election. The actual election timeout per replica is
 	// multiplied by a random factor of 1-2, to avoid ties.
@@ -233,12 +235,12 @@ var (
 	// SystemClass, avoiding head-of-line blocking by general RPC traffic. The 1-2
 	// random factor provides an additional buffer.
 	defaultRaftElectionTimeoutTicks = envutil.EnvOrDefaultInt(
-		"COCKROACH_RAFT_ELECTION_TIMEOUT_TICKS", 10)
+		"COCKROACH_RAFT_ELECTION_TIMEOUT_TICKS", 4)
 
 	// defaultRaftReproposalTimeoutTicks is the number of ticks before reproposing
 	// a Raft command.
 	defaultRaftReproposalTimeoutTicks = envutil.EnvOrDefaultInt(
-		"COCKROACH_RAFT_REPROPOSAL_TIMEOUT_TICKS", 15)
+		"COCKROACH_RAFT_REPROPOSAL_TIMEOUT_TICKS", 6)
 
 	// defaultRaftLogTruncationThreshold specifies the upper bound that a single
 	// Range's Raft log can grow to before log truncations are triggered while at

pkg/base/testdata/raft_config

@@ -1,10 +1,10 @@
 echo
 ----
 (base.RaftConfig) {
- RaftTickInterval: (time.Duration) 200ms,
- RaftElectionTimeoutTicks: (int) 10,
- RaftReproposalTimeoutTicks: (int) 15,
- RaftHeartbeatIntervalTicks: (int) 5,
+ RaftTickInterval: (time.Duration) 500ms,
+ RaftElectionTimeoutTicks: (int) 4,
+ RaftReproposalTimeoutTicks: (int) 6,
+ RaftHeartbeatIntervalTicks: (int) 2,
  RangeLeaseDuration: (time.Duration) 6s,
  RangeLeaseRenewalFraction: (float64) 0.5,
  RaftLogTruncationThreshold: (int64) 16777216,