Merge pull request #8811 from dt/quiesce

RFC: quiesce ranges [postponed]

docs/RFCS/quiesce_ranges.md

@@ -0,0 +1,101 @@
+- Feature Name: Quiesce Ranges
+- Status: postponed
+- Start Date: 2016-06-24
+- Authors: David Taylor, Daniel Harrison, Tobias Schottdorf
+- RFC PR: #8811
+- Cockroach Issue: [#357](https://github.com/cockroachdb/cockroach/issues/357)
+
+
+# Summary
+Replicas of inactive ranges could potentially shutdown their raft group to save
+cpu, memory, and network traffic.
+
+# Motivation
+As clusters grow to potentially very large numbers of ranges, if access patterns
+are such that some of those ranges see no traffic, maintaining the raft groups
+and other state for them wastes substantial resources for little benefit.
+
+Coalesced raft heartbeats potentially mitigate _some_ of this waste, in terms of
+number of network requests, but after unpacking still require processing and
+thus do not eliminate the marginal cost of maintaining inactive ranges.
+
+Detecting inactive ranges and quiescing them -- shutting down their raft group
+and possibly flagging them to skip other bookkeeping and maintenance -- could
+offer substantial savings for clusters with very large numbers of ranges.
+
+Additionally, some operations may be able to use the fact a range is in a
+quiesced state. For example, some possible implementations of bulk ingestion of
+*new* ranges could be simplified (by being able to assume the raft state is
+frozen).
+
+# Detailed design
+Replicas of a quiescent range do not maintain a raft group.
+
+The most significant challenge is coordinating the transition from active to
+quiesced: a leader choosing to not maintain a quiesced raft group is easily
+mistaken for failing to maintain an active one.
+
+To enter the quiesced state, the leader sends a quiesce command to all
+followers. Upon receipt, followers stop expecting heartbeats for the remainder
+of the term. A raft group is quiesced only for one term.
+
+Once followers are no longer expecting heartbeats, the leader can stop sending
+them. If, however, it does this before a follower has received and processed the
+quiesce command, that follower will assume the leader is lost and call a new
+election, waking the group back up. To prevent this, the leader should continue
+sending heartbeats for some period after the command is issued -- specifically
+either until all followers acknowledge it or until the election timeout has
+elapsed (the election timeout is, by definition, the amount of time after which
+a node is assumed to be unreachable).
+
+If a follower is unreachable for more than the election timeout and and thus is
+unaware the group has quiesced, it will attempt to call an election and wake the
+range back up. This however would happen anyway -- since it was unreachable for
+more than the election timeout, it was going to call the election anyway, thus
+ending the term that quiesced.
+
+Once quiesced, any replica asked to propose a command should restart raft and
+trigger an election to be able to acquire the lease.
+
+A node that is partitioned or killed when a range quiesces will be unaware of
+that change when it returns, and may initiate a new election causing the range
+to return to actively maintaining raft until it quiesces again.
+
+# Drawbacks
+The largest drawback is reasoning about and maintaining the additional
+complexity that this introduces.
+
+As mentioned above, this only benefits clusters where access patterns are such
+that large numbers of ranges see no activity.
+
+This almost certainly needs to be implemented at least partially upstream in
+raft -- inspecting the follower state to determine when it is safe to stop
+heartbeating involves inspecting internal raft state. Upstream raft may not see
+as much value in incorporating this as the majority of users likely have only a
+single raft group, so they would see little benefit.
+
+Additionally, the first request to a range that has quiesced will see a
+non-trivial latency penalty, as it will need to start the raft group back up
+before it can proceed (though a in-place revival by the leader discussed below
+could mitigate this).
+
+Killed or partitioned nodes will spuriously wake groups that had quiesced in
+their absence, but as the range will still have not seen recent activity, it can
+re-quiesce shortly thereafter. Ccombined with the fact that this should be some
+what rare in normal operation, the expected cost of spurious wakeups should be
+small (and no worse than if the ranges didn't quiesce at all).
+
+# Alternatives
+Rather than quiescing the remainder of the current term, one could potentially
+call an election with an extra quiesced flag. A follower that votes for that
+would not expect heartbeats from that leader for that term. This seems to
+involve strictly more modification to raft, and it harder to reason about as it
+spans multiple terms and must thus consider multiple leader scenarios.
+
+As a follow-up optimization, the former leader of a quiesced range could
+optimistically attempt to revive the quiesced group *in-place*, without starting
+a new term, by resuming heartbeats and proposing commands. Any other node
+wishing to revive the group would still need to initiate an election for a new
+term, as would the former leader if it failed to get a majority of ACKs on the
+first command it proposed after waking. This can be done purely as an
+optimization at a later time.