storage: transfer leases and leadership more thoroughly on graceful shutdown

[tbg]

When we drain a server, we spent a total of at most five seconds (per store) to move leases and Raft leaders to other nodes before we continue shutting down:

cockroach/pkg/storage/store.go

Lines 969 to 991 in f702e9d

	// The maximum amount of time waited for leadership shedding before commencing
	// to drain a store.
	const raftLeadershipTransferWait = 5 * time.Second
	// SetDraining (when called with 'true') causes incoming lease transfers to be
	// rejected, prevents all of the Store's Replicas from acquiring or extending
	// range leases, and attempts to transfer away any leases owned.
	// When called with 'false', returns to the normal mode of operation.
	func (s *Store) SetDraining(drain bool) {
	s.draining.Store(drain)
	if !drain {
	newStoreReplicaVisitor(s).Visit(func(r *Replica) bool {
	r.mu.Lock()
	r.mu.draining = false
	r.mu.Unlock()
	return true
	})
	return
	}
	var wg sync.WaitGroup
	ctx := logtags.AddTag(context.Background(), "drain", nil)

In large deployments, this may not be enough. We need to improve this logic; ideally it can determine with good enough accuracy when to give up (for example, a single surviving node in a three node deployment has no chance at transferring leases away), or we can justify trying "forever" (i.e. until the operator issues a hard shutdown). Ideally we don't need to introduce another knob.

[knz]

fixed by #45149