storage: deflake TestRefreshPendingCommands

[petermattis]

Node liveness heartbeats were causing a range lease to be reacquired
when the test was waiting for them to be drained. Due to the manual
clock, the draining would never properly complete.

Fixes #11771

---

This change is 

[tamird]

---

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

---

Comments from Reviewable

[petermattis]

Ha, I ran stress for 10m, then sent this PR and then the stress failed. Gah. There is something else that is also requesting the lease again.

[petermattis]

Second attempt. I'm trying to spur the inevitable failure by proclaiming that the flakiness is really fixed this time. PTAL.

[petermattis]

Making progress:

~ make stress TAGS=deadlock PKG=./storage TESTS=TestRefreshPendingCommands STRESSFLAGS="-maxfails 1 -stderr -p 16"
...
    --- FAIL: TestRefreshPendingCommands/#00 (2.34s)
    	client_raft_test.go:1000: range 1: replica {3 3 3} not lease holder; current lease is repl={3 3 3} start=0.900000123,17 exp=2.700000125,42 pro=1.800000125,43
FAIL


ERROR: exit status 1

7310 runs completed, 1 failures, over 20m31s

Before this PR the above stress invocation would fail within a minute or two.

@spencerkimball, @andreimatei How could the above error occur?

[petermattis]

Consulted with @andreimatei in person and he doesn't know how this error is being generated. I've added more logging and will reproduce.

[petermattis]

PS We saw this error before. We eventually attributed the error to strangeness in multiTestContext and the use of a manual clock. We're not doing any explicit lease transfers in TestRefreshPendingCommands, but perhaps the replicate queue is.

[petermattis]

Merging as this removes one source of flakiness from this test. I'll tackle the next bit of flakiness in a separate PR.