all: fix various badness when resuscitating a node

[petermattis]

Prior to this PR, zerosum would experience a significant hiccup when
restarting a node:

chaos: starting node 1
     35s      9425       26203    789.3       15        0        0        6              2 6 5 5
     36s      9430       26319    115.4       42        0        0        6              2 6 5 5
     37s      9430       26319      0.0       42        0        0        6              2 6 5 5
     38s      9430       26319      0.0       42        0        0        6              2 6 5 5
     39s      9430       26319      0.0       43        0        0       -1              0 0 0 0
     40s      9430       26319      0.0       43        0        0        6              2 6 5 5
     41s      9430       26319      0.0       43        0        0        6              2 6 5 5
     42s      9430       26319      0.0       43        0        0        6              2 6 5 5
     43s      9430       26319      0.0       43        0        0        6              2 6 5 5
     44s      9430       26319      0.0       43        0        0        6              2 6 5 5
     45s      9444       26528    208.7       43        0        0        6              2 6 5 5
     46s      9480       27312    783.9       43        0        0        6              2 6 5 5

And now:

chaos: starting node 1
     35s      9399       24615    873.5        6        0        0        6              4 5 5 4
     36s      9399       24663     47.8        6        0        0        6              4 5 5 4
     37s      9399       24663      0.0        6        0        0        6              4 5 5 4
     38s      9425       25124    460.9        7        0        0       -1              0 0 0 0
     39s      9463       25890    765.9        7        0        0        6              4 5 5 4

---

This change is

[tamird]

I don't understand what the first commit is doing.

In the third commit: s/following/fouling/

---

Reviewed 1 of 1 files at r1, 2 of 2 files at r2, 3 of 3 files at r3.
Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks pending.

---

cmd/zerosum/cluster.go, line 182 [r3] (raw file):

  if err != nil {
      if log.V(1) {
          log.Infof(context.Background(), "scan failed: %s", err)

this looks unrelated, and i can't tell why it's being changed from the commit message

---

kv/dist_sender.go, line 205 [r2] (raw file):

                  var desc roachpb.RangeDescriptor
                  if err := value.GetProto(&desc); err != nil {
                      log.Errorf(context.Background(),

extract a local context?

also, this needs a test. ah, i see you filed an issue.

---

Comments from Reviewable

[petermattis]

The first commit is fixing a silliness Spencer introduced in RaftSender.SendAsync where it was retrieving a gRPC connection on every call instead of only when a queue is being created.

---

Review status: 2 of 6 files reviewed at latest revision, 2 unresolved discussions, some commit checks pending.

---

cmd/zerosum/cluster.go, line 182 [r3] (raw file):

Previously, tamird (Tamir Duberstein) wrote…

this looks unrelated, and i can't tell why it's being changed from the commit message

`internal/client` operations now fail-fast instead of retrying until they hit a deadline.

---

kv/dist_sender.go, line 205 [r2] (raw file):

Previously, tamird (Tamir Duberstein) wrote…

extract a local context?

also, this needs a test. ah, i see you filed an issue.

Sure on the local context.

I tried valiantly to test this, but the test infrastructure in dist_sender_test.go beat me down.

---

Comments from Reviewable

[tamird]

Right, but why does that matter?

---

Reviewed 4 of 4 files at r4, 3 of 3 files at r5.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks pending.

---

Comments from Reviewable

[petermattis]

It doesn't matter for correctness, just something I noticed. Why should we retrieve the connection when the active queue already has the connection? When the connection closes the associated streams will be closed closing the queue and the next time RaftSender.SendAsync is called we'll re-open the connection.

---

Review status: all files reviewed at latest revision, all discussions resolved, some commit checks pending.

---

Comments from Reviewable

[tamird]

Ah, I see. Could you expand the comment?

---

Review status: all files reviewed at latest revision, all discussions resolved, some commit checks pending.

---

Comments from Reviewable

[bdarnell]

---

Review status: all files reviewed at latest revision, all discussions resolved, some commit checks pending.

---

Comments from Reviewable

[petermattis]

Expanded the first commit message. I also reverted the usage of fail-fast by client.sender. The acceptance tests are currently relying on fail-fast=false.

---

Review status: 1 of 5 files reviewed at latest revision, all discussions resolved, some commit checks pending.

---

Comments from Reviewable

[tamird]

---

Reviewed 1 of 1 files at r1, 5 of 5 files at r6, 2 of 2 files at r7, 2 of 2 files at r8.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

---

Comments from Reviewable