storage: avoid goroutine leaks in raft transport

[tamird]

Previously each snapshot would spawn a goroutine that would live for
the duration of the outgoing request channel being non-empty; under
high load, that may be an indefinite period of time.

This shortens the lifetime of the spawned goroutines to the duration
of a single outgoing snapshot each.

Fixes #6007.

---

This change is

[bdarnell]

LGTM

---

Review status: 0 of 1 files reviewed at latest revision, 2 unresolved discussions.

---

storage/raft_transport.go, line 199 [r1] (raw file):
Add a comment explaining that we start two streams so that we can use one for snapshots without blocking regular traffic. (but we do want snapshots to queue up behind each other)

---

storage/raft_transport.go, line 220 [r1] (raw file):
We'll write two errors to errCh but we'll only ever read one of them. That should be fine given the channel buffering but it's a little surprising and probably deserves a comment.

---

Comments from Reviewable

[tamird]

Review status: 0 of 1 files reviewed at latest revision, 2 unresolved discussions.

---

storage/raft_transport.go, line 199 [r1] (raw file):
Done.

---

storage/raft_transport.go, line 220 [r1] (raw file):
Done.

---

Comments from Reviewable