rpc: Perform initial-heartbeat validation on GRPC reconnections

[bdarnell]

GRPC will transparently reconnect when a connection fails, but if the
next process to use that port is not a part of the same cluster, this
leads to confusing errors and potential data corruption. (this is most
common in tests, but it can also occur in other situations).

This change disables grpc's automatic reconnections so that in the
event of a failed connection, we go through our full dialing process
including an initial heartbeat that validates certain parameters.

Fixes #20537

Release note (bug fix): Implement additional safeguards against RPC
connections between nodes that belong to different clusters.

[cockroach-teamcity]

This change is 

[petermattis]

---

Review status: 0 of 4 files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

---

pkg/rpc/context.go, line 428 at r1 (raw file):

}

// onlyOnceDialer implements the grpc.WithDialer interface but only

Pedantically, this isn't implementing an interface, but is used to provide a custom dialer for grpc.WithDialer.

---

Comments from Reviewable

[nvanbenschoten]

Reviewed 1 of 3 files at r1.
Review status: 1 of 4 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

---

pkg/rpc/context.go, line 479 at r1 (raw file):

		redialChan: make(chan struct{}),
	}
	dialOpts = append(dialOpts, grpc.WithDialer(dialer.dial))

Do we need to set FailOnNonTempDialError as well?

---

Comments from Reviewable

[bdarnell]

Review status: 1 of 4 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

---

pkg/rpc/context.go, line 479 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Do we need to set FailOnNonTempDialError as well?

It doesn't really make any difference; it'll fail either way because grpc only has one address to try.

---

Comments from Reviewable