core: RPC re-connections are not validated

[solongordon]

This is a follow-up to #20163.

We currently validate new RPC connections before handing them out by waiting for a successful heartbeat. (See rpc.Connect().) However, if a node goes down and is restarted at the same address, gRPC happily reconnects and this validation is not performed. You can easily demonstrate this as follows:

# Bootstrap node1.
$ cockroach start --insecure --store=/tmp/node1

# Connect node2.
$ cockroach start --insecure --store=/tmp/node2 --port=26258 --http-port=8081 --join=localhost:26257

# Kill node1 and wipe its store.
$ rm -r /tmp/node1

# Restart node1.
$ cockroach start --insecure --store=/tmp/node1

Ideally node2 would revalidate its node1 connection at this point and refuse to connect due to a cluster ID mismatch. But instead the connection is reused, and node2 panics.

@petermattis suggests we could avoid this issue by using a custom gRPC dialer. See #20163 (comment).

cc @tschottdorf

[tbg]

We should also address the issue that the Ping heartbeat loop will practically never stop heartbeating (this was an unfortunate and likely unintended side effect of letting grpc handle heartbeats). I suggest the following:

• if no Ping has succeeded within the last $LARGEDURATION (hour?), decide that this isn't just TCP connection throttling holding us back; there is definitely something wrong and the connection should be closed.
• if a Ping goes through, and your validation fails, definitely close the connection as well.

The first would likely be obviated by inserting a Dialer that doesn't retry infinitely (I suppose our dialer would never retry and we'd just throw away the connection instead).

[tbg]

Assigning myself for bookkeeping, not actually planning to tackle this anytime soon.

[a-robinson]

Also for bookkeeping purposes, this is the same issue as #15898

[bdarnell]

I'm taking this over, on the theory that #20764 and #22320 are due to port reuse.