kv: partition raft transport over high RTT links across multiple TCP connections

[nvanbenschoten]

Bandwidth-delay product principles dictate that the maximum throughput a system can push over a single network link is limited by the round-trip time ("delay") and the window size ("amount of unacknowledged bytes allowed"). Even high bandwidth links can have a relatively low effective throughput if they also have a high delay ("long fat networks"). For more details, see https://networklessons.com/cisco/ccnp-route/bandwidth-delay-product.

In multi-region deployments, such long fat networks (LFNs) are common. Raft is currently not configured to handle them as well as it could be, due to its use of a single TCP connection between nodes and the default maximum window sizes users typically deploy CockroachDB with. As a result, the Raft transport can be saturated by a sustained write load, leading to instability on both the ranges driving the load and other ranges that are not. Recall that the raft transport is shared across all ranges¹ on a node that talk to a common destination node.

Let's talk about window sizes first.

Connection Window Sizes

Over a high latency link, a single connection's throughput is limited by its window size (in addition to other factors, like packet loss). Modern Linux distributions typically default the maximum TCP window size to 4MB, like we can see in roachprod:

ubuntu@nathan-high-rtt-0001:~$ uname -a
Linux nathan-high-rtt-0001 5.4.0-1044-gcp #47-Ubuntu SMP Tue May 11 15:51:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@nathan-high-rtt-0001:~$ cat /proc/sys/net/ipv4/tcp_window_scaling
1
ubuntu@nathan-high-rtt-0001:~$ cat /proc/sys/net/ipv4/tcp_wmem
4096	16384	4194304
ubuntu@nathan-high-rtt-0001:~$ cat /proc/sys/net/ipv4/tcp_rmem
4096	131072	6291456

CockroachDB goes a step further and limits the per gRPC connection window size to 2MB:

cockroach/pkg/rpc/settings.go

Lines 90 to 91 in 967fe5e

	s.values.initialWindowSize = getWindowSize(ctx,
	"COCKROACH_RPC_INITIAL_WINDOW_SIZE", DefaultClass, defaultWindowSize*32)

#35161 tried to disable this constant window size for gRPC's new dynamic window resizing, which would let a gRPC connection scale up to the OS's limit. That PR was closed at the time.

By 2MB, 4MB, or some other value in the low MBs, these window sizes are all limiting over high-latency links. For example, a 2MB window size over a 100ms link limits throughput to 1s/100ms * 2MB = 20MB/s.

We could adjust these on a case-by-case basis (and should where appropriate), but this still involves CRDB and OS-level manual tuning.

Multiple Connections

Assuming we can't always increase the maximum TCP window size above the OS-level default, we should explore alternatives to increase the raft transport throughput between nodes.

One approach would be to partition the raft transport across multiple gRPC/TCP connections. The simplest approach here would be to partition the raft transport's connections based on range ID, as there is no need to maintain the ordering of raft messages across ranges. Even within a range, ordering is not assumed for correctness, but too much reordering can hurt performance.

Jira issue: CRDB-31830

Epic CRDB-32846

Footnotes

1. this is ignoring the system range connection class separation, which protects system ranges from instability when the default connection class's raft transport is saturated. ↩

[blathers-crl]

cc @cockroachdb/replication

[bdarnell]

Note that there are two levels of window size in play: there's the TCP window size, and then HTTP/2 (which underlies GRPC) has its own similar window size mechanism. COCKROACH_RPC_INITIAL_WINDOW_SIZE (and subsequent comments about dynamic sizing) refers to the HTTP/2 window size, not the TCP one. (there's also a third one, HTTP/2's connection-level window, but IIRC we set that one to effectively infinite)

Modern Linux distributions typically default the maximum TCP window size to 4MB, like we can see in roachprod:

The first google result for tcp_wmem recommends setting the maximum to 16MB or higher "especially for 10 Gigabit adapters". https://www.ibm.com/docs/en/linux-on-systems?topic=tuning-tcpip-ipv4-settings. This default is either outdated or overly conservative and we should probably recommend increasing it for multi-region CRDB deployments.

My sense is that we should be able to tune the kernel parameters and GRPC for higher throughput in this scenario. I'd consider spreading the traffic across multiple TCP connections to be a last resort if the only consideration is increasing throughput. (There may be other benefits of exploring this path, such as reducing the impact of dropped packets).

Longer-term, QUIC is nice for getting the kernel TCP implementation out of the loop completely. And we might consider sending raft log messages individually instead of as a single HTTP/2 or QUIC stream (we can reconstruct dropped messages from our local copy of the log, so we don't need to have that data duplicated in the kernel's tcp buffer and be limited by tcp_wmem)

[erikgrinaker]

I'm inclined to agree with Ben here: we should set this at the OS level first, and we should recommend 16 MB by default (I sort of thought everyone already did this).

I do think there might be some value in splitting out rangefeed and Raft traffic to separate connections though, separating them from other RPC traffic, since these can be fairly high-volume and RPC head-of-line blocking will directly impact foreground tail latencies. We could even consider splitting out a separate connection for Raft snapshots. But the last time I ran some basic benchmarks, this didn't appear to have any significant impact, at least not for the cases I was looking at.

There's likely also other factors at play here, e.g. RaftMaxInflightMsgs and RaftMaxInflightBytes, which affect the per-replica bandwidth-delay product.

Let's explore this for 24.1 once the 23.2 branch is cut, it should be straightforward (apart from coming up with representative benchmarks).

[erikgrinaker]

I opened #111262 to consider a separate Raft connection class, at least, and we have #108992 to enable the rangefeed connection class by default.

[nvanbenschoten]

I also opened cockroachdb/docs#17924 to track the docs improvement for making this recommendation.

[pav-kv]

Based on the discussion above:

• We should recommend tuning TPC parameters for multi-region deployments: deployment: recommend increasing TCP max window size for multi-region deployents docs#17924.
• For now, we separated raft and rangefeed traffic into dedicated connection classes: kvserver: separate Raft RPC connection class #111262, rangefeed: enable and improve rangefeed connection class #108992.