rpc: permit gRPC stream window sizes up to 64 MB

[nvanbenschoten]

Informs #111238.
Informs #111241.

CockroachDB accepts a COCKROACH_RPC_INITIAL_WINDOW_SIZE environment variable that can be used to set the initial window size for gRPC streams. However, the setting could previously only be used to reduce the initial window size, not to increase it.

This commit fixes that, allowing the initial window size to be increased up to 64 MB. This is useful for multi-region clusters that want to push high volumes of writes over high-latency WAN links.

Release note (general change): Increased the maximum permitted value of the COCKROACH_RPC_INITIAL_WINDOW_SIZE environment variable to 64MB. When used in conjunction with a tuned OS-level maximum TCP window size, this can increase the throughput that Raft replication can sustain over high-latency network links.

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[knz]

Even with this change, I'm not sure how much traffic we can pump across regions until the OS-level TCP Window size is also tuned. See ref.

[knz]

This could perhaps also benefit from a release note.

[erikgrinaker]

Note that higher window sizes carry a risk of OOMs, since this also affects the maximum size of gRPC's per-stream buffers. This was the motivation for the separate rangefeed connection class with a significantly lower initial window size: when rangefeed consumers struggled to keep up, we saw very high memory usage as gRPC stream buffers filled up across thousands of ranges.

[nvanbenschoten]

I'm not sure how much traffic we can pump across regions until the OS-level TCP Window size is also tuned.

Agreed, this only allows the gRPC window size to keep up with the OS-level TCP window size, but that also need to be tuned. See the experimentation in #111241.

This could perhaps also benefit from a release note.

Done.

I've also opened to cockroachdb/docs#17924 to suggest in our docs that users increase this setting.

TFTRs!

bors r=knz,erikgrinaker

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 1f77f7f to blathers/backport-release-22.2-111255: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.

---

error creating merge commit from 1f77f7f to blathers/backport-release-23.1-111255: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}