release-23.1.0: base: reduce RaftMaxInflightBytes to 32 MB

[blathers-crl]

Backport 1/1 commits from #101437 on behalf of @erikgrinaker.

/cc @cockroachdb/release

---

This patch reduces the default RaftMaxInflightBytes from 256 MB to 32 MB, to reduce the out-of-memory incidence during bulk operations like RESTORE on clusters with overloaded disks.

RaftMaxInflightBytes specifies the maximum aggregate byte size of Raft log entries that a leader will send to a follower without hearing responses. As such, it also bounds the amount of replication data buffered in memory on the receiver. Individual messages can still exceed this limit (consider the default command size limit at 64 MB).

Normally, RaftMaxInflightMsgs * RaftMaxSizePerMsg will bound this at 4 MB (128 messages at 32 KB each). However, individual messages are allowed to exceed the 32 KB limit, typically large AddSSTable commands that can be around 10 MB each. To prevent followers running out of memory, we place an additional total byte limit of 32 MB, which is 8 times more than normal.

A survey of CC clusters over the past 30 days showed that, excluding a single outlier cluster, the total outstanding raft.rcvd.queued_bytes of any individual node never exceeded 500 MB, and was roughly 0 across all clusters for the majority of time.

Touches #71805.
Resolves #100341.
Resolves #100804.
Resolves #100983.
Resolves #101426.

Epic: none
Release note (ops change): the amount of replication traffic in flight from a single Raft leader to a follower has been reduced from 256 MB to 32 MB, in order to reduce the chance of running out of memory during bulk write operations. This can be controlled via the environment variable COCKROACH_RAFT_MAX_INFLIGHT_BYTES.

---

Release justification:

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[cockroach-teamcity]

This change is