kvserver: add rangelog writes to a queue

[miraradeva]

We saw a scatter test with multiple rangelog timeouts in #104709. Bumping the timeout for the rangelog async task in #105898 helps but in the case of a scatter we may end up spawning too many async tasks that end up overwhelming the go scheduler. E.g. running a scatter on 100k ranges can result in 1 million rangelog events in a short amount of time.

Before #102813, writing to rangelog synchronously would act as a backpressure mechanism -- we wouldn't be able to go ahead with further conf changes until we'd flushed the write to the rangelog. Now we don't have that backpressure anymore, so we can pile up a huge backlog of writes.

One potential solution is to add all rangelog writes to a queue and then batch them.

Jira issue: CRDB-29264

[blathers-crl]

Hi @miraradeva, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[nvanbenschoten]

Before #102813, writing to rangelog synchronously would act as a backpressure mechanism -- we wouldn't be able to go ahead with further conf changes until we'd flushed the write to the rangelog. Now we don't have that backpressure anymore, so we can pile up a huge backlog of writes.

@miraradeva after #102813, we write to the rangelog non-atomically with respect to the replication change's txn, but we still log synchronously in the txn's commit trigger, right? Won't this still provide backpressure?

[miraradeva]

Just seeing the above comment now, and I agree there shouldn't be a need for a separate queue for writing to the rangelog. We haven't seen any issues with it in the mean time either.