rangefeed: measure impact of scheduler rangefeeds on system ranges

[aliher1911]

We need to measure impact of rangefeed throughput on timely delivery of rangefeed messages.

Regardless of overall combined rangefeed throughput, timely delivery of messages on system rangefeeds is critical.
Overall, processing of individual messages should be quick, but we may have bursts of scheduled events for all ranges and those, should not affect timely message delivery.

Jira issue: CRDB-31388

Epic CRDB-26372

[blathers-crl]

cc @cockroachdb/replication

[aliher1911]

So far scheduler latency reaches 2-3s when cluster is under heavy write load. I used cdc benchmarks with 5 nodes 16vcpu machines. When writes are made in 10k batches across multiple ranges latency spikes during changefeed startups.

I did experiments with reducing number of events that each processor handles in one go. First I did some measurements by attaching histograms to processor to record batch sizes and it shows that it can handle between 64 and 128 events. Since we support postponing part of the work for next iteration, I tried limiting batch to 32 events but it showed marginal improvements.

I think we should proceed with priority shard approach. That will give us better guarantees that low throughtput ranges are unaffected by high throughput within processor domain at least.

[erikgrinaker]

I believe we did see multi-second scheduling latencies during startup (specifically due to catchup scan iterator construction). While this is being moved off of the scheduler goroutine in #111045, it still seems useful to prioritize system ranges, so we're merging that in #110810. We also have scheduling latency metrics, both for the normal and system shard, in #110458, so we can continue to investigate this.