changefeedccl/kvevent: implement parallel event consumer

[jayshrivastava]

Implement parallel kv event consumers which parallelize the encoding and emitting of KV events to external sinks.

Since we consume (encode and emit) an event in a single thread on each node we can only process events so fast. Adding parallelism here will help ensure that we are not the bottleneck. That is encoding and emitting events should keep up with events coming in from rangefeeds.

WIP PR: https://github.com/jayshrivastava/cockroach/tree/nprocs-3

Some issues to be addressed are:

• Error reporting: Errors currently do not get reported back to the change aggregator because we encode/emit in the background.
  • Errors should not be out of order (ie. we should not emit an KV event if the preceeding on caused an error).
  • Errors should not violate ordering in terms of restarts/checkpoints. We should flush all events which are queued to be emitted before checkpointing.
  • Multiple tests expect synchronous errors. We should add a knob which makes the parallel consumer synchronous by default for these tests.
• Per-key ordering: This is currently not maintained by the PR. We should used a fixed # of workers and shard by key to fix this.
• Race conditions with ALTER CHANGEFEED. Some tests fail because we modify options when altering a changefeed while the options are being used by the parallel consumer in the background. This should be fixed by returning copies of data in certain spots instead of pointers.
• Race conditions with the buffered sink and changeAggregator.changedRowBuf

Jira issue: CRDB-18996

Epic CRDB-24463

[blathers-crl]

cc @cockroachdb/cdc