perf: block_writer performance degradation

[petermattis]

The recent RocksDB tunings (#7532 and #7562) along with the optimization to parallelize processing of Replica.handleRaftReady (#7531) have improved the state of performance, but degradation still occurs. The first graph shows performance prior to these changes for a 1 hour run of block_writer:

After the changes we see:

A definite improvement, yet there is a sustained performance dip at ~45 min. My initial suspicion was that we reached another level of sstables. At the end of this test we were at 5 levels of sstables. But we reached the 5th level 4min into the test. I've run this test a few times and the dip always occurs around the same time frame. A few minutes after that I see an increase in the compaction rate.

[petermattis]

Somewhere along the way this was fixed.

Note that the above graph was taken from a 1h block_writer run on my laptop while the original message was from block_writer running on a 32-core GCE machine (so don't compare the absolute numbers). Since originally filing this issue I have replicated the original problem on my laptop, so this isn't a laptop vs GCE difference.