engine: Limit size of merged rocksdb batches

[bdarnell]

When many threads are writing to rocksdb at once, a syncer goroutine collects all their write batches to write at once, minimizing the number of distinct disk writes. However, this can result in very large rocksdb WAL entries, and processing a rocksdb WAL entry during recovery appears to require substantially more memory than writing it the first time. A crash while a very large entry is in the uncompacted portion of the WAL can render the node unable to restart. Empirically, we've seen a node with 8GB of ram write a 1.6GB entry to the log and be unable to play it back on restart.

The syncer thread should monitor the size of the batches it builds and limit them to a reasonable size.

cc @petermattis

[petermattis]

I'm a bit surprised this actually occurred in practice. I'd really love to see this super-large batch and verify it contains keys from multiple different ranges. Failing that, I'll probably make an attempt at reproducing the problem first.

[bdarnell]

Yeah, this is speculation on my part since I don't know anything about what's actually in this batch. It's definitely true that there's no limit there now so it could produce a batch of unbounded size. It doesn't sound too far-fetched to me that all the node restarts could provide a forcing function that would result in more simultaneous disk writes that would hit this batching mechanism.

[petermattis]

I'm pretty sure RocksDB doesn't have a limit on the size of batches that it will merge either. Let me check that.

[petermattis]

I was wrong, RocksDB does have a limit on the size of batches it groups together. I'll add something similar.

  // Allow the group to grow up to a maximum size, but if the
  // original write is small, limit the growth so we do not slow
  // down the small write too much.
  size_t max_size = 1 << 20;
  if (size <= (128 << 10)) {
    max_size = size + (128 << 10);
  }