db: improve ingested flushable flushing logic

[bananabrick]

Presently in #2212, we split a flush into chunks so that ingested flushables have an updated view of the lsm while flushing.

This is unnecessary. If we have some queue of flushables, and the flushable with sstables at some position i is f_i, then it is possible to determine the target level for the files associated with f_i, if we've determined the target level of all flushables upto f_i-1. f_0 is the base case as it can use the current version to determine its target level.

This will prevent a potential slowdown of flushes when ingested flushables are present in the flushable queue.

Jira issue: PEBBLE-145

[jbowens]

@bananabrick can this slip to 23.2? I know you've got a lot on your plate for 23.1

[bananabrick]

@jbowens No this one can't. I want to do this before enabling flushable ingests.

[jbowens]

@bananabrick can you explain why it can't wait? My understanding is that this is purely an optimization and is not a regression over 22.2.

[bananabrick]

@jbowens Let's say we have some memtables m1, m2 in the queue, followed by ingested flushables i1, followed by memtable m3, followed by ingested flushable i2, followed by memtable m4. So, the queue is m1,m2,i1,m3,i2,m4. If we try and flush this right now, we're going to incur 4 manifest writes + syncs. In 22.2, we won't have i2, i2, and just m1,m2,m3,m4. To flush m1, m2 and m3, we'll incur exactly one manifest write + sync.

Maybe writing the memtable to disk as sstables and syncing those will dominate over a few manifests writes + syncs, and this is okay. Not a 100% sure.

[bananabrick]

Maybe I could just enable it and only do this issue if we see a regression in any benchmarks which perform pebble ingestions.

[jbowens]

So, the queue is m1,m2,i1,m3,i2,m4. If we try and flush this right now, we're going to incur 4 manifest writes + syncs. In 22.2, we won't have i2, i2, and just m1,m2,m3,m4.

In 22.2, wouldn't the same sequence of ingests result in:

1. flush of m1+m2
2. ingest of i1
3. flush of m3
4. ingest of i2
5. flush of m4

with corresponding manifests writes and syncs for each?