db: tune number of ingestedFlushable's in the flushable queue

[bananabrick]

Right now, we restrict the number of ingested flushable's here: https://github.com/cockroachdb/pebble/blob/master/ingest.go#L847. The restriction is quite conservative. If the MemtableStopWritesThreshold is 3, and we currently have one flushable in the flushable queue, then we'll allow at most 2 ingestions as flushable before the next flush takes place.

To determine an appropriate limit, we should run a Cockroach workload which performs ingestions, compute stats on how often a memtable overlap is encountered, and then weigh that against how much additional read amp we're willing to take on.

Jira issue: PEBBLE-201

[jbowens]

I wonder in practice how frequently we would be able to slot in more ingested sstables without increasing the MemtableStopWritesThreshold. Rephrased, when an ingest overlaps with a memtable, how often does it not overlap with the mutable memtable. We can only add an additional ingested flushable without adding a new memtable if the ingest doesn't overlap with the mutable memtable. Raising MemtableStopWritesThreshold comes with its own costs (eg, 64MB memory).

Related to #766.

[bananabrick]

Rephrased, when an ingest overlaps with a memtable, how often does it not overlap with the mutable memtable.

Right now, we rotate the mutable memtable no matter what. Only rotating it if it has data overlap with the ingested files might be a good idea.