db: adaptively throttle file deletions

[jbowens]

Observed during a scale test that strove to write 50 MiB/s/node in user data. The goroutine count was increasing monotonically. The growth was all goroutines responsible for deleting files, which were being paced at the default rate 128 MiB of delete files per second. The already in-flight #2641 will resolve the goroutine growth, but it does not resolve the issue of falling behind in file deletions.

We should not be setting a fixed deletion rate, and instead should smooth deletions based on the rate file deletions observed. See #2641 (review)

[RaduBerinde]

With the new design, we can use the number of queued jobs as a signal to increase the target byte rate. I'll think about a more specific proposal.

[RaduBerinde]

Or perhaps the total number of bytes to be deleted rather than the number of jobs

[RaduBerinde]

@jbowens here is a proposal.

Currently we disable pacing when free space falls below 16GB or when obsolete to live bytes ratio goes above 20%. Otherwise we pace at the target rate (default in CRDB is 128MB/s).

New proposal: we have pairs of low/high thresholds; on one side of the thresholds we pace at the target rate and on the other side we disable pacing. In-between we scale the pacing wait time per byte linearly. We also look at the absolute value for obsolete bytes - if we have very large stores, we would have fallen behind a lot when the obsolete to live ratio is large enough.

Proposed thresholds:

	Start increasing rate at	Disable pacing at
Free space	32GB	16GB
Obsolete to live bytes ratio	5%	20%
Obsolete bytes (new)	1GB	10GB

[jbowens]

With high-throughput stores, I wonder if we might see deletions be disruptive to foreground traffic due to obsolete bytes thresholds. A fully-utilized disk with 2 GiB/s bandwidth and 12 concurrent compactions can generate significant obsolete data very quickly. Maybe it's just a matter of choosing the right thresholds.

In-between we scale the pacing wait time per byte linearly.

Can you explain the scaling a bit more?

---

Do you think there is value in recording the rate at which deletions have been requested (eg, in the past 5 minutes) and adjusting our pacing rate to that rate, with the Options' rate setting a floor? I'm thinking it would keep deletions well-distributed over time, while still tolerating spikes.

[jbowens]

Linking this issue to #18, which is a generalization of the concept of pacing and prioritizing I/O.

[RaduBerinde]

Can you explain the scaling a bit more?

Let's say we have thresholds 16GB and 32GB for free space. Above 32GB free space, we limit at target rate. At 16GB, we don't throttle. At the midpoint (24GB) we throttle at double the rate. At 17.6GB we throttle at 10x the rate.

Do you think there is value in recording the rate at which deletions have been requested (eg, in the past 5 minutes) and adjusting our pacing rate to that rate, with the Options' rate setting a floor? I'm thinking it would keep deletions well-distributed over time, while still tolerating spikes.

Sounds smart, will think about it.

[RaduBerinde]

I like your suggestion more. I will leave the current checks as safety valves that should never fire in practice and add the history-based rate increase.