Prevent very fast learner recovery rates from causing OOM

[mattschumpert]

We've seen that tuning learner recovery rate very high can cause an OOM. We've separately reduced the memory consumption required for recovery, but we should still defend against OOMing at all costs as admins will not make the connection that this is an unsafe configuration.

The tradeoff should be to deny the requested learner recovery rate with a log warning and/or in general leave a bit more buffer before allocating more for recovery.

[vshtokman]

Long term, this item should be addressed by item #3 in the cluster ops doc.

Let me check with the team whether it is possible to add a short-term guardrail here.

[mmaslankaprv]

I think this issue is based on incorrect assumption, the recovery rate should not lead to OOM as we have a separate semaphore controlling recovery memory usage.

[mattschumpert]

@mmaslankaprv How did this cause OOM before? Is this an overall memory management issue? The CS request is the same: be more responsible about memory usage even if the user sets it to infinity.

[mmaslankaprv]

it didn't cause an OOM before, the OOM was related with a different thing, yes it is an overall memory management issue.

[mattschumpert]

@mmaslankaprv I'll close this. Do we have a ticket tracking the root cause of OOM?

[piyushredpanda]

@mattschumpert : the root issue here was most likely fixed by #9484