kvserver: don't repropose probes

[tbg]

We saw this likely contribute to OOMs during log application in
https://github.com/cockroachlabs/support/issues/2287.

Touches #98563.

Epic: CRDB-25503
Release note: None

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[tbg]

With the change as is, probes get an ambiguous result after 3s, which means a new probe can start after 3s instead of after 60s, so while avoiding quadratic build-up in the log, we get 20x more in-mem proposals and the log will grow at 20x the rate. Not sure this is a great trade-off.

We should probably repropose a probe exactly once, and give it an ambiguous result after 60s.

We should also commonly be able to avoid reproposing. If the local raft instance is the leader (as it mostly is), if it has unapplied log entries, we might as well wait for those to get applied first before reproposing. Also, commands that were proposed to a local leader and not dropped (i.e. if we handled ErrProposalDropped) are known to be in the log and could only be replaced if local leadership were lost, for which we get an event.

Long story short, what we're doing now is extremely inefficient. Log build-up during outages is totally avoidable.

In the same vein, we keep commands in the command map (r.mu.proposals) even after the callers have given up. It is true that we need to keep the latches in place until we have proven that the command can no longer apply. But here too we could be a lot more effective if, instead of reproposing the exact command multiple times (and doing so for possibly many in-flight commands), we'd instead send a single probe only. This would cause more spurious failures in the case in which the commands actually never made it into the log: we need to be much better with our tracking of such things to avoid flakes during the early life of ranges (while elections are happening, etc).

But this all can be done, if we are in the mood for a larger rework.

[tbg]

#105896 goes the other way and removes the timeout on probes to avoid the quadratic build-up. We can still discuss optimizing our reproposal behavior, but it's independent of probes at this point.