kvserver: prevent recursive Replica.mu.RLock

[pav-kv]

Previously, we were passing an RLocked Replica to addToReplicasByKeyLocked.
That method internally visits the descriptors in replicasByKey, which calls
Desc() that may cause a recursive call to RLock if the added replica intersects
itself. This is only possible in tests which try to reinsert the Replica to the
map; in prod code each Replica is inserted to replicasByKey only once.

This PR avoids the possibility of this deadlock by shifting the responsibility
of locking to the caller.

Fixes #96931

Release note: none
Epic: none

[cockroach-teamcity]

This change is 

[tbg]

Commit message would be helpful:

Previously, we were passing an RLocked *Replica to (*Store).addToReplicasByKeyLocked. That method internally visits the descriptors and called Desc(), which would constitute a recursive call to RLock (caught by the deadlock detector due to the risk of a deadlock in case of an interleaving exclusive lock attempt to the mutex). <explain here how it can happen that while inserting a replica we're visiting the same replica? Was this a split? Or just a test doing funky things?>

[pav-kv]

@tbg Could you take another look? The previous version of this PR was deadlock-full. I fixed the deadlock at the cost of some duplication: the caller now passes the replica's Desc() or descRLocked() depending on the locking state.

[tbg]

👍🏽
also 😬 at this locking in general

[pav-kv]

bors r=tbg

[pav-kv]

bors cancel

[craig]

Canceled.

[pav-kv]

bors r=tbg

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)