kvserver,allocator: decouple allocator from storepool

[AlexTalks]

This change refactors the allocator to accept the store pool as an
argument, moving it closer to being stateless and purely functional.
This is done by moving to using the storepool through its primary
owner (the store configuration), or through references held by
users of the allocator. The refactoring enables the the allocator to
consider potential scenarios rather than those simply based on the
current liveness.

Part of #91570.

Release Note: None

[cockroach-teamcity]

This change is 

[kvoli]

I'm not a fan of the decorated methods ...WithStorePool(...). It adds to the complexity of the allocator API by exposing additional methods.

Would it be possible to instead pass it as an argument for every top level function instead?

This also relates to the approach taken on subsequent PRs - is it possible to use planOneChange in place of any forked methods.

Reviewed 2 of 2 files at r1, 2 of 2 files at r2, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @AlexTalks and @andrewbaptist)

---

-- commits line 4 at r1:
nit: drop the "to be"

[AlexTalks]

This is doable - let me know if you think it makes more sense to always pass in the StorePool on every allocator method (which may complicate things with callers), or if it would make sense to simply add an override parameter, like:

func (a *Allocator) ComputeAction(..., overrideStorePool storepool.AllocatorStorePool) {
  storePool := a.StorePool
  if overrideStorePool != nil {
     storePool = overrideStorePool
  }
  ...
}

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andrewbaptist)

[kvoli]

It would be good to add the storepool to the replicate queue similar to the store rebalancer.

Reviewed 1 of 9 files at r3, 23 of 23 files at r4, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @AlexTalks and @andrewbaptist)

---

pkg/kv/kvserver/replicate_queue.go line 614 at r4 (raw file):

) (shouldQueue bool, priority float64) {
	desc, conf := repl.DescAndSpanConfig()
	action, priority := rq.allocator.ComputeAction(ctx, rq.store.cfg.StorePool, conf, desc)

I'm thinking it would be better if the replicate queue were instantiated with a ref to the StorePool similar to the store rebalancer.

---

pkg/kv/kvserver/allocator/allocatorimpl/allocator.go line 512 at r4 (raw file):

// MakeAllocator creates a new allocator.
// The deterministic flag indicates that this allocator is intended to be used

What if it is not? This seems pretty fragile. There is a deterministic test which should catch this however, so it should be okay if any unlucky soul runs into it.

[AlexTalks]

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)