kv: refactor CompareWithLocality to use enum

[wenyihu6]

Prior to this commit, CompareWithLocality returned two boolean values
indicating whether two localities were cross-region and cross-zone. However,
this required callers to perform additional cross-comparison of these boolean
values to make meaningful metrics updates.

To simplify this, this commit introduces a new enum type
LocalityComparisonType. It provides four locality comparison results: cross
region, same region cross zone, same region same zone, and undefined
(indicating error behavior). This refactoring allow the caller to directly use
the comparison result without additional operations.

In addition, this commit also updates the logic to classify activities between
different regions as cross-regional, regardless of the zone tiers’
configuration. Initially, cross-region but same-zone tiers activities were
flagged as misconfiguration. After some discussion, we have decided that
regions should be non-overlapping. Hence, same zone tiers from different
regions should still be considered as different zones.

In addition, this commit also includes some refactoring of function
parameters.

Note that this commit does not change any existing functionality.

Part of: #103983

Release note: None

[cockroach-teamcity]

This change is 

[kvoli]

last commit

Reviewed 3 of 8 files at r1, 3 of 6 files at r2, 9 of 9 files at r3, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @tbg and @wenyihu6)

---

pkg/kv/kvclient/kvcoord/dist_sender.go line 2588 at r3 (raw file):

	gatewayNodeDesc, err := ds.nodeDescs.GetNodeDescriptor(fromNodeID)
	if err != nil {
		log.VEventf(ctx, 2, "failed to perform look up for node descriptor %+v", err)

Why %+v and not %v?

[tbg]

Looking good! Only small changes.

This reviews both commits. There is a lot of churn between the two so that in my opinion, they don't make sense as standalone commits. A better sequence of commits is possible, but there are few benefits to introducing it now.

So my suggestion would be to squash the current two commits with an updated message (and PR description) and closing the old PR (that contains the first commit only).
Please add fix-up commits as new commits (only to be squashed immediately before merge, to save me another full review).

[wenyihu6]

Looking good! Only small changes.

This reviews both commits. There is a lot of churn between the two so that in my opinion, they don't make sense as standalone commits. A better sequence of commits is possible, but there are few benefits to introducing it now.

So my suggestion would be to squash the current two commits with an updated message (and PR description) and closing the old PR (that contains the first commit only). Please add fix-up commits as new commits (only to be squashed immediately before merge, to save me another full review).

Ack. Added a new commit "to be squashed commit" which fixes the comments.

[tbg]

Looking good! Now all that's left is to squash all commits into the first one 💪🏽

[tbg]

LGTM once squashed into the first commit.

[tbg]

LGTM after squash, on the assumption that the first commit is identical with what I reviewed in the other PR.

[wenyihu6]

Thank you so much for the review, Tobi and Austen! (the last push is for a comment from a separate PR and has been approved.)

bors r=tbg,kvoli

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)