release-23.2: kv: split LockConflictError, revive WriteIntentError over wire

[blathers-crl]

Backport 1/1 commits from #113646 on behalf of @nvanbenschoten.

/cc @cockroachdb/release

---

Fixes #113271.

This commit resolves the backwards incompatibility introduced by 350dc60 when WriteIntentError was renamed to LockConflictError. This rename broke mixed-version compatibility, because error details in kvpb.Error are packaged into an errorspb.EncodedError, which internally uses a protobuf/types.Any. protobuf/types.Any encodes the error's name as a string, relying on the receiving node having a matching type in order to decode the error.

Without this, we saw the following logs on v23.1 nodes.

error while unmarshalling error: ‹any: message type "cockroach.kv.kvpb.LockConflictError" isn't linked

As a result, error handling for requests that used WaitPolicy_Error was broken.

This commit resolves the issue by re-introducing WriteIntentError over the wire, so that v23.1 and v23.2 nodes still use the same name to refer to the same error. It does so without reverting 350dc60 and losing the naming improvement in most of the code by splitting LockConflictError into its two distinct roles. LockConflictError remains in the kvserver to communicate locking conflicts between batch evaluation and concurrency handling. However, the smaller role of communicating locking conflicts to clients that use a WaitPolicy_Error, a lock timeout, or a maximum wait-queue length is split into a "new" error called WriteIntentError. Splitting these errors was a cleanup we wanted to do anyway, so this commit just does it now to fix the bug. The unfortunate naming of WriteIntentError is a battle that we can fight another day.

While this commit doesn't introduce any new tests, we have sufficient testing of the two uses of WriteIntentError for single-version clusters in the unit tests. For mixed-version clusters, we have the backup-restore/mixed-version roachtest, which caught the bug and exercises backup's use of WriteIntentError.

The remaining place where this broke mixed-version compatibility was SELECT FOR UPDATE NOWAIT. We should add mixed-version testing for all SELECT FOR UPDATE variants. In the meantime, I have manually verified that the following script works on a mixed-version cluster:

roachprod create nathan-113271 -n3
roachprod stage  nathan-113271 release v23.1.11
roachprod start  nathan-113271
roachprod stop   nathan-113271:2-3
roachprod put    nathan-113271:2-3 cockroach # with this commit
roachprod start  nathan-113271:2-3 --sequential=false

roachprod sql nathan-113271:1
roachprod sql nathan-113271:2

-- either shell
create table t(i int primary key);
insert into t values (1);
select lease_holder from [show ranges from table t with details];
alter table t scatter;

-- 23.2 shell
begin; select * from t for update;

-- 23.1 shell
begin; select * from t for update nowait;

-- if broken:
ERROR: conflicting locks on /Table/104/1/1/0 [reason=wait_policy]
-- if fixed:
ERROR: could not obtain lock on row (i)=(1) in t@t_pkey

-- same thing but in opposite direction, with 23.1 leaseholder and 23.2 gateway

Release note: None

---

Release justification: avoids unexpected errors during mixed version backups.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[cockroach-teamcity]

This change is