kvcoord: eliminate potential ambiguous errors by querying intents in distsender

[AlexTalks]

As described in #103817, when RPC failures occur on KV write operations parallel to a commit operation, retries of the operations cannot necessarily be considered idempotent. While #107658 introduced proper handling of these cases to raise AmbiguousResultErrors when necessary, it should be possible to improve handling in some cases with a longer-term solution.

While we cannot avoid all cases that would require us to propagate an AmbiguousResultError on a transaction that encounters an ambiguous error in a batch with a commit, we should be able to narrow down the cases in which we have to do so. To do this, we need to add handling in the DistSender to query intents for writes that had ambiguous errors and/or to attempt recovering the transaction entirely. In recovering the transaction, we would need to use a modified form of RecoverTxn that only explicitly commits or leaves the transaction state unaltered (if writes cannot be verified), rather than causing an abort.

• This approach allows us to explicitly commit and return the correct transaction state to the client whenever possible, minimizing the cases when propagating an ambiguous error is necessary.
• This approach should also be tested using data-driven test reproductions
• Lastly, the approach should be formally verified using TLA+.

Jira issue: CRDB-30432

[daniel-crlabs]

Affects this customer: https://cockroachdb.zendesk.com/agent/tickets/19344