release-23.2: rangefeed: fix premature checkpoint due to intent resolution race

[erikgrinaker]

Only the 2 last commits should be reviewed here, the earlier commits in this PR are from the following prerequisite backports:

• release-23.2: batcheval: add BarrierRequest.WithLeaseAppliedIndex #118407
• release-23.2: batcheval: add PushTxnResponse.AmbiguousAbort #118409
• release-23.2: rangefeed: cancel async tasks on processor stop #118411

This backport includes new RPC protocol additions that are necessary to fix the bug. The new behavior (applying a Barrier command to wait for a Raft pipeline flush) is controlled by a cluster setting kv.rangefeed.push_txns.barrier.enabled, which defaults to true. According to the backport policy, such changes must be opt-in and disabled by default. However, making a bug fix opt-in appears questionable, especially one as serious as this one. I consider the protocol changes to be low-risk high-reward, and recommend we backport this behavior enabled by default.

@rharding6373 Assigning you as secondary TL reviewer, since you're the upcoming CDC TL. Let me know if you'd like me to route this elsewhere.

---

Backport 2/2 commits from #117612. Also pull in parts of #118469 and #118265.

Release justification: fixes a bug which could cause changefeeds to omit events in some scenarios.

/cc @cockroachdb/release

---

It was possible for rangefeeds to emit a premature checkpoint, before all writes below its timestamp had been emitted. This in turn would cause changefeeds to not emit these write events at all.

This could happen because PushTxn may return a false ABORTED status for a transaction that has in fact been committed, if the transaction record has been GCed (after resolving all intents). The timestamp cache does not retain sufficient information to disambiguate a committed transaction from an aborted one in this case, so it pessimistically
assumes an abort (see Replica.CanCreateTxnRecord and batcheval.SynthesizeTxnFromMeta).

However, the rangefeed txn pusher trusted this ABORTED status, ignoring the pending txn intents and allowing the resolved timestamp to advance past them before emitting the committed intents. This can lead to the following scenario:

• A rangefeed is running on a lagging follower.
• A txn writes an intent, which is replicated to the follower.
• The closed timestamp advances past the intent.
• The txn commits and resolves the intent at the original write timestamp, then GCs its txn record. This is not yet applied on the follower.
• The rangefeed pushes the txn to advance its resolved timestamp.
• The txn is GCed, so the push returns ABORTED (it can't know whether the txn was committed or aborted after its record is GCed).
• The rangefeed disregards the "aborted" txn and advances the resolved timestamp, emitting a checkpoint.
• The follower applies the resolved intent and emits an event below the checkpoint, violating the checkpoint guarantee.
• The changefeed sees an event below its frontier and drops it, never emitting these events at all.

This patch fixes the bug by submitting a barrier command to the leaseholder which waits for all past and ongoing writes (including intent resolution) to complete and apply, and then waits for the local replica to apply the barrier as well. This ensures any committed intent resolution will be applied and emitted before the transaction is removed from resolved timestamp tracking.

Resolves #104309.
Epic: none
Release note (bug fix): fixed a bug where a changefeed could omit events in rare cases, logging the error "cdc ux violation: detected timestamp ... that is less or equal to the local frontier". This can happen if a rangefeed runs on a follower replica that lags significantly behind the leaseholder, a transaction commits and removes its transaction record before its intent resolution is applied on the follower, the follower's closed timestamp has advanced past the transaction commit timestamp, and the rangefeed attempts to push the transaction to a new timestamp (at least 10 seconds after the transaction began). This may cause the rangefeed to prematurely emit a checkpoint before emitting writes at lower timestamps, which in turn may cause the changefeed to drop these events entirely, never emitting them.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[blathers-crl]

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[erikgrinaker]

Approved by CTO.

[nvanbenschoten]

We're now passing op by reference. Does that cause the argument to escape to the heap and result in a new heap allocation even when the assertion is not firing?

[erikgrinaker]

Great catch, especially considering I slipped this in here from #118265 -- thanks. You're probably right, I'll check this tomorrow and either shadow it in the branch or revert to passing by value.

[erikgrinaker]

Fixed, also in #118265.

[rharding6373]

Thanks for all the work finding, fixing, and backporting the fixes for this issue!

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @erikgrinaker)

[nvanbenschoten]

LGTM once the discussion from #118265 is resolved.

[yuzefovich]

Just a heads up that I think this currently is not on 23.2.1-rc branch.

[erikgrinaker]

Thanks for the headsup, will bump it over.