cdc: Lagging span checkpoint with end_time option might be broken

[miretskiy]

Lagging span checkpointing supposed to address a case when
changefeed executes rangefeeds against large table, and some of
the spans might be lagging behind the rest of the spans (due
to catchup scan throttling, or any other reasons).
In those cases, changefeed suppose to produce span based checkpoint
that is meant to ensure that catchup scan completes.

Consider the case where changefeed executes against large table, but
has both cursor and end_time options specified, with end_time in the past
(something like: I want to re-emit all events for the past hour, and then exit).
In this case, changefeed will start the catchup scan from cursor position, and
is supposed to terminate at end_time.

It appears (but needs to be verified) that in those cases, as soon. as the event
for the end_time is reached, all subsequent events are skipped -- which is fine.
What's not fine is that in this case we will never emit range based checkpoint, and that
will render lagging span checkpointing ineffective.

(Granted, this is a bit of a corner case, and was discovered via #108157).

Jira issue: CRDB-30482

[samiskin]

Would it not do so because the ranges are never sufficiently "lagging" in terms of their timestamp with respect to the frontier?

[miretskiy]

I think we start from non-zero frontier (cursor). We then never emit a single checkpoint until we reach end time.
And then we skip those checkpoints (because they have timestamp >= endTime).
As a result, Resolved event never sent to the change aggregator. We may very well be running this over 100k ranges -- and we would never checkpoint any progress (because we never see a single checkpoint).

[samiskin]

Ah that makes sense, I guess ideally we send resolved timestamps at the end time instead (at least once)