Resume CHANGEFEED from last committed cursor timestamp

[GlennFawcett-doordash]

Is your feature request related to a problem? Please describe.

A CHANGEFEED can fail for multiple reasons internal to CRDB, the Kafka endpoint, etc... After the CHANGEFEED fails, you have to figure out exactly when it failed from the logs and restart from that point after converting to the cluster_logical_timestamp epoch format.

Describe the solution you'd like

I would like for CRDB to store the last committed timestamp and have some way to simply RESUME from that point. You can easily PAUSE and RESUME jobs, but if the job fails you must resubmit. If the changefeed was created as an object in the database instead of a job, we could do something like:

RESUME CHANGEFEED <mychangefeed>

And it would simple pickup from where it left off.

Epic CRDB-2397

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

I have CC'd a few people who may be able to assist you:

• @cockroachdb/cdc (found keywords: CHANGEFEED,Kafka)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[amruss]

Note: Bulk IO also considering something similar, we probably want to do something more generic like RESUME FAILED JOB _job_id_ (RESUME JOB FROM FAILURE, RESUME FAILED JOB, RESURECT JOB, etc.) and spawn a new job id - with the same parameters / settings

We would want to error out for jobs that aren't changefeed jobs right now

[amruss]

After talking with the team, we will likely want to do this instead

[miretskiy]

@amruss this can probably be closed? We have #36887 issue and few others that we're working on.

[spiffyy99]

going to go ahead and close this, have a fix here: #68176