bulkio: Add onfailure option to restore

[miretskiy]

Add on_failure option to the restore command.
Intended usage:
RESTORE ... WITH onfailure='rollback': That's the default
RESTORE ... WITH onfailure='pause': If something happens, pause the job.

Why: Restores take long time (in general) to run. If something happen (transient error, running out of disk,
failure to mark table PUBLIC, etc), we currently rollback everything. This may be too aggressive and undesirable:
often times we have made significant progress in the restore, and simply retrying might get us to the final
success.

Epic CRDB-7909

Jira issue: CRDB-3623

[pbardea]

Thanks for filing!

I wonder if it would be useful to allow the user to control this flow for all jobs and make the change at the job level. I found #36887, which seems to imply that this would also be useful for CDC use cases.

[miretskiy]

We would definitely need to make job level changes to support this (in a reasonable way).
For example, if the job went into "paused" state (or maybe paused-because-of-error), we need to make sure
that we can cancel + rollback. We also need to make sure that when job framework invokes OnFailCancel, we can tell jobs
framework to not kill this job, but instead go to the "paused" state.

Regardless, the option itself needs to be implemented on the statement level (i.e. add it to import/restore/etc in sql.y)

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!