jobs: infinite loop in v23.1.x when jobs run before cluster version is finalized

[dt]

Previously, in some narrow race conditions, a transaction may start at at a timestamp before a migration has run, meaning it would read a state where migration effects, such as creating tables, were not visible, however if migration concurrently executed and completed, any IsActive checks done during that txn might then return true, causing problems where a txn would expect a given table to exist, due to an IsActive check saying it had been created, but then not observe it when it read due its chosen timestamp. This has been previously described in #106764 and #109039.

In #108357 we wrapped the write to the job_info table with MaybeGenerateForcedRetryableError, which meant errors due to a table not existing were always considered retryable. This was done to mitigate this rare race condition, typically only observed in automated tests of upgrades, that interacted with the jobs tables heavily while performing upgrades and as such would fail due to jobs related code that expected the info table to exist (based on IsActive) but failed with errors stating it did not exist.

The change in #108357 however had a significant unintended consequence for executions of job info transaction that were not racing with a version upgrade, i.e. callers who were not executing under IsActive(V23_1) == true check. Several such examples of callers that are not conditional on V23_1 are the jobsprofiler.StorePlanDiagram calls in backup, restore, changefeeds and IMPORT.

When these StorePlanDiagram calls were added, since they were just there to capture nice-to-have debugging information, they were explicitly added as "fire-and-forget" calls, done on a separate goroutine and with any errors logged and discarded, with the intention that they could not negatively impact or affect the code to which they were added in any way. This meant they were able to be added without wrapping them in version gates that would ensure the table to which they'd try to log existed, as any errors would simply be ignored per their contract.

However, the subsequent addition of this retry-on-non-existent-table logic to the info table writing helper -- which is shared by both these calls, which did not mind the errors, and those that do generate and return user-visible errors, where the retry was needed, meant that this retry logic was now being hit by callers beyond those in the narrow race condition of IsActive vs txn time, including these callers that were not checking IsActive at all, and as such could be running on a cluster where the cluster version has not and perhaps will not actually advance or run migrations any time soon.

Concretely, this means that on a cluster running 23.1.11 (the first public release where #108357 was backported) where persevere_downgrade_option or anything else is blocking the version from advancing into the 23.1 migrations that create the info table, any execution planning of changefeed/backup/restore/import job planning will kick off this debug logging task, which will then enter an infinite retry loop until it hits some other, non-retryable error or the version upgrade does eventually happen. This appears to be a very tight loop, consuming significant CPU. It would appear that this would worsen each time a new job kicks off one of these logging tasks.

Jira issue: CRDB-33165

[blathers-crl]

Hi @dt, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/disaster-recovery

[dt]

Removing release-blocker from this now that #113929 is in.

[dt]

We don't need to backport anything to upcoming 23.2 for this as it does not support upgrades from 22.2 so with the 23.1 backports merged I'm closing this as fixed (those backports should be in 23.1.12 later this month)