jobs,*: stop writing payload and progress to system.jobs

[adityamaru]

This change introduces a cluster version after which the
payload and progress of a job will not be written to the
system.jobs table. This will ensure that the system.job_info
table is the single, source of truth for these two pieces of
information.

This cluster version has an associated upgrade that schema changes
the payload column of the system.jobs table to be nullable,
thereby allowing us to stop writing to it. This upgrade step
is necessary for a future patch where we will drop the payload
and progress columns. Without this intermediate upgrade step the
ALTER TABLE ... DROP COLUMN upgrade job will attempt to write
to dropped columns as part of its execution thereby failing to
run the upgrade.

Informs: #97762

Release note: None

[cockroach-teamcity]

This change is 

[adityamaru]

cc: @fqazi this test used to directly mutates the payload column in system.jobs. In a local-mixed-22.2-23.1 the job_info table does not exist until the relevant upgrades have been run. Is it true that under this mixed configuration this test will always run at V22_2 i.e. without running any of the V23_1 upgrades? In that case I can keep both the queries and conditionally run them based on the config the test is being run under.

[miretskiy]

Do we have a concise explanation on who (what packages) should be allowed to access system.jobs? Is it only underlying jobs package (and maybe tests)? Should there be a lint to warn about
system.jobs usage instead of crdb_internal.system_jobs?

[adityamaru]

Ideally, we'd have everyone going through accessors but that cat is out of the bag for system.jobs. Hopefully, we'll be a little stricter about directly reaching into job_info in non-test code. As far as a linter I think it'll become much clearer once these columns are dropped at which point there is no option but to go to the job_info table for payload and progress.

[dt]

This looks like it was exceedingly tedious. 🙌

[adityamaru]

TFTR!

bors r=miretskiy,dt

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Merge conflict.

[adityamaru]

bors r+

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[adityamaru]

bors retry

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[adityamaru]

The failures are all INFO: slow quiesce. stack traces: but surprisingly none of the stacks are the job registry so this looks like a new manifestation of the slow quiesce problem.

[adityamaru]

let's try again

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 5d9b97a to blathers/backport-release-23.1-99458: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}