jobs: excessive dsp-diag-url profiler rows

[dt]

In 23.1 we started storing the physical plan for many job types in rows of their prefix of the job info table to aid in debugging. These rows are created every time the job plans a physical flow, and generally are deleted when the job is deleted. For very long-running jobs that replan many, many times in their lifetime, such as a changefeed that runs for months or years and encounters errors, node restarted, rolling upgrades and restarts, etc, the number of persisted plans can grow to be large (currently a fact significantly compounded by the fact that the diagrams URLs for changefeeds in particular can be massive, even for just a single plan, due to a lack of limiting on detail such as number or length of spans from the spec copied into the diagram URL).

At one point we considered deleting "debugging" data like these persisted plans even before the job itself was deleted and all of its data was deleted. Indeed, in 23.2 we started prefixing these plan rows and other similar rows that were persisted only for later manual debugging inspection with the character ~ so that this prefix could be what was scanned and culled if we later added a clean-up loop similar to the terminal job one to clean up specifically "ephemeral" debugging data.

A time-based expiration for all "ephemeral" data has the downside however of potentially removing the most-recent diagram, that is still actively executing, if it executes for longer than the retention period. Also, a job in a tight replanning loop could still find itself with "too many" rows even without any of them being old enough to be eligible for cleanup under this time based policy. This led us to hold off on implementing this blanket policy until we saw evidence that it was needed and justified its downsides.

However for the specific case of diagram URLs, we can have the task that persists them also delete older ones for that job, to keep the number retained per job limited, regardless of their age.

Jira issue: CRDB-39767

[blathers-crl]

Hi @dt, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[benbardin]

Will this be an issue with any other type of metadata, or just DSP diagrams?

[dt]

re-opening until the backports land