spanconfig: reduce default gc TTL for system.span_configurations

[irfansharif]

Is your feature request related to a problem? Please describe.

This table's opted out of cluster backups:

cockroach/pkg/ccl/backupccl/system_schema.go

Lines 358 to 360 in 0ec2bcd

	systemschema.SpanConfigurationsTable.GetName(): {
	shouldIncludeInClusterBackup: optOutOfClusterBackup,
	},

Frequent zone config changes will a lot of garbage accumulated over this table; in the order of O(descriptors * changes). Reducing the default GC TTL from 25h to something much shorter seems reasonable. spanconfig.KVSubscriber establishes a rangefeed over this table to learn of updates, but it'll retry if GC occurs from underneath it.

Jira issue: CRDB-15798

[irfansharif]

+cc @ajwerner, @arulajmani is this reasonable? If so, can I pawn this off to you Arul?

[arulajmani]

This seems reasonable, though I don't think we did anything special for system.zones. Is the idea that given the flattened structure of system.span_configurations, there's going to be more churn on this table?

can I pawn this off to you Arul?

I can make the change, should mostly be pretty mechanical. What number are you suggesting? 1 hour?

Separately, this is for 22.2, yeah? Given we'd want to introduce a migration here?

[adityamaru]

Fwiw we want to add a migration for all system tables opted out of backup to be marked as exclude_data_from_backup, so that the short GC TTL isn't held up by a cluster backup.

[irfansharif]

Is the idea that given the flattened structure of system.span_configurations, there's going to be more churn on this table?

Yes. system.zones has been around for a lot longer, probably could also benefit from a lowered GC TTL but system.span_configurations will be more prominent. 1 hr seems reasonable to me.

Separately, this is for 22.2, yeah? Given we'd want to introduce a migration here?

I was thinking for 22.1 actually, but I'm not sure if it's important enough to block GA.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!