upgrades: try to repair catalog corruptions during upgrade

[postamar]

This change enhances the recently-added upgrade precondition check which
checks for catalog corruptions by trying to repair these automatically
when possible.

Fixes: #104425
Fixes: #85265

Release note (general change): upgrading the cluster version to a new
release will not only check for descriptor- and other catalog
corruptions but will attempt to repair some of them on a best-effort
basis. This should seamlessly get rid of all longstanding descriptor
back-reference corruptions, which typically don't manifest themselves
until a schema change or an upgrade are performed.

[cockroach-teamcity]

This change is 

[postamar]

This is ready for review, I'll reconcile the conflicts around hardcoded OIDs before borsing.

[fqazi]

Great work, this really good!

Reviewed 1 of 1 files at r1, 6 of 6 files at r2, 16 of 16 files at r3, 4 of 4 files at r4, 4 of 4 files at r5, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @dt, @knz, @postamar, and @rafiss)

---

pkg/sql/sem/builtins/builtins.go line 5226 at r4 (raw file):

SELECT
	CASE corruption
	WHEN 'descriptor'

Maybe document the corruption types in the code, just to make things a bit cleaner.

---

pkg/upgrade/upgrades/first_upgrade.go line 139 at r5 (raw file):

	} else if hasRows {
		// Attempt to repair catalog corruptions.
		log.Info(ctx, "auto-repairing catalog corruptions detected during upgrade attempt")

From a diagnosability viewpoint, do you think logging which descriptors/corruption type is worth logging? It might be helpful for customer scenarios where we didn't cover something.

---

pkg/sql/logictest/testdata/logic_test/schema_repair line 284 at r4 (raw file):

SELECT * FROM "".crdb_internal.kv_repairable_catalog_corruptions ORDER BY 1,2,3,4
----
104  105  _corrupt_backref_typ  114    descriptor

Nice!

[postamar]

Thanks for the review!

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @dt, @fqazi, @knz, and @rafiss)

---

pkg/sql/sem/builtins/builtins.go line 5226 at r4 (raw file):

Previously, fqazi (Faizan Qazi) wrote…

Maybe document the corruption types in the code, just to make things a bit cleaner.

Done, in the vtable definition mostly.

---

pkg/upgrade/upgrades/first_upgrade.go line 139 at r5 (raw file):

Previously, fqazi (Faizan Qazi) wrote…

From a diagnosability viewpoint, do you think logging which descriptors/corruption type is worth logging? It might be helpful for customer scenarios where we didn't cover something.

Good question. I'm worried there's a real possibility of there being too many of them. In fact, I should perform the repair in batches!

---

pkg/sql/logictest/testdata/logic_test/schema_repair line 284 at r4 (raw file):

Previously, fqazi (Faizan Qazi) wrote…

Nice!

Done.

[fqazi]

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @dt, @knz, @postamar, and @rafiss)

---

pkg/upgrade/upgrades/first_upgrade.go line 139 at r5 (raw file):

Previously, postamar (Marius Posta) wrote…

Good question. I'm worried there's a real possibility of there being too many of them. In fact, I should perform the repair in batches!

Batching here makes tons of sense. I guess we could limit that type of logging into a small number or the First N of these, but I'll leave that up to you.

[postamar]

I'll leave that for a later time, I think.

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from d8a2197 to blathers/backport-release-23.1-105832: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}