sql-foundations: tweak regionless restore syntax and semantics

[msbutler]

#110606 Added the new strip_localities option to RESTORE. The DR team thinks two small things should change before this feature is released into the wild. I apologize for overlooking these during the review!!

1. we should change strip_localities to remove_regions (or something similar), because 1) strip_localities sounds too similar to the existing skip_localities option and 2) and "localities" (e.g. locality aware backups) are slightly different from "regions". This feature strips regions from sql descriptors.
2. this is more philosophical: the restore cmd has a contract that all tables must be available to serve writes once the restore command finishes. Currently, an rbr table restored via this command is unavailable for writes. The DR team thinks an unavailable table, even if there is a post restore workaround, is worse than a failed Restore. To that end, the DR team proposes:

• a quick fix that @dt will comment about in issue
• if this workaround is a no go, regionless restores that contain an rbr table should fail.

3. since the plan is for this to be backported to 23.1, we should add mixed-version testing

Jira issue: CRDB-31866

[blathers-crl]

Hi @msbutler, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[dt]

For an RBR table restored into a non-RBR database, can we a) restore the enum but rename it to something like tblname_gateway_region and just make it a normal UDT and then b) change the DEFAULT expression on the column to just be a constant (maybe we add a new value to the enum like unknown or empty when we convert it to a plain UDF and then use that)?

[msbutler]

@annrpom one other thing we ought to do: version gate this feature, i.e. in restore_planning.go, add the following pseudocode:

if !p.ExecCfg().Settings.Version.IsActive(ctx, cluserv23_2) && args.removeLocalities {
return errors.New("can only use remove localities on a cluster that has fully updgraded to 23.2")
}

This pseudo code would avoid the following mixed version bug: a 23.2 node plans the regionless restore, then a 23.1 node picks up the job but is completely unaware of this new feature and will execute a restore as if this feature did not exist.

[annrpom]

i was going to leave a comment about where i landed with the result of (2.), but @msbutler is quick with it :)

essentially, i think that if we cannot completely undo the effects of regional by row, we shouldn't try to at all (otherwise, everything can "seem" to be working, but internally not be what the user expects - which might cause some trickle-down bugs). i believe it is up to the user to figure out how they want regional by row to be consolidated

this was said after I tinkered with David's solution - I could be wrong, but it seemed like there was a little more to it (in particular, i am not sure how to resolve the fact that if such enum is renamed, its "array type descriptor" still causes a collision)

thus, i have my PR up for "failing fast" (or as fast as I think we can) if someone tries to perform a regionless restore with rbr table(s)

feel free to correct me if i am wrong or pair with me if you would like to dig into this further