multiregionccl: add unsafe_optimize_system_database

[jeffswenson]

crdb_internal.unsafe_optimize_system_database optimizes the system
database for multi region deployments by configuring latency critical
tables as global or regional by row. Tables that are sensitive to read
latency were converted to global. Table that are sensitive to write
latency were converted to regional by row.

system.sqlliveness and system.sql_instances were reworked by (#90408)
and (#92010) to have a primary key index that is binary compatible with
regional by row tables. The conversion to regional by row is
implemented as a built in function because it is not possible to use
ALTER TABLE ... SET LOCALITY REGIONAL BY ROW to perform the conversion.
Using alter table to convert the RBR tables has two challenges:

1. There is no way to convert the crdb_region column from []byte to the
   region enum type.
2. Changing the locality to RBR always rewrites the primary index. The
   sqlliveness and sql_instance subsystems use the raw kv API and expect a
   specific index ID. Rewriting the primary index changes the ID and breaks
   the subsystems.

Release note: None

Part of #85736

[cockroach-teamcity]

This change is 

[jeffswenson]

@ajwerner this needs more testing before I'm ready to submit it, but can you take a look at let me know what you think of the overall approach?

[jaylim-crl]

Changing the crdb_region column to an enum type is OK, but the behavior doesn't seem right. Once the type has been changed, we couldn't compare with a string anymore:

root@localhost:26257/defaultdb> select * from system.sql_instances where crdb_region = 'gcp-us-central1';
ERROR: unsupported comparison: bytes to crdb_internal_region
SQLSTATE: 42804

On the other hand, for a regular regional by row table, I could still filter by crdb_region, and the DB knows what the possible values are:

[email protected]:26257/test> select * from users where crdb_region = 'us-central1';
ERROR: invalid input value for enum crdb_internal_region: "us-central1"
SQLSTATE: 22P02
[email protected]:26257/test> select * from users where crdb_region = 'gcp-us-central1';
                   id                  |  city   | first_name | last_name | address
---------------------------------------+---------+------------+-----------+----------
  6f0be85d-4d46-48a5-af3d-b3925b7aaa54 | chicago | foo        | bar       | baz
(1 row)

[ajwerner]

Yes, this approach is sound

[jeffswenson]

Changing the crdb_region column to an enum type is OK, but the behavior doesn't seem right. Once the type has been changed, we couldn't compare with a string anymore:

root@localhost:26257/defaultdb> select * from system.sql_instances where crdb_region = 'gcp-us-central1';
ERROR: unsupported comparison: bytes to crdb_internal_region
SQLSTATE: 42804

On the other hand, for a regular regional by row table, I could still filter by crdb_region, and the DB knows what the possible values are:

[email protected]:26257/test> select * from users where crdb_region = 'us-central1';
ERROR: invalid input value for enum crdb_internal_region: "us-central1"
SQLSTATE: 22P02
[email protected]:26257/test> select * from users where crdb_region = 'gcp-us-central1';
                   id                  |  city   | first_name | last_name | address
---------------------------------------+---------+------------+-----------+----------
  6f0be85d-4d46-48a5-af3d-b3925b7aaa54 | chicago | foo        | bar       | baz
(1 row)

Andrew and I debugged this. The error is coming from the optimizer. The statistics for the table contain the type of the column and that type is used when planning the query. Regenerating statistics for the table causes the error to go away. Arguably we are breaking an invariant in the optimizer: the type of a column is changing, but the id of the column is not changing. For now we will work around this by clearing stats in the unsafe_optimize_system_database built in. During tenant creation, erasing stats will be a no-op, since stats will not have been generated by the time the tenant is created.

[jeffswenson]

I removed the draft label. This change is ready for review.

[ajwerner]

Reviewed 7 of 8 files at r2, 1 of 1 files at r3, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @jaylim-crl)

[jeffswenson]

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)