RESTORE: Restores may split ranges between column families of same row

[dt]

This is coming from https://github.com/cockroachlabs/support/issues/2549#issuecomment-1692104972

If a DB was restored, and the schema of that DB has a table with multiple column families, SQL may fail reading or return wrong results.

Restore calls EnsureSafeSplitKey() before using a key as a split point, and then passes it to AdminSplit(). If the key was a valid key then EnsureSafeSplitKey() modifies the key to not be between column families, meaning it ensures the key would be a safe split point.

If the key is not a table key then EnsureSafeSplitKey() fails, but Restore can still split on that key - it is safe to split on non-table keys. But, if the key is a table key that EnsureSafeSplitKey() cannot parse then it will fail, and Restore (wrongfully) assumes that the key is again, not a table key.

In the bug here, the key was produced by Key.Next() which adds a 0 (NULL) to the end of the key, and that key cannot be parsed properly by EnsureSafeSplitKey(). Restore still used that key as a split point even though it may be between column families.

If the schema has column families, and an unsafe split point falls between column families then, when reading the table, SQL may either fail because it cannot find the column that is in another range (a different column family), or worse, SQL may succeed but return NULL if those missing columns are nullable (wrong result).

Worth mentioning that the data is not lost, and therefore by changing the range splits the DB can be fixed.

There are a things to do here:

• Fix: avoid sending a potentially unsafe split point to KV. If we only implement this fix, then Restore will be slower because we will not issue about half of the split point.
• Fix: avoid using Key.Next() for restore spans (or at least avoid allowing those to be used as split point).
• Improve testing: the reason this was not found in our nightly roachtests is that we run a KV level fingerprinting on the restored DB, but from the perspective of KV the data exists and correct. In order to avoid this issue we need to run a SQL fingerprinting in those tests.
• Produce a script/tool/instructions on how to fix a DB that was restored and may have unsafe split points.

Jira issue: CRDB-30944

[blathers-crl]

cc @cockroachdb/disaster-recovery

[blathers-crl]

cc @cockroachdb/disaster-recovery