sstable: errors while using writer with two level index

[pbardea]

Doing a RESTORE on a registration backup cluster was causing an error (see: cockroachdb/cockroach#40670). Disabling two level indexes in Pebble. The first commit that introduced this issue was a Pebble version change: cockroachdb/cockroach@1c1c8ee (which, among other changes, introduced the two level index writer).

This can be reproduced on current CockroachDB current master (8e15411a18211652792baa92369c6cd02deff15c).

Reproduction steps:

1. Checkout commit 8e15411a18211652792baa92369c6cd02deff15c of CockroachDB.
2. Set up a roachprod cluster with the appropriate binary (note that $USER should be your Cockroach email username):

CLUSTER=$USER-secure
build/builder.sh mkrelease amd64-linux-gnu
roachprod create $CLUSTER -n 3 --clouds=aws --aws-machine-type-ssd=c5d.4xlarge
roachprod put $CLUSTER cockroach-linux-2.6.32-gnu-amd64 cockroach
roachprod start $CLUSTER:1-3 --secure
roachprod sql $CLUSTER:1 --secure

3. Restore the registration cluster backup in the SQL shell. (Note that the date may need to be updated to something more recent if this backup is eventually cleared.)

SET CLUSTER SETTING cluster.organization = '<redacted>';
SET CLUSTER SETTING enterprise.license = '<redacted>';
CREATE DATABASE registration;
RESTORE TABLE registration.* FROM 's3://cockroach-reg-backups/2019-09-01?AWS_ACCESS_KEY_ID=<redacted>&AWS_SECRET_ACCESS_KEY=<redacted>';

[petermattis]

I'm about 90% certain the problem is that we're reading the table using golang/leveldb/table which doesn't support two-level indexes. Pebble should disable the use of two-level indexes if pebble.TableFormatLevelDB is specified. But really we should be using pebble/sstable to read the table as the golang/leveldb/table code isn't maintained.