Backfill Index: Minimize rejects due to schema version mismatch.

[amitanandaiyer]

If we are in the middle of updating IndexPermission(s) from one state to another, say from INDEX_PERM_DELETE_ONLY to INDEX_PERM_WRITE_AND_DELETE, then the YBClients will be given the older version (INDEX_PERM_DELETE_ONLY) while the permission is updated on all the tablets.

During this time, YBClients talking to the "updated" tablets may be rejected.

One way to address this may be to use a 8-step approach instead of a 4-step approach and allow the tservers to accept either of the 2 consecutive schema numbers.

Another way to get around this would be to pre-allocate the version numbers for the multiple stages of the Index creation; and allow tservers to accept versions that are newer than it's current version. This is only safe if the only change is the update to IndexPermissions -- if there is an update to the schema itself, the tserver must reject mismatched schemas.

[amitanandaiyer]

partially addressed by #3627.

can be optimised further

[jaki]

This issue seems to be accentuated when using the TEST_slowdown_backfill_alter_table_rpcs flag.

YCQL:

./bin/yb-ctl create --master_flags 'disable_index_backfill=false,TEST_slowdown_backfill_alter_table_rpcs_ms=10000' --tserver_flags 'disable_index_backfill=false'

CREATE KEYSPACE k;
CREATE TABLE k.hn (
    i int PRIMARY KEY,
    j int)
  WITH
    transactions = { 'enabled' : true };
INSERT INTO k.hn (i, j) VALUES (1, 10);

./bin/ycqlsh --execute 'CREATE INDEX hn_idx ON k.hn (j);' &
while [ "$(jobs | wc -l)" -gt 0 ]; do
  ./bin/ycqlsh --execute 'UPDATE k.hn SET j = j + 100 WHERE i = 1;'
  sleep 0.2s
done

You should see

InvalidRequest: Error from server: code=2200 [Invalid query] message="Wrong Metadata Version

when replicating the DELETE ONLY to WRITE AND DELETE permission (tservers receive the new schema, but master still doesn't fully apply it).

YSQL:

./bin/yb-ctl create --master_flags `ysql_disable_index_backfill=false,TEST_slowdown_backfill_alter_table_rpcs_ms=10000' --tserver_flags 'ysql_disable_index_backfill=false'

CREATE TABLE hn (i int, j int, PRIMARY KEY (i));
INSERT INTO hn (i, j) VALUES (1, 10);

./bin/ysqlsh --command 'CREATE INDEX ON hn (j);' &
while [ "$(jobs | wc -l)" -gt 0 ]; do
  ./bin/ycqlsh --command 'SELECT * FROM hn;'
  sleep 0.2s
done

You should see

ERROR:  Query error: schema version mismatch for table 000030a9000030008000000000004005: 2 != 1

when replicating the DELETE ONLY to WRITE AND DELETE permission (tservers receive the new schema, but master still doesn't fully apply it).

The schema version mismatch is only for DELETE ONLY to WRITE AND DELETE because it goes through the UpdateIndexPermission code path (where the test flag adds sleeps).