Over-replicated ranges after zone config change

[jseldess]

I'm running a 5-node cluster on Digital Ocean. The version running on each node is:

root@docs-1:~# cockroach version
Build Tag:    v1.1-beta.20170921
Build Time:   2017/09/21 15:48:53
Distribution: CCL
Platform:     linux amd64
Go Version:   go1.8.3
C Compiler:   gcc 6.3.0
Build SHA-1:  be60cc5a4ccfb6a597a47f5eb9d0a88a0e1b1289
Build Type:   release-gnu

I was running tpcc against the cluster for a while. Then I created a zone config to replicate the tpcc.customer table 5 times instead of 3:

echo 'num_replicas: 5' | cockroach zone set tpcc.customer --insecure --host=67.205.145.32 -f -

After a few minutes, I checked to make sure the table was, in fact, replicated 5 times:

[email protected]:26257/> show testing_ranges from table tpcc.customer;
+-----------+---------+-------------+--------------+
| Start Key | End Key |  Replicas   | Lease Holder |
+-----------+---------+-------------+--------------+
| NULL      | NULL    | {1,2,3,4,5} |            3 |
+-----------+---------+-------------+--------------+
(1 row)

Time: 938.839773ms

A little while later, I change the tables replication factor back to 3:

echo 'num_replicas: 3' | cockroach zone set tpcc.customer --insecure --host=67.205.145.32 -f -

I confirmed that the zone config was updated:

root@docs-1:~# cockroach zone get tpcc.customer --insecure --host=67.205.145.32
# Server version: CockroachDB CCL v1.1-beta.20170921 (linux amd64, built 2017/09/21 15:48:53, go1.8.3) (same version as client)
# Cluster ID: c3779182-655d-4c0e-955b-5b4e814e47b4
tpcc.customer
range_min_bytes: 1048576
range_max_bytes: 67108864
gc:
  ttlseconds: 90000
num_replicas: 3
constraints: []

But after another 20 min or so, the table still has 5 replicas:

[email protected]:26257/> show testing_ranges from table tpcc.customer;
+-----------+---------+-------------+--------------+
| Start Key | End Key |  Replicas   | Lease Holder |
+-----------+---------+-------------+--------------+
| NULL      | NULL    | {1,2,3,4,5} |            3 |
+-----------+---------+-------------+--------------+
(1 row)

Time: 1.008673392s

[a-robinson]

@BramGruneir you mentioned earlier that this was due to replicas not being GC'ed after being removed from the raft group (rather than being due to not removing any of the replicas from the raft group). How did you determine that from this info? Were you inspecting the cluster directly? And if so, is it perhaps still up to take a look at?

[tbg]

We should cake this specific type of operation into a replication-minded roachtest to see if this is a reproducible issue.

[tbg]

Closing for #17000.