Nodes drop their copy of auto-expanded data when coming up, only to sync it again

[avar]

When you have an index with index.auto_expand_replicas=0-all running on 3 nodes and you bring down one node the number of replicas will be reduced by the master from 2 to 1. Then when the node that just went down comes up again ElasticSearch on that node will will:

1. Go up, notice that the number of replicas for that index is 1, and promptly drop its own data as redundant
2. The master will notice that it has a new node in the cluster, set the number of replicas to 2.
3. The node that just dropped its data will now have the data it just dropped re-synced to it.

Instead ElasticSearch should:

1. Go up, wait for the master to adjust the number of replicas if needed
2. Only after that's done drop anything, if needed.
3. Not re-sync any data since it didn't drop the data in the brief interim when the master was adjusting the number of replicas from 1 to 2.

This'll aid recovery time where you have a setup where a relatively small index is available on all the nodes for capacity reasons, and you bring up a new node that should serve search requests right away.

[clintongormley]

Yeah, I was just hit by this one too. Wonder what happens if you disable reallocation before shutting down?

[martijnvg]

Perhaps scheduling the deletion of the physical shard files when a shard is no longer allocated on a node can help here. Then there is time window the master node can react the the node rejoining and the deletion of physical shard files can be cancelled.

[portante]

@ywelsch, @clintongormley, is this issue going to be addressed for 2.4.x any time soon? Is this issue a problem with 5.x?

[lukas-vlcek]

@portante I just tested with ES v5.2.2 and I am able to replicate this issue.

[elasticmachine]

Pinging @elastic/es-distributed