Shards should rebalance across multiple path.data on one node

[mikemccand]

Follow-on from #11185:

As of #9398 we now allocate an entire shard to one path on node's path.data, instead of file by file.

Even though we do the initial allocation to the path.data with the most free space, if the shards then grow in lopsided ways (an "adversary"), we can get to a state where one path.data is nearly full while others are very empty, on a single node. I suspect such adversaries would not be uncommon in practice...

Yet, DiskThresholdDecider only looks at total usable space on the node (not per-path on path.data) so it won't notice when only one path is close to full... we need to fix that? Also, the shard allocation process needs to see/address each path.data separately somehow?

Sometimes a shard would just move from one path.data to another path.data on the same node (maybe we should bias for that, all other criteria being nearly equal, since we save on network traffic).

I think it's important to fix this but this is beyond my knowledge of ES now ...

[dakrone]

DiskThresholdDecider only looks at total usable space on the node (not per-path on path.data) so it won't notice when only one path is close to full... we need to fix that?

+1, I think instead of doing the average for all the paths we can just use the max used data path on the node?

[mikemccand]

I think instead of doing the average for all the paths we can just use the max used data path on the node?

I think so, at least for the logic that notices when a node is "getting full" and triggers a rebalance.

[clintongormley]

I think there are only two possible solutions here:

1. Write a path-aware allocator
2. Remove multi-path support

[clintongormley]

If we remove multi-path support, users can run multiple instances per server, and we'd need to provide a tool to migrate data before starting the cluster (or use snapshot/restore)