[BUG] DiskThresholdDecider should consider shard size of all relocating shards on the target node while making relocation decisions

[RS146BIJAY]

Describe the bug
OpenSearch relocates shards away from the node on which it has breached high disk watermark. While selecting the target node for a relocating shard, DiskThresholdDecider considers that allocating this shard on a node will not bring the target node above the high watermark. Since relocation can happen concurrently, it is possible that DiskThresholdDecider selects the same node as the target node for multiple relocations. Here, it is possible that even though individual checks for shard relocation decision can pass, relocating all the shards (across concurrent relocations) can cause free disk space on the target node to become 0.

Expected behavior
While selecting the target node, DiskThresholdDecider should consider shard size of all the shards that are migrated to this node, not just the current relocating shard.

[adnapibar]

@RS146BIJAY Thanks for reporting the issue, can you please provide details on how to reproduce this or at least a test for this scenario?

[jayeshathila]

As per my understanding of the issue , we are not failing fast before starting the allocation in case of disk space breach.

If no one has started on the issue, I can pick this one.

[RS146BIJAY]

@jayeshathila we are already working on this fix.

[shwetathareja]

Since relocation can happen concurrently, it is possible that DiskThresholdDecider selects the same node as the target node for multiple relocations. Here, it is possible that even though individual checks for shard relocation decision can pass, relocating all the shards (across concurrent relocations) can cause free disk space on the target node to become 0.

@RS146BIJAY In The DiskThresholdDecider, it iterates over all the relocating shards which are in "INITIALIZING" state so it does handle multiple recoveries in single reroute operation.

OpenSearch/server/src/main/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDecider.java

Line 133 in 4b4d84e

final List<ShardRouting> initializingShards = node.shardsWithState(ShardRoutingState.INITIALIZING);

Can you expand when you say concurrent relocation? Any change in the shard allocation/ movement is executed by the active leader which processes them sequentially so there will never be concurrent relocations triggered from multiple threads.

Now there could be issues with

1. estimation logic for expected shard size during recovery
2. shards which are taking active write, their size can change during recovery itself.

[RS146BIJAY]

@shwetathareja yeah we crossed check this and validated that parallel relocation may not be possible. We have kept this issue on hold as of now. We will revisit this issue again once we have a bit more data points around what may have caused the issue on the affected domain.