Delete By Query under heavy indexing load causes OOM errors

[xyu]

Hello,

We are running an ES 1.1.1 cluster and when we bulk index into it with a high write load we have found that it can trigger OOM errors if we run deletebyquery calls. The symptoms of the problems appears to be the following:

1. Index into the cluster with a heavy write load, this will cause merges to pile up. During the pileup the merge thread pool queue goes up and stays at ~120 per node. (Maybe related to Merges might not be picked up when they are ready #5779?)
2. Merges are no longer able to keep up with new segments created from indexing operations and segments per node shoots up. (We have seen over 30k segments per node.)
3. Long GC cycles are triggered which may or may not cause the node to drop out of the cluster.
4. When a Delete By Query call is issued an internal refresh is triggered (Delete by query should not silently refresh index #3593) which fails due to an out of memory error and causes the node to be removed from the cluster for not responding. (See gist for logs. In this example es13.iad sends a call to es13.sat which experiences OOM causing es13.iad to remove it from the cluster.)

Please let me know if there are any other info I can provide that may be helpful.

[xyu]

It appears that we are experiencing segment explosion issues during these times and are seeing queues in the merge thread pool. For now we are working around this issue by backing off on bulk index when merges fall behind in our indexing jobs. It looks like this issue is also being worked on in #6066

[clintongormley]

Fixed by #6066

[mikemccand]

Alas, deleteByQuery is not throttled when merges are falling behind.

[mikemccand]

I'll make this operation throttled like we do for index/create ops.

However I'm doubtful this is "enough" to always prevent segment explosion, i.e. #7052 is a better (trickier) long term solution.

[s1monw]

@mikemccand can we close this?

[mikemccand]

Yeah, I'll close it ... I think you can still easily provoke OOME on 1.6, but in 2.0 we are switching to a plugin that runs scan/scroll query and then bulk delete the resulting IDs, which does fix it.