Force faster failures on slow requests.

[Mpdreamz]

I'm purposefully issuing a badly formed query into my cluster which has enough indices for the returned exception message to exceed 200kb (around ~630 indices).

POST /_search?timeout=1000&master_timeout=1000
{
  "query": { "badjson": {}  }
}

I then start issuing ~20 of these in repetition:

While the first request comes back with a 400 all the others return 503 and some of them take 14s to complete even if I'm specifying timeouts on the request. I suppose they are waiting for the queue since a single search takes 600 of the 1000 slots (for each index?), I'm assuming this because the 503's return and im seeing a 503 straight after the first request.

EsRejectedExecutionException[rejected execution (queue capacity 1000)

Am i missing a toggle here? Ideally the timeout also forces requests to never exceed the specified timeout and return with an appropiate HTTP status code (504?).

[clintongormley]

Closed in favour of #4586