Add Circuit breaker on Transport ResponseHandlers

[easyice]

for every in flight Transport request， we add a handler at org.elasticsearch.transport.Transport.ResponseHandlers

public long add(ResponseContext<? extends TransportResponse> holder) {
            long requestId = newRequestId();
            ResponseContext existing = handlers.put(requestId, holder);
            assert existing == null : "request ID already in use: " + requestId;
            return requestId;
      }

some times, the users sent a lot of request, such as query, that they exceed the system's capacity，then, CPU utilization reached 100%, but ,The client still sends a large number of requests，the requests will be add in ResponseHandlers#handlers

then the Elasticsearch nodes will be oom

This happened to me a few times,I dump the jvm memory，open with Eclipse Memory Analyzer, it show the ResponseHandlers#handlers used 7.7GB memory

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I don't think there's a general transport-level solution for limiting the size retained by a response handler since the memory retained by a handler is pretty much independent of the transport layer. The OP doesn't tell us anything about what requests correspond with the problematic response handlers, nor what version they're using. Likely culprits include searches (in which case this duplicates #67478), stats (in which case this duplicates #55550) or bulks (in which case this is resolved by the new indexing pressure mechanisms). @easyice to what requests do the problematic handlers relate?

[Hailei]

I don't think there's a general transport-level solution for limiting the size retained by a response handler since the memory retained by a handler is pretty much independent of the transport layer.

I can't agree any more

@easyice is my coworker, The cluster that this issue mentioned is the same to me

ES version: 6.8.0, The request were bulk in the last two accident @DaveCTurner

[DaveCTurner]

Ok, there is a solution for limiting the size retained by bulk response handlers, and it's already implemented (as of 7.9) so I think there's no further action needed here.

[easyice]

@DaveCTurner Thanks for reply, Let me add the relate requests, It doesn't just appear in bulk，also in search request,like this:

[DaveCTurner]

Indeed, we're tracking the issue for search responses at #67478.