[Metrics UI] Too Many Buckets crashes Kibana when previewing inventory alerts

[Zacqary]

It's difficult to implement a Too Many Buckets error handler for previewing inventory alerts, because the Snapshot API seems to crash Kibana when it hits one of these.

Acceptance Criteria

• Gracefully handle too_many_buckets_exception from the Snapshot API without crashing Kibana

To Reproduce

Using data from our shared cluster, get a Snapshot of a node type that there are a LOT of (Kubernetes Pods seems to do it for me), and set the lookbackSize to a month.

Output of the error I'm hitting:

Unhandled Promise rejection detected:

{ Error: [search_phase_execution_exception] 
    at respond (/Users/zacqary/Code/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/zacqary/Code/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/zacqary/Code/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/zacqary/Code/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 503,
  displayName: 'ServiceUnavailable',
  message: '[search_phase_execution_exception] ',
  path: '/metricbeat-*/_search',
  query:
   { allow_no_indices: true,
     ignore_unavailable: true,
     ignore_throttled: true },
  body:
   { error:
      { root_cause: [],
        type: 'search_phase_execution_exception',
        reason: '',
        phase: 'fetch',
        grouped: true,
        failed_shards: [],
        caused_by: [Object] },
     status: 503 },
  statusCode: 503,
  response:
   '{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":[],"caused_by":{"type":"too_many_buckets_exception","reason":"Trying to create too many buckets. Must be less than or equal to: [65535] but was [66286]. This limit can be set by changing the [search.max_buckets] cluster level setting.","max_buckets":65535}},"status":503}',
  toString: [Function],
  toJSON: [Function] }

Terminating process...
 server crashed  with status code 1

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[simianhacker]

What about setting timerange.ignoreLookback: true and timerange.forceInterval: true then you can control everything with the timerange.to, timerange.from and timerange.interval? The number of buckets is going to be to - from / (intervalInSeconds * 1000). If you need a months worth of 1 minute data you will need to make several requests to prevent hitting the circuit breakers.

[Zacqary]

That could definitely work, but I'm concerned that this API crashes Kibana entirely when you hit this error. Like it just freezes, I have to Control-C it and restart and everything.

[simianhacker]

Is the issue that the promise is not being handled in the Alert controller? I guess it's not clear to me what changes need to be made.

[Zacqary]

Not sure where the promise rejection occurs. Here's the traceback I got:

{ Error: [search_phase_execution_exception] 
    at respond (/Users/zacqary/Code/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/zacqary/Code/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/zacqary/Code/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/zacqary/Code/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Looks like one of those broken call stack kind of situations, I doubt that tells us where in the code the actual problem occurred.

It might not be happening in the Snapshot API. Let's keep this issue open to track it down.

[Zacqary]

Yep, turns out that this is an uncaught error within the Inventory alert and not the Snapshot API. I'll add a handler.

[Zacqary]

Spoke too soon. The handler I wrote worked once and has now ceased to work. This might be a deeper problem.