Add a 'hot queries' API for sampling query details

[dakrone]

I have an idea for an API that I would love to see. This API would sample the
queries that came into a node, returning information about the query execution.

This is similar to the hot_threads API that we currently have, but wouldn't
require a user to enable the query slowlog, since that is a higher operational
burden for debugging information than this API would be.

An invocation could look like:

POST /_nodes/hot_queries

With some optional parameters:

• sample_time - how long to sample, maybe a default of 10 seconds?
• maximum_queries - maximum number of queries to sample before returning
• lower_limit - discard queries that take less than this amount of time, ie "10ms"
• human - return things in human readable format also

And the response would look something like:

{
  "queries": [
    { <details> }
  ]
}

And each query response details would look something like:

{
  // The raw query JSON
  "raw_query": "{\"query\": {\"match\": {\"foo\": \"bar\"}}}",

  // The lucene rewritten query similar to what /_validate/query?rewrite=true returns
  "rewritten_query": "foo:bar",

  // How long the query stayed in the queue
  "queue_time": "129ms",
  "queue_time_millis": 129,

  // How long the query spent actually executing
  "execution_time": "2.3s",
  "execution_time_millis": 2309,

  // The node that received this query
  "executed_on": "my_node_id",

  // How many hits the query returned
  "hits": 124082,

  // What indices this query was executed against
  "indices": ["foo", "bar"]
}

This would probably be local information (executed_on above may not be
helpful) and thus it would need to be executed on nodes that do most of the
query coordination (client nodes if they're being used).

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

This ties into the need for better query performance analysis tooling. I wonder if sampling should be done leveraging APM now that #84369 is implemented though.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

This has been open for quite a while, and hasn't had a lot of interest. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.