Add info on each HTTP client to HTTP stats

[danhermann]

Elasticsearch currently provides only the count of HTTP clients currently holding an open connection and the number of HTTP connections that have been opened on the node. This is less than what is reported for transport clients and, as #61609 points out, additional information on connected HTTP clients would be very helpful in a number of troubleshooting and monitoring scenarios.

This change adds the following to the "http" section of node stats:

"http" : {
  "current_open" : 14,
  "total_opened" : 2735,
  "clients" : [
    {
      "agent" : "kibana",
      "local_address" : "/192.168.5.60:9200",
      "remote_address" : "/192.168.5.60:58156",
      "last_uri" : "/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip",
      "opened_time_millis" : 1604420317590,
      "closed_time_millis" : -1,
      "last_request_time_millis" : 1604438560498,
      "request_count" : 4408,
      "request_size_bytes" : 261106
    },
    {
      "agent" : "curl/7.68.0",
      "local_address" : "/192.168.5.60:9200",
      "remote_address" : "/192.168.5.203:35834",
      "last_uri" : "/_nodes/stats/http?pretty",
      "opened_time_millis" : 1604438562302,
      "closed_time_millis" : 1604438562302,
      "last_request_time_millis" : 1604438562302,
      "request_count" : 1,
      "request_size_bytes" : 0
    }
    ...
  ]

Some notes about the fields above:

• agent: This is pulled from the x-elastic-product-origin or User-Agent HTTP header if either is present. Note that it would be nice if Elastic products identified themselves with a user-friendly tag. The User-Agent header reports the Go client or the JS client for Beats and Kibana, respectively, which is better than nothing but not as nice as "filebeat" or "kibana", etc. The x-elastic-product-origin header reports a friendly "kibana" value for Kibana, but I don't believe that its purpose nor is it consistent across Elastic products.
• closed_time_millis: This reports -1 for clients that are currently connected. As implemented, clients that disconnected within the last 5 minutes are still included to aid with tracking either clients such as curl that intentionally disconnect immediately or clients that are dropping connections for other reasons.
• request_size_bytes: This tracks cumulative request size.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Stats)

[danhermann]

@elasticmachine update branch

[danhermann]

@elasticmachine run elasticsearch-ci/2

[danhermann]

@elasticmachine update branch

[danhermann]

@elasticmachine run elasticsearch-ci/1

[danhermann]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[danhermann]

@elasticmachine update branch

[danhermann]

@elasticmachine update branch

[danhermann]

@elasticmachine update branch

[danhermann]

@elasticmachine update branch

[dakrone]

LGTM, thanks for iterating Dan!

[danhermann]

Thanks, @dakrone!