Extend metrics intake to accept histograms

[axw]

Agents should be able to send self-describing histogram (values and counts) and summary (sum and count) metrics. We need to be able to identify the metric type, either from an explicit "type" field or inferred from the type-specific fields that are specified (e.g. counts & values specified? it's a histogram).

[axw]

Application metrics sent by agents currently look like:

{
    "timestamp": 123456,
    "tags": {"a": "b", "c": "d"},
    "samples": {
      "metric_name": {
        "value": 123.456
      }
    }
}

There's nothing saying whether a metric is a gauge (point in time) or a counter (monotonically increasing). It's also not possible to record multi-value aggregated metrics (histograms, summaries).

I propose we extend the object format of samples with additional optional fields:

• type: one of "gauge", "counter", "distribution" or "summary"
• unit: one of "percent", "byte", or a time unit (i.e. follow the rules defined at https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-field-meta.html)
• counts and values: counts and values for a distribution-type metric (both required if type is "distribution", and must have the same length; neither allowed otherwise)
• count, sum, min, max: metrics for a summary-type metric (count and sum are required and min and max are optional if type is summary; none are allowed otherwise)

The server will use unit and type to set field metadata:

• if unit is set (and valid), we will set the unit field metadata to the same value
• if type is set to gauge or counter, we will set the metric_type field metadata to the same value
• if neither type nor unit are specified, we will not set any field metadata

Example latency distribution metric:

{
    "timestamp": 123456,
    "samples": {
      "latency_distribution": {
        "type": "distribution",
        "unit": "micros",
        "counts": [3, 2, 1, 0, 0, 1],
        "values": [0.01, 0.1, 1, 10, 100, 1000]
      }
    }
}

Example summary metric:

{
    "timestamp": 123456,
    "samples": {
      "span.self_time": {
        "type": "summary",
        "unit": "micros",
        "sum": "123.456",
        "count": "7"
      }
    }
}

[felixbarny]

Looks good!
Questions:

• How is the unit going to be used? Does it have an effect to the UI?
• Which type should we use for resetting counters? The type counterhttps://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-field-meta.html) seems to be reserved for monotonically incrementing counters.
• Backward compatibility with the old metrics format. Are these interpreted similarly?

{
    "timestamp": 123456,
    "samples": {
      "span.self_time": {
        "type": "summary",
        "unit": "micros",
        "sum": "123.456",
        "count": "7"
      }
    }
}

{
    "timestamp": 123456,
    "samples": {
      "span.self_time.count": {
        "value": "7"
      }
    }
},
{
    "timestamp": 123456,
    "samples": {
      "span.self_time.sum": {
        "value": "123.456"
      }
    }
}

Should agents update send their breakdown metrics in the new format if they are sending data to an APM Server that understands the new format?
Should agents not sent histograms to APM server if they are connected to an older version that doesn't support them yet?

[axw]

@felixbarny

How is the unit going to be used? Does it have an effect to the UI?

It will eventually be used by Kibana: elastic/kibana#82348

Which type should we use for resetting counters?

I think for resetting counters we should omit type for now, and discuss an addition like "resetting_counter" or "delta" with the Metrics, Elasticsearch, and Kibana/Lens teams.

Backward compatibility with the old metrics format. Are these interpreted similarly?

No, a summary type will end up in a single aggregate_metric_double field.

Should agents update send their breakdown metrics in the new format if they are sending data to an APM Server that understands the new format?

Eventually yes, but I don't think we should do that just now. We'll need to figure out a migration plan. I'm not sure how that'll work yet.

Should agents not sent histograms to APM server if they are connected to an older version that doesn't support them yet?

I think that would be best. The current JSON Schema says that "value" is required, but that field will not be sent for histograms or summaries.

[axw]

Due to some issues related to the aggregate_metric_double field type, I'm going to remove summaries from the initial implementation. I've created #5359 to track that separately.