Field to identify different kinds of metrics

[dgieselaar]

We have various types of metrics:

• system metrics (e.g. memory usage, cpu usage)
• transaction breakdown/duration metrics
• span metrics

IIRC, a user might have also have custom metrics.

Currently we don't really have a way to tell what kind of metric a document is. This makes it a little harder to write efficient queries, as we have to add safe-guards to make sure that a certain field is present. It's also a (small) burden on maintenance, and we've found that it's hard for us to understand how we should get the data we need.

It would help if a field was added that identified the type of metrics, e.g. metric.type:transaction_duration. There might be just a group, but maybe a sub-group as well.

See also: elastic/kibana#75881 and #4004.

[axw]

I'm fine with this, though I'd probably call it metricset.name for consistency with Metricbeat: https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-common.html.

System, runtime, and transaction breakdown metrics come agents. In theory we could look for the specific metric names in the server and set the metric set name; but this falls apart when new metrics are added. I think we would need agents to start reporting a metric set name. @felixbarny WDYT?

[felixbarny]

The way agents currently create metric sets is currently not based on explicit grouping. Instead, they create a metric set document for all metrics that have the same dimensions.

It's certainly possible to add a grouping based on a metric set name for internal metrics. But for custom metrics, it might not be. Especially if 3rd party metric APIs, such as Prometheus or Micrometer are used that don't have a notion of metric sets. But probably it's fine to just have different metric set names for built-in metrics and a single metric set name for custom metrics. Based on the metric set name, the APM Server could route the metrics to different indices, enabling different ILM policies.

However, it does take coordinated effort across all agents. And even after all agents have implemented it, folks might use new versions of APM Server and Kibana with old versions of the agents that don't set the metric set name. So the UI can't rely on these fields to be present.

This makes it a little harder to write efficient queries

Is that because an exists query is less efficient than a term query?

I think the current plan is to have different indices for internal metrics and custom metrics. Does that cover your needs?

I guess I don't fully grasp what the benefit of having a metricset.name would be in practice and whether it's a crucial thing we need or if it's a nice-to-have.

[dgieselaar]

@felixbarny it might be one exists query, but it also might be multiple exists queries.

I think the current plan is to have different indices for internal metrics and custom metrics. Does that cover your needs?

That would only help in differentiating between internal metrics and custom metrics, right? So no distinction between for instance transaction duration metrics and cpu metrics.

[jalvz]

In theory we could look for the specific metric names in the server and set the metric set name; but this falls apart when new metrics are added

Can we do that for existing / known metrics, and require agents to set names for upcoming / new metrics?

And/or be explicit about what kind of metrics can't be guaranteed to have a name so that the UI can narrow down the usage of exists clauses?

[axw]

@dgieselaar would you be able to enumerate the cases where this field would be used in the UI? That would help us narrow down the requirements a bit.

If we can get away with just identifying transaction histogram metrics, service destination metrics, and breakdown metrics, then it should be straightforward to implement this in the server. If we also need to identify system metrics then it's less straightforward, as they may be grouped with application metrics, and the server would need to split them apart.

[dgieselaar]

@axw missed this one, I'll add it to my to-do list, should be able to do this later this week.

[dgieselaar]

Ideally we have a subcategory of transaction/span metric types, ie:

• transaction metric (for all transaction stats)
• span metric (for service destination stats)
• transaction breakdown metric (for time spent by span type)
• span breakdown metric (for time spent by span type)

For system or custom metrics, it'd be great if we have system_cpu, system_memory, etc but from what you're saying that's not really possible. For those metrics, system_metric or custom_metric is (IMHO) fine.