Load balance metrics for cumulative to delta

[gouthamve]

Component(s)

exporter/loadbalancing

Is your feature request related to a problem? Please describe.

The cumulative to delta processor cannot be scaled right because the same metrics need to end up in the same Collector. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor#warnings

Describe the solution you'd like

Extend the load balancing exporter to also support load balancing metrics.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for exporter/loadbalancing: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

Do you think it's sufficient to take only the metric name into consideration, or should it take the resource attributes (names and values) as well?

[gouthamve]

It should also have attributes. There could be one metric name which is 90% of all the metrics.

We've seen this in Cortex. Before we used to only shard on metric name and this caused extreme hot spotting. We then switched to include the labels too.

[jpkrohling]

Given we have code for both logs and traces, I'm marking this as "good first issue" before working on that myself.

[claudiobastos]

@jpkrohling I would like to contribute on this one

WIP: https://github.com/claudiobastos/opentelemetry-collector-contrib/tree/claudiobastos/loadexporter-add-metrics

[claudiobastos]

@jpkrohling

Does it make sense that routing based on Service and Routing based on Metric (+Resource attr/values) should be mutualy exclusive?

When balancing metrics, if service routing is defined, it should only balance a subset of them ?

[jpkrohling]

@gouthamve, it would be nice to get your opinion, but I think the routing_key for metrics should be one of: service, metric, resource, attribute:

• when service is provided, the service.name resource attribute alone is used for load-balancing
• when metric is provided, the metric name alone should be used
• when resource is used, the metric name plus all resource attribute names and their values should be used
• when attribute is used, the metric name plus all resource attribute names and their values should be used, in addition to the data point's attribute names and values

[gouthamve]

While it makes sense, which would be the default?

I'd lean towards attribute, but only to provide the best load-balancing. But we might be missing use-cases, where resource might make more sense.

[claudiobastos]

How about one routing key specific for metrics ? @jpkrohling @gouthamve

[jpkrohling]

I think service could be a good default, as it probably has the best performance of them all while still being reasonably good for deployments with a good number of services. Otherwise, resource, which would provide the best balancing but with higher costs.

I think we should not implement attribute at the beginning, as it requires going too deep into the metric data points.