[Feature/SPM]: Support spanmetrics connector

[albertteoh]

Requirement

As a jaeger operator, I want to use the newly introduced spanmetrics connector for the following reasons:

• Introduces a number of improvements to the spanmetrics processor.
• The spanmetrics processor is deprecated and so future enhancements will no longer be added to the processor.
• Avoid being in a situation where the spanmetrics processor is decommissioned preventing me from upgrading my OTEL collector version.

Problem

The known breaking issues include:

• Renaming the calls_total metric to calls [1], while also allowing for a namespace prefix [2], so a metric name could resemble: my_namespace_calls.
  • [1]: [connectors/spanmetrics] Drop _total from generated metrics names. open-telemetry/opentelemetry-collector-contrib#18677
  • [2]: [processor/spanmetricsprocessor] Support set metrics namespace. open-telemetry/opentelemetry-collector-contrib#18697
• Renaming the latency metric to duration [3], while also allowing for a namespace prefix [4], so a metric name could resemble: my_namespace_duration_bucket.
  • [3]: [connectors/spanmetrics] Rename latency histogram metrics to duration. open-telemetry/opentelemetry-collector-contrib#19371
  • [4]: [processor/spanmetricsprocessor] Support set metrics namespace. open-telemetry/opentelemetry-collector-contrib#18697

Proposal

Make the metric names configurable.

Perhaps introduce an spm parameter namespace where metric names can be configured. e.g. --spm.calls-metric-name and --spm.latency-metric-name.

This would also require an update to the example provided in docker-compose/monitor.

Other suggestions welcome.

Open questions

No response

[warning-explosive]

I have a workaround based on metricstransform processor. Here an example of otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  metricstransform/insert:
    transforms:
      - include: calls
        match_type: strict
        action: update
        new_name: calls_total
        operations:
        - action: update_label
          label: span.name
          new_label: operation
      - include: duration
        match_type: strict
        action: update
        new_name: latency
        operations:
          - action: update_label
            label: span.name
            new_label: operation

exporters:
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
  prometheus:
    endpoint: "otel-collector:9464"
    resource_to_telemetry_conversion:
      enabled: true
    enable_open_metrics: true

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
    dimensions:
      - name: http.method
        default: GET
      - name: http.status_code
    dimensions_cache_size: 1000
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, spanmetrics]
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [metricstransform/insert]
      exporters: [prometheus]

[Owenxh]

I want to use the spanmetrics connector too.

[utezduyar]

I have a workaround based on metricstransform processor. Here an example of otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  metricstransform/insert:
    transforms:
      - include: calls
        match_type: strict
        action: update
        new_name: calls_total
        operations:
        - action: update_label
          label: span.name
          new_label: operation
      - include: duration
        match_type: strict
        action: update
        new_name: latency
        operations:
          - action: update_label
            label: span.name
            new_label: operation

exporters:
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
  prometheus:
    endpoint: "otel-collector:9464"
    resource_to_telemetry_conversion:
      enabled: true
    enable_open_metrics: true

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
    dimensions:
      - name: http.method
        default: GET
      - name: http.status_code
    dimensions_cache_size: 1000
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, spanmetrics]
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [metricstransform/insert]
      exporters: [prometheus]

This worked well for me however something is not entirely right. I have a demo application with 5 services talking to each other. All 4 of them has data under Monitor but one of them does not have data. One application out of 4 is very similar to this one that is not working which puzzled me.

Maybe something about the name of the span? Any tips on how to debug it?

[warning-explosive]

I have a workaround based on metricstransform processor. Here an example of otel-collector-config.yaml:

receivers:

otlp:

protocols:

  grpc:

  http:

processors:

batch:

metricstransform/insert:

transforms:

  - include: calls

    match_type: strict

    action: update

    new_name: calls_total

    operations:

    - action: update_label

      label: span.name

      new_label: operation

  - include: duration

    match_type: strict

    action: update

    new_name: latency

    operations:

      - action: update_label

        label: span.name

        new_label: operation

exporters:

otlp:

endpoint: "jaeger:4317"

tls:

  insecure: true

prometheus:

endpoint: "otel-collector:9464"

resource_to_telemetry_conversion:

  enabled: true

enable_open_metrics: true

connectors:

spanmetrics:

histogram:

  explicit:

    buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]

dimensions:

  - name: http.method

    default: GET

  - name: http.status_code

dimensions_cache_size: 1000

aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"

service:

pipelines:

traces:

  receivers: [otlp]

  processors: [batch]

  exporters: [otlp, spanmetrics]

metrics:

  receivers: [otlp, spanmetrics]

  processors: [metricstransform/insert]

  exporters: [prometheus]

This worked well for me however something is not entirely right. I have a demo application with 5 services talking to each other. All 4 of them has data under Monitor but one of them does not have data. One application out of 4 is very similar to this one that is not working which puzzled me.

Maybe something about the name of the span? Any tips on how to debug it?

Several possible issues and solutions that comes to my mind:

1. Spanmetrics namespaces - check if there are any
2. Rejected/lost/dropped spans - check otel-collector own metrics, maybe you have some configuration or connectivity issues
3. Check your app exporter - substitute otel exporter with console exporter so as to be sure that you have expected spans
4. See debug logs - https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md

[utezduyar]

Thanks! The problem was due to the application missing http semantic conventions. Otherwise your workaround works like a charm.