Noisy null values in metric point data_points

[srikanthccv]

Noticed large number of None values from returned metric point data_points. Run this example for some time and you should notice it.

import random
import time
from typing import Iterable

from opentelemetry.metrics import (
    CallbackOptions,
    Observation,
    get_meter_provider,
    set_meter_provider,
)
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader

reader = InMemoryMetricReader()
provider = MeterProvider(metric_readers=[reader])
set_meter_provider(provider)


values = ["v1", "v2", "v3", "v4"]


def observable_counter_func(options: CallbackOptions) -> Iterable[Observation]:
    yield Observation(1, {"k1": random.choice(values), "k2": random.choice(values)})


def observable_up_down_counter_func(
    options: CallbackOptions,
) -> Iterable[Observation]:
    yield Observation(-10, {"k2": random.choice(values), "k3": random.choice(values)})


def observable_gauge_func(options: CallbackOptions) -> Iterable[Observation]:
    yield Observation(9, {"k4": random.choice(values)})


meter = get_meter_provider().get_meter("getting-started", "0.1.2")

# Counter
counter = meter.create_counter("counter")
counter.add(1)

# Async Counter
observable_counter = meter.create_observable_counter(
    "observable_counter",
    [observable_counter_func],
)

# UpDownCounter
updown_counter = meter.create_up_down_counter("updown_counter")
updown_counter.add(1)
updown_counter.add(-5)

# Async UpDownCounter
observable_updown_counter = meter.create_observable_up_down_counter(
    "observable_updown_counter", [observable_up_down_counter_func]
)

# Histogram
histogram = meter.create_histogram("histogram")
histogram.record(99.9)

# Async Gauge
gauge = meter.create_observable_gauge("gauge", [observable_gauge_func])


while 1:

    metrics = reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics
    for metric in metrics:
        data_points = list(metric.data.data_points)
        print(data_points)
    time.sleep(1)

[agrapsas]

I am getting a crash in the opentelemetry-exporter-otlp-proto-grpc exporter because of this.

2022-06-03 20:25:28 [16,298ms] [Error] [opentelemetry.sdk.metrics._internal.export] [/home/**REDACTED**/opentelemetry/sdk/metrics/_internal/export/__init__.py:406] Exception while exporting metrics 'NoneType' object has no attribute 'attributes'
Traceback (most recent call last):
  File "/home/**REDACTED**/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 404, in _receive_metrics
    self._exporter.export(metrics_data, timeout_millis=timeout_millis)
  File "/home/**REDACTED**/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 207, in export
    return self._export(metrics_data)
  File "/home/**REDACTED**/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 296, in _export
    request=self._translate_data(data),
  File "/home/**REDACTED**/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 143, in _translate_data
    data_point.attributes
AttributeError: 'NoneType' object has no attribute 'attributes'

The exporter does not check for None, so, this causes an exception when iterating over the data_points: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py#L136

Do we have an idea why there are None values? Is it expected behavior?

[srikanthccv]

Quoting @aabmass answer from personal discussion few days back The cause is most likely a metric labelset didn't see an observation for the collection cycle. Throwing AttributeError is definitely not an expected behaviour.

[srikanthccv]

@agrapsas Thanks for trying out the metrics rc1 SDK and reporting the issues.

[ocelotl]

I am getting a crash in the opentelemetry-exporter-otlp-proto-grpc exporter because of this.

2022-06-03 20:25:28 [16,298ms] [Error] [opentelemetry.sdk.metrics._internal.export] [/home/**REDACTED**/opentelemetry/sdk/metrics/_internal/export/__init__.py:406] Exception while exporting metrics 'NoneType' object has no attribute 'attributes'
Traceback (most recent call last):
  File "/home/**REDACTED**/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 404, in _receive_metrics
    self._exporter.export(metrics_data, timeout_millis=timeout_millis)
  File "/home/**REDACTED**/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 207, in export
    return self._export(metrics_data)
  File "/home/**REDACTED**/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 296, in _export
    request=self._translate_data(data),
  File "/home/**REDACTED**/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 143, in _translate_data
    data_point.attributes
AttributeError: 'NoneType' object has no attribute 'attributes'

The exporter does not check for None, so, this causes an exception when iterating over the data_points: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py#L136

Do we have an idea why there are None values? Is it expected behavior?

@agrapsas I have opened #2743 to track this issue and should be submitting a PR to fix it shortly.

[ocelotl]

#2745 was merged, closing.