Getting duplicate metrics being sent from azure function

[Yogesh19921]

I am using OTLP exporter with Azure functions to emit metrics. However, I see that duplicate metrics are being sent every minute (whatever the last value was). This keeps on going until I shut down the function app.

Here's the code for reference:

Configuration of otel with metrics

services.AddOpenTelemetry()
    .WithMetrics((options) =>
    {
        string siteName = Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME") ?? "WEBSITE_SITE_NAME_UNKNOWN";
        OpenTelemetryOptions opentelemetryOptions = serviceProvider.GetService<IOptions<OpenTelemetryOptions>>().Value;
        EvidenceServiceOptions evidenceOptions = serviceProvider.GetService<IOptions<EvidenceServiceOptions>>().Value;
        options
        .AddOtlpExporter((opt, metricReaderOptions) =>
        {
            opt.Protocol = OtlpExportProtocol.HttpProtobuf;
            opt.Endpoint = otlpCollectorEndpoint;
            opt.HttpClientFactory = () =>
            {
               // This part is to make use of oidc with function app.
                var client = new HttpClient(new OpenTelemetryMessageHandler(new DefaultAzureCredential(), new[] {"sometoken" }));
                return client;
            };
        })
        .AddMeter(new string[] { "meter1", "meter2" }) // List of meters for the meter provider
        .ConfigureResource(resourceBuilder => resourceBuilder.AddService("myServiceName").AddAttributes(new Dictionary<string, object>
        {
            ["service.instance.name"] = siteName,
        }));
    });

Code for meter in the function. Not the exact code for obvious reasons.

public class Function1
{
    private static Meter meter = new Meter("meter1");

    // This is through service bus trigger
    public async Task someTask()
    {
            // Some code
            meter.CreateObservableGauge("aNewMetric", () => 1);

           // some more metrics like this

    }
}

I am also using logging in the same function which works perfectly. Just for reference, I am adding it here.

services.AddLogging(logBuilder =>
{
    logBuilder.ClearProviders();
    logBuilder.AddOpenTelemetry(options =>
    {
        OpenTelemetryOptions opentelemetryOptions = serviceProvider.GetService<IOptions<OpenTelemetryOptions>>().Value;

        options.AddOtlpExporter((opt) => {
            opt.Protocol = OtlpExportProtocol.HttpProtobuf;
            opt.Endpoint = otlpCollectorEndpoint;
            opt.HttpClientFactory = () =>
            {
               // This part is to make use of oidc with function app.
                var client = new HttpClient(new OpenTelemetryMessageHandler(new DefaultAzureCredential(), new[] {"sometoken" }));
                return client;
            };
        });
        
        options.SetResourceBuilder(
            ResourceBuilder.CreateDefault()
            .AddService(evidenceServiceOptions.Value.EventSourceName)
            .AddAttributes(new Dictionary<string, object>
            {
                ["service.instance.name"] = siteName,
            }));
    });
});

TLDR:

1. OTLP exporter with metrics keeps on sending same metrics every minute. I can confirm this as I added console exporter and checked the logs being written. They are being sent every minute with same metric values.
2. Logging with the same otlp exporter works perfectly.

What I want:

1. Metrics to be sent every minute (with value zero or null if nothing has been set).

[cijothomas]

What you are seeing is the expected, normal behavior! It maybe surprising if you are new to Observable instruments and the concept of temporalities!

meter.CreateObservableGauge("aNewMetric", () => 1); registed an observableGauge that reports "1" whenever invoked. ObservableGauge is invoked during every export, i.e every 60 secs by default.
The exporter is invoked every 60 secs, and that triggers ObservableGauge callback where you return 1, which gets exported.

You can of course modify ObservableGauge to return nothing instead of 1, based on condition. https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.Instrumentation.Runtime/RuntimeMetrics.cs#L117-L120 shows an example.

However, OTLP/Console Exporter still exports "1" every 60 seconds!! This is the expected behavior of Cumulative temporality, which is the default in OTel. You can change temporality to Delta, by using metricReaderOptions.Temporality=Delta and that will ensure exporting clears the prior state, and nothing is exported, unless reported.

It is bit surprising that we don't have much docs on Temporality. Part of the reason is - what temporality one should use is usually dictated by the Metrics backend! If the metrics is stored in prometheus, you'll need cumulative. If using vendor specific backends like Microsoft Azure, NewRelic, they need delta.

[Yogesh19921]

I tried setting metricReaderOptions.Temporality=Delta, but it still didn't work. What ended up working though is setting metricReaderOptions.Temporality=Delta and switching to using Histogram instead of ObservableGauge. Would have been nice to have this documentation though.

[cijothomas]

I tried setting metricReaderOptions.Temporality=Delta, but it still didn't work. What ended up working though is setting metricReaderOptions.Temporality=Delta and switching to using Histogram instead of ObservableGauge. Would have been nice to have this documentation though.

Switching to Delta alone won't work for ObservableInstruments. You need to modify the callback also to return empty measurement conditionally! I shared this example as a reference :
https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.Instrumentation.Runtime/RuntimeMetrics.cs#L117-L120

Totally agree that the docs are relatively less on these aspects. https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics#example does show some examples to help understanding, which should be useful reading.

[cijothomas]

Closing this as the question is resolved.
Possible doc improvements need to tracked via a separate, dedicated issue.

[cijothomas]

#4417 can be used to track.