Prometheus fails to scrap Process Instrumentation metrics due to issues with units

[kharabasz]

Issue with OpenTelemetry.Instrumentation.Process

List of all OpenTelemetry NuGet
packages and version that you are
using (e.g. OpenTelemetry 1.3.2):

• OpenTelemetry.Exporter.Prometheus.AspNetCore 1.8.0-beta.1
• OpenTelemetry.Extensions.Hosting 1.8.0-beta.1
• OpenTelemetry.Instrumentation.AspNetCore 1.7.1
• OpenTelemetry.Instrumentation.EventCounters 1.5.1-alpha.1
• OpenTelemetry.Instrumentation.Http 1.7.1
• OpenTelemetry.Instrumentation.Process 0.5.0-beta.4
• OpenTelemetry.Instrumentation.Runtime 1.7.0

Runtime version (e.g. net462, net48, net6.0, net7.0 etc. You can
find this information from the *.csproj file):

• net8.0

Is this a feature request or a bug?

• Feature Request
• Bug

What is the expected behavior?

.AddProcessInstrumentation() coupled with .AddPrometheusExported() should correctly export metrics to Prometheus.

What is the actual behavior?

We receive an exemplar error when Prometheus hits the scrape target:
unit "seconds" not a suffix of metric "process_cpu_time_seconds_total"

Process and Runtime metrics exported in Prometheus format do not meet OpenMetrics specification. The unit used attached in metadata does not match the metric suffix in several cases, causing Prometheus server and OpenTelemetry Collector's Prometheus receiver to throw errors while scraping the target.

This unit was defined in metadata for that metric:

# UNIT process_cpu_time_seconds_total seconds

OpenMetrics specification regarding units: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#unit

Additional Context

OT configuration:

var otel = services.AddOpenTelemetry();

// Add OpenTelemetry Metrics and export to Prometheus.
otel.WithMetrics(meterProvider => meterProvider
    .AddAspNetCoreInstrumentation() // Inbound HTTP connections: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Instrumentation.AspNetCore
    .AddHttpClientInstrumentation() // Outbound HTTP connections: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Exporter.Prometheus.HttpListener
    .AddEventCountersInstrumentation(options =>
    {
        // https://learn.microsoft.com/en-us/dotnet/core/diagnostics/event-counters
        options.AddEventSources("Microsoft.Data.SqlClient.EventSource"); // https://github.com/dotnet/SqlClient/blob/v5.1.5/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlClientEventSource.cs#L73
        options.AddEventSources("Microsoft.EntityFrameworkCore"); // https://github.com/dotnet/efcore/blob/v8.0.3/src/EFCore/Infrastructure/EntityFrameworkEventSource.cs#L45
    })
    .AddMeter("Microsoft.AspNetCore.Diagnostics") // https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-aspnetcore#microsoftaspnetcorediagnostics
    .AddMeter("Microsoft.AspNetCore.Routing") // https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-aspnetcore#microsoftaspnetcorerouting
    .AddMeter("Microsoft.AspNetCore.Server.Kestrel") // https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-aspnetcore#microsoftaspnetcoreserverkestrel
    .AddProcessInstrumentation() // https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Process
    .AddRuntimeInstrumentation() // https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Runtime
    .AddPrometheusExporter());
    
// ...

application.MapPrometheusScrapingEndpoint("/_internal/metrics");

With appsettings.json containing:

  "OTEL_DOTNET_EXPERIMENTAL_ASPNETCORE_ENABLE_GRPC_INSTRUMENTATION": "true"

[Kielek]

Duplicates #1617