[Metrics API] Ability to record a metric with a custom timestamp

[ThePlenkov]

What are you trying to achieve?

I'd like to be able to develop so-called proxy layers with SDK, where I'd like to register metrics such as gauge and histogram providing custom timestamps. Currently timestamp defaults to the current time such as Date.now( ) in JavaScript, but in case if we provide metrics exposed from alternative sources ( not with SDK ) - we're not able to proxy those metrics correctly because of the different timestamp.

This issue can be easily implemented, but will make SDK much more flexible in terms of registering metrics in a right moment.

So the proposal is:

• method add (counters) should contain optional parameter timestamp which is default to a current time
• method record (gauge,histogram) should contain optional parameter timestamp which is default to a current time

Additional context.

In our ABAP runtime (internal SAP language) we do not have OTel SDK available out-of-the box. For that purpose we created own SDK ( https://github.com/abapify/otel/ ) and set of exporters ( such as MQTT ) which are streaming telemetry events in a ABAP compatible format ( not OTLP, own format chosen by set of reasons explained here ). So as intermediate layer - we created a NodeJS service which is not generating, but streaming/proxying OpenTelemetry events using JS SDK.

Because of that our charts attract metrics according to how they were processed by proxy service but as they were actually recorded. Currently we face some shift, but there could be more complicated patterns when service is temporary down - metrics will be shown completely wrong..

Unfortunately there are few problem points here. This is one of the problems we do not have a solution for yet.

[trask]

hi @ThePlenkov! have you looked at writing a metric producer to solve your needs?

bridges to third-party metric sources

[ThePlenkov]

hi @ThePlenkov! have you looked at writing a metric producer to solve your needs?

That's exactly what I've been advised here

Going to try it today

[ThePlenkov]

Here is our solution:

• we parse metric message ( it's not OTLP, own format ) and create a metric using metric SDK
• when we record/add a data point - we also provide a technical attribute TS containing original timestamp
• During inititalisation we created a class inherited from PeriodicExportingMetricReader and redefined collect method
• In redefined method we first collect resource metrics from super.collect( ) and then replace start time from attribute TS
• To avoid conflicts we also update end time with keeping same difference as it was before comparing to the initial start time
• Finally we delete TS attribute from the resource metric

So as you can see we managed to solve a problem , but the solution still looks a bit "hacky", but indeed much less risky than redefining _record method of the metric.

Unfortunately I didn't find the solution how we could just implement the whole MetricProducer in a standalone class from the scratch without extending MetricReader. We would prefer still to construct resource metrics with SDK rather than define it manually.

Having the option to provide metric timestamps when recording them would help to avoid even doing this and would help to construct resource metrics with SDK.