Remove stale gauges

[rikonor]

Hi!

I have the following example that exports a set of gauges.

In some cases, my gauges become stale (i.e the thing they track no longer exists), in which case, I don't wish for them to be exported anymore.

Failing to do this just makes the gauge retain it's value forever, which makes my metric aggregations wrong.
Is there a way to remove these stale metrics from my meter or registry?

[TommyCpp]

You can set with_memory(false) so that the metrics provider will not remember stale gauges. #946 will remove this option and forget in all cases

[rikonor]

Hi @TommyCpp, thanks for your reply!

Can you elaborate a bit? with_memory(...) says the following:

Memory controls whether the processor remembers metric instruments and attribute sets that were previously reported.
When Memory is true, Reader::try_for_each will visit metrics that were not updated in the most recent interval.

Does this mean that there's some internally defined interval after which a metric expires and is removed from the registry? If so, what is that interval?

Also, what if only want to drop a specific metric? with_memory(...) sounds like it makes a decision globally.

[TommyCpp]

Sorry was in a hurry this morning so I didn't realize you are using prometheus exporter, which only supports Cumulative temporality. It means the SDK track the state for instruments and aggregate all data points since application starts. with_memory setting will only work with Delta temporality.

Looking at your example, I feel like the instrument you are looking for is Histogram ?

---

Does this mean that there's some internally defined interval after which a metric expires

For prometheus use case, where the exporter is pull based. The interval will be the time between two consecutive collect call.

Also, what if only want to drop a specific metric?

That's part of the reason we are dropping with_memory method. We are working on align the implmentation with new spec, which has View to help you tweak and drop metrics from certain instrument.

[rikonor]

Looking at your example, I feel like the instrument you are looking for is Histogram ?

I'm not sure that's correct. I'll try to give a better explanation of our use case.

We have a service that periodically obtains a listing of nodes and runs a health check on them. Among other metrics, it also exposes a gauge with a unique label for each node indicating 0 (for down) or 1 (for up).

Occasionally, a node that was previously in the listing is removed, so from that point on our service won't do any health checks against that removed node. In that case, we don't want to retain the gauge with the label for that node, since it will just always keep the last value it had (e.g 0).

It would be nice if there was a way to remove that specific instance of the gauge (while retaining all the other ones), but I haven't been able to find a way to do it with the SDK.

It's a bit hacky, but a short-term solution for now would be to manually remove those entries from the exporter's text output.

[TommyCpp]

Yeah, I think you are right. The ideal solution will be unregister support.

I will create an issue to add unregister support. I think there are a few things that are worth discussing before we implement it. Sorry we don't have a better solution for you 😔

Close with feature tracking issue #958.

[ostkrok]

Hi @TommyCpp,

We're also interested in the same behavior as @rikonor.

However, I don't think that unregister as it's currently defined does what we want. It only unregisters the callbacks that update the instruments, but doesn't remove the actually values from the registry.

So I think this issue should be re-opened as #958 won't solve this issue.