Own-telemetry: Ability to enabled/disable individual metrics

[djaglowski]

Is your feature request related to a problem? Please describe.

Our notion of basic/normal/detailed telemetry levels is a quick and convenient way for users to tune the volume of the collector's own telemetry, but it is quite crude compared to what we provide to control telemetry not generated by the collector. For example, out mdatagen library provides the ability to enable or disable any individual metric. Not having this ability for the collector's own telemetry means that users may not be able to get the metrics they need without enabling all metrics.

Describe the solution you'd like

In the telemetry configuration, we should offer the ability to enable or disable individual metrics, regardless of telemetry level.

The telemetry level should be used as the starting point for which metrics are enabled or disabled, but then individual metric settings can override.

[codeboten]

Thanks for opening the issue. One question I have is how this works w/ views, because configuration of views would allow this same functionality. I'd rather not have multiple ways of doing the same thing if possible

[djaglowski]

I'm not familiar enough with views to give a good answer here. Are they exposed to our users via the collector config? Or are they an implementation detail? For that matter, is there a reason we don't use views instead of in-code checks against metric level?

[codeboten]

@djaglowski views will be configurable in the same way as they are for SDK via config, see https://github.com/open-telemetry/opentelemetry-configuration/blob/cc7cd375d893e440afdc294a00f98038ebaa4eab/examples/kitchen-sink.yaml#L221 as an example

The main reason I can see wanting to keep a telemetry level configuration option is for convenience to end users. It would be easier to configure one setting than to have to configure views for all metrics. But then, if we have multiple ways to configured things, we get into the issue of precedence.

[djaglowski]

The main reason I can see wanting to keep a telemetry level configuration option is for convenience to end users.

I agree but couldn't these be implemented as views? (Then for example component developers don't have to check if telemetrySettings.Level == Detailed, etc.)

If we did that, does that normalize configuration in a way where users could add or subtract from our predefined views by adding their own?

[codeboten]

I agree but couldn't these be implemented as views? (Then for example component developers don't have to check if telemetrySettings.Level == Detailed, etc.)

There's no concept of levels built into views. I considered adding a level in mdatagen block for each metric configured via telemetry::metrics to allow the code to be automatically generated and remove the need to check in the components themselves. Otherwise we'd need to provide a mechanism for components to return views based on different levels.

An alternative proposed was to have a different MeterProvider for everything level which would again remove the need to check a level before recording telemetry. This would mean that the component developer would have to make a decision on what they consider appropriate for the different levels.

If we did that, does that normalize configuration in a way where users could add or subtract from our predefined views by adding their own?

We could definitely publish a configuration for each level that users could modify as they see fit. This would be somewhat easy to generate via mdatagen in the documentation as well

[crobert-1]

I hope I'm not getting sidetracked from the original purpose of this issue, but is this line in the mdatagen metadata schema the same thing as what Dan's proposing here? I'm pretty new to this area of the collector so I may be missing something obvious, but the schema appears to state that this should be supported.

Please let me know if I'm missing something.

[djaglowski]

Yes, that's basically what I suggested @crobert-1. I don't care if it looks just like that or not, but it should be possible to enable or disable individual metrics produced by the collector.

[ptodev]

Hello! 👋 I could take a look into implementing this, but I'm still not yet sure what a good solution would look like.

One solution is to use views to disable (but not enable) metrics using views. The user defined views would override the settings configured by the level config argument so that the Collector emits less metrics than it normally would:

service:
  telemetry:
    metrics:
      level: basic
      views:
      - selector:
          instrument_name: otelcol_exporter_send_failed_spans
        stream:
          aggregation:
            drop:

Another solution could be to make the service/telemetry/metrics/level and service/telemetry/metrics/views config arguments mutually exclusive, and to let users create a view which drops specific metrics. I suppose this is similar to @codeboten's suggestion above:

We could definitely publish a configuration for each level that users could modify as they see fit.

However, I'm not sure how ergonomic both of these approaches are. I suppose most users want to specify the metrics they are interested in, not the ones they are not interested in. That's because if you have a dashboard which uses a particular metric, you want to make sure the metric is always present.

[mx-psi]

Re-reading this I am not sure what the decision if any was. Do we want to have a separate mechanism for configuring inidividual metrics different from views or service::telemetry::metrics?