v1.33.2 router.yaml OpenTelemetry metric error occurred: Metrics error: Warning: Maximum data points for metric stream exceeded. Entry added to overflow.

[billnbell2]

We are having an error that happens several time a minute.

OpenTelemetry metric error occurred: Metrics error: Warning: Maximum data points for metric stream exceeded. Entry added to overflow.

It does not indicate what is causing it?

Expected behavior
No error in logs for router.

Self-hosted on Kubernetes.

Should I change something in here?

telemetry:
  apollo:
    send_headers: all
    send_variable_values: all
    errors:
      subgraph:
        all:
          send: true
          redact: false
  metrics:
    prometheus:
      enabled: true

[billnbell2]

??

[abernix]

This is getting reported by the OpenTelemetry error handler, but this isn't something that we're familiar with. It'll have to be investigated a bit more (including looking at the source code) to get more information on what this even means.

Any other details you can provide about the environment could be really useful, but do you have any sort of minimal reproduction that you can put together for us?

[BrynCooke]

This is caused by having too many permutations for a particular metric. There is a hard limit of 2000 permutations in otel.

We need to allow users to control which instruments are active and the attributes that are present on them. Part of #3226.