Warning of missing Cluster Roles for Target Allocator despite them being present

[avillela]

Component(s)

target allocator

What happened?

Description

Getting a warning after applying the OpenTelemetryCollector CR with TargetAllocator enabled, in spite of having created the appropriate ServiceAccount, ClusterRole, and ClusterRoleBinding for the TA based on the documentation here. See warning message below.

Steps to Reproduce

Create ServiceAccount, RoleBinding, and ClusterRoleBinding as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: opentelemetry-targetallocator-sa
  namespace: opentelemetry
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-targetallocator-role
rules:
- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  - podmonitors
  verbs:
  - '*'
- apiGroups: [""]
  resources:
  - namespaces
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs: ["get", "list", "watch"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: opentelemetry-targetallocator-rb
subjects:
  - kind: ServiceAccount
    name: opentelemetry-targetallocator-sa
    namespace: opentelemetry
roleRef:
  kind: ClusterRole
  name: opentelemetry-targetallocator-role
  apiGroup: rbac.authorization.k8s.io

Create the following OpenTelemetryCollector CR using the ServiceAccount defined above:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: opentelemetry
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-targetallocator-sa
    prometheusCR:
      enabled: true
      serviceMonitorSelector:
        app: my-app
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 30s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]
        target_allocator:
          endpoint: http://otelcol-targetallocator.opentelemetry.svc.cluster.local
          interval: 30s
          collector_id: "${POD_NAME}"

    exporters:
      logging:
        verbosity: detailed

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: []
          exporters: [logging]

Expected Result

I should not be seeing the warning below.

Actual Result

The following warning comes up after I apply the OpenTelemetryCollector CR. Note that this is non-blocking - my code is working; however, the warning is misleading.

Warning: missing the following rules for namespaces: [get,list,watch]
Warning: missing the following rules for networking.k8s.io/ingresses: [get,list,watch]
Warning: missing the following rules for nonResourceURL: /metrics: [get]
Warning: missing the following rules for monitoring.coreos.com/podmonitors: [*]
Warning: missing the following rules for nodes: [get,list,watch]
Warning: missing the following rules for services: [get,list,watch]
Warning: missing the following rules for endpoints: [get,list,watch]
Warning: missing the following rules for pods: [get,list,watch]
Warning: missing the following rules for configmaps: [get]
Warning: missing the following rules for discovery.k8s.io/endpointslices: [get,list,watch]
Warning: missing the following rules for monitoring.coreos.com/servicemonitors: [*]
Warning: missing the following rules for nodes/metrics: [get,list,watch]
opentelemetrycollector.opentelemetry.io/otelcol created

Kubernetes Version

1.27.3

Operator version

0.94.0

Collector version

0.94.0

Environment information

Environment

OS: MacOS Sonoma 14.1.1 (also got the same result in GH codespaces)
Compiler(if manually compiled): N/A

Log output

No response

Additional context

No response

[seb-835]

Hi, i got same behaviour.
Kubernetes Version 1.24.17
Operator version : 0.97.1
Collector version :0.97.0

Is it something to worry about for the scrapping ?

[jaronoff97]

@yuriolisa are you able to look at this next week?

[yuriolisa]

Hi @jaronoff97, I will look at this today.

[andrewdinunzio]

I'm seeing these warnings with version 0.107.0, despite seemingly having the proper permissions. Will try to collect more info. My current theory is that the operator creates a service account (name ending in -collector) for the given OpenTelemetryCollector, and when I specify a different one to use, it still looks for permissions on the one it creates, instead of the one I assigned to it.

Strangely, I only get warnings on the otel instance that has target allocator enabled. I have other instances for non-metrics telemetry, and even though they also seem to have an extra service account created by the operator (not in use), they don't get the same warnings.