Annotations Filter Added in v0.96.0 does not filter annotations

[ryanohnemus]

Component(s)

collector

What happened?

I am hoping this is possibly just user error, but I attempted to use the annotations-filter added in the v0.96.0 release.
I tested this out using the latest v0.96.0 image and the annotations aren't being filtered out as I'd expect.

I setup a kind cluster using the kind-1.29.yaml included in the repo:

# setup kind
kind create cluster --config kind-1.29.yaml

# add required cert-manager install
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

# Download the latest operator install
curl -Lo otel-op.yaml https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

I then edited the otel-op.yaml to add the --annotations-filter="config.*.gke.io.*" arg below for the operator's Deployment:

kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: opentelemetry-operator
    control-plane: controller-manager
  name: opentelemetry-operator-controller-manager
  namespace: opentelemetry-operator-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetry-operator
      control-plane: controller-manager
  template:
    metadata:
      labels:
        app.kubernetes.io/name: opentelemetry-operator
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --zap-log-level=info
        - --zap-time-encoding=rfc3339nano
        - --feature-gates=+operator.autoinstrumentation.go,+operator.autoinstrumentation.nginx
        - --annotations-filter="config.*.gke.io.*"
        image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.96.0

 # Applied operator to the cluster and confirmed the operator pod args contained the annotations-filter above: 
 kubectl apply -f otel-op.yaml

Finally I added an OpenTelemetryCollector resource that has an annotation configmanagement.gke.io/token I was expecting filtered out:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
  annotations:
    configmanagement.gke.io/token: "asdfasdf"
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [debug]

The configmanagement.gke.io/token: asdfasdf is still added to the template for the deployment of the collector and is still added to the pod.

Kubernetes Version

1.29.0

Operator version

v0.96.0

Collector version

v0.96.0

Environment information

Environment

kind cluster 1.29 config included with repo

Log output

No response

Additional context

No response

[yuriolisa]

@ryanohnemus, can you please try to use the filter parameter like config.*.gke.io ?

[ryanohnemus]

@yuriolisa, i've tried both --annotations-filter="config.*.gke.io.*" and --annotations-filter="config.*.gke.io" and neither are removing the annotation.

There's also a separate issue if multiple annotations filters are supplied where only the first is checked, which i believe is from here, this should only return match if it matched. Otherwise it returns false when there's not a match but it hasn't checked all filters. That's unfortunately as far as my debugging took me :(.

[yuriolisa]

I've just checked and there is a real bug regarding the multiple filters not being checked. I will submit a fix this week.

[ryanohnemus]

Thanks @yuriolisa, are you able to reproduce the original issue for this ticket? I have not been able to run the operator even with a single annotation filter and have those annotations filtered on the collector deployment that is created.

[yuriolisa]

Using a single annotation, I couldn't reproduce the error you mentioned. However, using multiple annotations or labels on the filter parameters, I found a bug there. Therefore, I decided to move from regexp.MatchString to Strings.Contains, which will be simpler to use.

[ryanohnemus]

@jaronoff97 @yuriolisa - this bug still persists in the latest release (0.101.0) even after #2923

I know the test exists here passes: https://github.com/open-telemetry/opentelemetry-operator/blob/main/internal/manifests/manifestutils/annotations_test.go#L162

but when running this in a cluster as described in this ticket, the annotations are still applied to the template and added to the Deployment object for the opentelemetry so the annotations-filter doesn't actually filter anything. Can this be re-opened?

[jaronoff97]

@ryanohnemus Could you open a new issue following our issue template? It would be helpful to know things like your operator version, CRD versions, operator logs, etc. and I would rather capture that in a new issue.