UserOperator creates Secrets with app.kubernetes.io/instance label

[diranged]

Describe the bug
We use ArgoCD as our deployment tool to launch Kafka, KafkaTopics, KafkaUsers, and our other apps. We ran into a situation where ArgoCD kept deleting "Secret" resources that were created by Strimzi. The root cause we discovered was that the Strimzi User Operator is creating Secrets with the app.kubernetes.io/instance label.

Why is this a problem?

If you have an ArgoCD Application named MyApp, it tracks all of the resources that it creates for that application with the standard app.kubernetes.io/* labels - including the app.kubernetes.io/instance label. So each resource it creates will have app.kubernetes.io/instance: MyApp for example as a label. When Argo sees resources that have that label, and they are not in the desired manifest for the MyApp application, it will delete them.

Now consider that we have a Kafka Application as well, and it creates a KafkaUser named MyApp for our application we described above? It turns out that if you do this, Strimzi creates the resource with the label app.kubernetes.io/instance: MyApp because (I think) of this code.

Well what happens at that point is that Argo's Application for MyApp does not have a Secret resource in it with that label.. so it deletes the resource. Then the User Operator goes and re-creates it.. then argo deletes it again.

It seems like a bug that the instance label would be used in this way. If anything, the instance label should refer to the cluster that the KafkaUser is a part of - not match the username itself.

Environment (please complete the following information):

• Strimzi version: 0.24.0
• Installation method: Helm Chart
• Kubernetes cluster: 1.21
• Infrastructure: EKS

[scholzj]

It is hard to call this bug because there is no real specification of how it should behave. And different apps and different users have different requirements and expectations.

[scholzj]

I guess you could implement the same configurable exclusion which we already have in the Cluster Operator also for the USer Operator if you want. That might allow you to make it configurable.

[diranged]

It is hard to call this bug because there is no real specification of how it should behave. And different apps and different users have different requirements and expectations.

If you read https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels, the instance label is defined as:

A unique name identifying the instance of an application

I would argue that is the cluster itself..but I think even more so, this should be a configurable behavior not a hard-coded behavior. If you have a hard-coded need to use labels to track resources that the operator is managing, that should be done through strimzi.io/... labels that are explicitly internal, in my opinion.

[scholzj]

A unique name identifying the instance of an application

I would argue that is the cluster itself..but I think even more so, this should be a configurable behavior not a hard-coded behavior. If you have a hard-coded need to use labels to track resources that the operator is managing, that should be done through strimzi.io/... labels that are explicitly internal, in my opinion.

Instance of what application? The application which created the user? The application which created the Kafka cluster which the user belongs to? The User Operator application which created it? You can argue for any of these. And because these labels are used for many different use-cases including UIs etc., there are conflicting semantics and what does not work for you works fine for someone else.

That is why I suggested that the same thing which was done to the Cluster Operator can be implemented to make it configurable what labels make it through and which don't.

[scholzj]

Triaged on 21.7.2022: There does not seem to be an easy solution because the Kubernetes labels are badly specified and everyone is using them in a slightly different way. We considered removing these labels completely, but that can break it for any users relying on them. So if anyone wants, it might be possible to implement something similar to #4791 to filter the automatically assigned labels.

[see-quick]

I have created a proposal for this issue (strimzi/proposals#60)

[scholzj]

Filtering labels included from the parent custom resource seems to be already possible in the same way as in the Cluster Operator by configuring the exclusion pattern using the STRIMZI_LABELS_EXCLUSION_PATTERN environment variable.