Multiple Datacenters compete over seed service ownership

[burmanm]

What happened?

If there are two datacenters in the same namespace, the seed-service's ownership will be competed and causes infinite loop in the reconciliation process since we keep updating the ownership forever. This does not prevent any real actions from happening, but causes unnecessary writes and logging.

➜  cass-operator git:(master) ✗ kubectl get svc cluster1-seed-service -o yaml | grep name:
    app.kubernetes.io/name: cassandra
  name: cluster1-seed-service
    name: dc2
➜  cass-operator git:(master) ✗ kubectl get svc cluster1-seed-service -o yaml | grep name:
    app.kubernetes.io/name: cassandra
  name: cluster1-seed-service
    name: dc2
➜  cass-operator git:(master) ✗ kubectl get svc cluster1-seed-service -o yaml | grep name:
    app.kubernetes.io/name: cassandra
  name: cluster1-seed-service
    name: dc1
➜  cass-operator git:(master) ✗

➜  cass-operator git:(master) ✗ kubectl get svc cluster1-seed-service -o yaml | grep resourceVersion
  resourceVersion: "8937"
➜  cass-operator git:(master) ✗ kubectl get svc cluster1-seed-service -o yaml | grep resourceVersion
  resourceVersion: "8962"
➜  cass-operator git:(master) 

This has started to happen some time after update to Kubernetes 1.28 in the tests and controller-runtime to newer version, so around 1.19.0.

What did you expect to happen?

No response

How can we reproduce it (as minimally and precisely as possible)?

Run decommission_dc test.

cass-operator version

1.19.0

Kubernetes version

1.28

Method of installation

No response

Anything else we need to know?

No response