Operator is constantly trying to update OpensearchUser

[evheniyt]

After we had created users with OpensearchUser we noticed that operator is constantly trying to update all created users even when there were no changes.
In operator logs we could see:

operator-controller-manager {"level":"info","ts":"2023-10-10T15:18:07.874Z","msg":"Reconciling OpensearchUser","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUs
er","OpensearchUser":{"name":"es-consumer","namespace":"opensearch"},"namespace":"opensearch","name":"es-consumer","reconcileID":"4cd41a78-4a6a-4e3e-9ac4-103e0e08980a","user":{"name":"es-consumer","namespace":"op
ensearch"}}                                                                                                                                                                                                         
operator-controller-manager {"level":"info","ts":"2023-10-10T15:18:08.227Z","msg":"user requires update","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUser","O
pensearchUser":{"name":"es-consumer","namespace":"opensearch"},"namespace":"opensearch","name":"es-consumer","reconcileID":"4cd41a78-4a6a-4e3e-9ac4-103e0e08980a","os_service":"security"}

In OpensearchUser events we could see that OpensearchAPIUpdated counter is constantly increasing:

  Normal  OpensearchAPIUpdated  105s (x26 over 7m5s)  user-controller  user updated in opensearch                                                                                                                   

Unfortunately, from the code it's not clear why exactly that happens, looks like this reflect.DeepEqual(user, userResponse[username]) could be the root cause. I think to understand what is not equal between the current user and the response that operator gets there should be some additional logs (e.g. what exactly is different)

Opensearch version 2.3.0
Operator version 2.4.0

[IIIRADIII]

Hi @evheniyt .
We will take a look at it, can you provide the configuration you used to create cluster and user, so that we could try and reproduce it?

[evheniyt]

@IIIRADIII very basic configuration for cluster and user

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  annotations:
    meta.helm.sh/release-name: test
    meta.helm.sh/release-namespace: opensearch-test
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 2.7.0
    helm.sh/chart: test-opensearch-1.0.0
  name: test-opensearch
  namespace: opensearch-test
spec:
  bootstrap:
    affinity: {}
    resources: {}
  confMgmt: {}
  dashboards:
    affinity: {}
    enable: true
    image: docker.io/opensearchproject/opensearch-dashboards:2.7.0
    imagePullPolicy: IfNotPresent
    opensearchCredentialsSecret: {}
    podSecurityContext: {}
    replicas: 1
    resources: {}
    securityContext: {}
    service:
      type: ClusterIP
    tls:
      caSecret: {}
      generate: true
      secret: {}
    version: 2.7.0
  general:
    drainDataNodes: true
    httpPort: 9200
    image: docker.io/opensearchproject/opensearch:2.7.0
    imagePullPolicy: IfNotPresent
    monitoring:
      scrapeInterval: 30s
      tlsConfig: {}
    podSecurityContext: {}
    securityContext: {}
    serviceName: test-opensearch
    setVMMaxMapCount: true
    vendor: Opensearch
    version: 2.7.0
  initHelper:
    imagePullPolicy: IfNotPresent
    resources: {}
    version: 1.27.2-buildx
  nodePools:
  - component: masters
    diskSize: 30Gi
    replicas: 3
    resources:
      limits:
        cpu: 500m
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 2Gi
    roles:
    - master
    - data
  security:
    config:
      adminCredentialsSecret: {}
      adminSecret: {}
      securityConfigSecret: {}
    tls:
      http:
        caSecret: {}
        generate: true
        secret: {}
      transport:
        caSecret: {}
        generate: true
        perNode: true
        secret: {}

apiVersion: opensearch.opster.io/v1
kind: OpensearchUser
metadata:
  name: test-user
  namespace: opensearch-test
spec:
  opensearchCluster:
    name: test-opensearch
  passwordFrom:
    key: test
    name: users-credentials

[oleksii-kalinin]

Same here with opensearchroles.opensearch.opster.io

apiVersion: opensearch.opster.io/v1
kind: OpensearchRole
metadata:
  name: dashboards-role
spec:
  opensearchCluster:
    name: opensearch-cluster
  clusterPermissions:
  - cluster_monitor
  - cluster_composite_ops
  - manage_point_in_time
  - indices:admin/template*
  - indices:admin/index_template*
  - indices:data/read/scroll*
  indexPermissions:
  - indexPatterns:
    - ".kibana"
    - ".opensearch_dashboards"
    allowedActions:
    - indices_all
  - indexPatterns:
    - ".kibana-6"
    - ".opensearch_dashboards-6"
    allowedActions:
    - indices_all
  - indexPatterns:
    - ".kibana_*"
    - ".opensearch_dashboards_*"
    allowedActions:
    - indices_all
  - indexPatterns:
    - ".tasks"
    allowedActions:
    - indices_all
  - indexPatterns:
    - ".management-beats*"
    allowedActions:
    - indices_all
  - indexPatterns:
    - "*"
    allowedActions:
    - indices:admin/aliases*

Events:
  Type    Reason                Age                  From             Message
  ----    ------                ----                 ----             -------
  Normal  OpensearchAPIUpdated  9s (x12 over 5m14s)  role-controller  role updated in opensearch

[evheniyt]

Looks like the problem is in different types of OpendistroSecurityRoles and BackendRoles
I have debugged it with "github.com/go-test/deep" and it returns this:

[OpendistroSecurityRoles: <nil slice> != [] BackendRoles: <nil slice> != []]{Password: OpendistroSecurityRoles:[] BackendRoles:[] Attributes:map[k8s-uid:811abea9-daea-48db-9c53-4c6771fca5d6]}

Here is the output for user and userResponse[username] that are being used to compare in reflect.DeepEqual(user, userResponse[username]) here

user: requests.User{Password:"", OpendistroSecurityRoles:[]string(nil), BackendRoles:[]string(nil), Attributes:map[string]string{"k8s-uid":"0e3b50e9-6869-4928-be2b-3e92aee0071b"}}                         
userResponse[username]: requests.User{Password:"", OpendistroSecurityRoles:[]string{}, BackendRoles:[]string{}, Attributes:map[string]string{"k8s-uid":"0e3b50e9-6869-4928-be2b-3e92aee0071b"}}

Because nil slice is not the same as [] slice reflect.DeepEqual(user, userResponse[username]) returns false