Add labels for Redis-cluster

[sondrelg]

Is your feature request related to a problem? Please describe.

I would like to prevent co-hosting of leader/follower pairs for a redis-cluster. I don't mind if two followers are hosted on the same node, as long as:

• No leaders are hosted on the same node
• leader-n isn't hosted on the same node as follower-n

So for example, I imagine this would be fine:

node1	node2	node3	node4	node5
leader-1	leader-2	leader-3	-	-
follower-2	follower-3	-	-	-
-	follower-1	-	-	-

Describe the solution you'd like

✔️ Preventing co-hosting of leaders is already fine. We can use the role label.

Preventing co-hosting of pairs is missing (I believe - please let me know if you there's a way I haven't thought of). I think this could be achieved using podAntiAffinity rules if the pod labels were expanded so that follower-1 shared a label with leader-1. For example:

metadata:
  labels:
    pair: 1  # <-- this would be the new label and should contain the pod count that's used in the name
    redis_setup_type: cluster
    role: follower
    statefulset.kubernetes.io/pod-name: follower-1

Describe alternatives you've considered
I haven't really considered anything else. Very open to alternative suggestions 🙇

What version of redis-operator are you using?

0.14.0

Context

The reason this matters is that this solution would mean I need a minimum of 3 nodes in my k8s cluster to host the redis-cluster, rather than 6.

[sondrelg]

After a bit of thinking, I came up with this as a solution for my specific case:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      # Avoid leaders being scheduled with other leaders
      - labelSelector:
          matchExpressions:
            - key: "role"
              operator: In
              values:
                - leader
        topologyKey: "kubernetes.io/hostname"
      # Prevent leader-1 from being co-hosted with follower-1
      - labelSelector:
          matchExpressions:
            - key: "statefulset.kubernetes.io/pod-name"
              operator: In
              values:
                - project-name-follower-1
                - project-name-leader-1
        # Applied to the node-level
        topologyKey: "kubernetes.io/hostname"
      # Prevent leader-2 from being co-hosted with follower-2
      - labelSelector:
          matchExpressions:
            - key: "statefulset.kubernetes.io/pod-name"
              operator: In
              values:
                - project-name-follower-2
                - project-name-leader-2
        topologyKey: "kubernetes.io/hostname"
      # Prevent leader-3 from being co-hosted with follower-3            
      - labelSelector:
          matchExpressions:
            - key: "statefulset.kubernetes.io/pod-name"
              operator: In
              values:
                - project-name-follower-3
                - project-name-leader-3
        topologyKey: "kubernetes.io/hostname"

Does that seem reasonable? If there's no clear value add from adding extra labels, then please feel free to just close this - I think I'll be able to achieve what I wanted 👍

[shubham-cmyk]

Have to tried this :
https://github.com/OT-CONTAINER-KIT/redis-operator/blob/master/example/affinity/clusterd.yaml#L19.

By default, we provide the label :

 redis-cluster-leader
 redis-cluster-follower

On the leader and follower respectively.
The leader and follower could be placed separately using the pod anti affinity.

[sondrelg]

The difference is requiring at least 6 nodes to run a 3 leader/3 follower cluster, vs. needing just 3. I don't mind if my leader-2 and follower-1 are on the same node 🙂

[shubham-cmyk]

For that purpose you need to place the pod-Antiaffinity for the leader pods.
You can ignore the label of the redis-cluster-follower
I think this would solve the problem.

@sondrelg

[zposloncec]

@sondrelg did you manage to solve this? I'm having the same issue and unable to split followers from the leaders.

[zposloncec]

For that purpose you need to place the pod-Antiaffinity for the leader pods. You can ignore the label of the redis-cluster-follower I think this would solve the problem.

@sondrelg

That will allow that leader and follower are running on the same node, but won't split pods with the same index to different nodes

[shubham-cmyk]

It could you have to check the pod-Anti affinity :
https://github.com/OT-CONTAINER-KIT/redis-operator/blob/master/example/v1beta2/affinity/clusterd.yaml

[sondrelg]

Yes @zposloncec, I think we did. This is our setup:

    # Prevent leader from being co-hosted on the same node
    # as another leader or its follower
    leader:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: 'kubernetes.io/arch'
                    operator: In
                    values:
                      - 'amd64'

        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            # Avoid leaders being scheduled with other leaders
            - labelSelector:
                matchExpressions:
                  - key: 'role'
                    operator: In
                    values:
                      - leader
              topologyKey: 'kubernetes.io/hostname'
            # Prevent leader-1 from being co-hosted with follower-1
            - labelSelector:
                matchExpressions:
                  - key: 'statefulset.kubernetes.io/pod-name'
                    operator: In
                    values:
                      - <name-of-our-deployment>-follower-0
                      - <name-of-our-deployment>-leader-0
              # Applied to the node-level
              topologyKey: 'kubernetes.io/hostname'
            # Prevent leader-2 from being co-hosted with follower-2
            - labelSelector:
                matchExpressions:
                  - key: 'statefulset.kubernetes.io/pod-name'
                    operator: In
                    values:
                      - <name-of-our-deployment>-follower-1
                      - <name-of-our-deployment>-leader-1
              topologyKey: 'kubernetes.io/hostname'
            # Prevent leader-3 from being co-hosted with follower-3
            - labelSelector:
                matchExpressions:
                  - key: 'statefulset.kubernetes.io/pod-name'
                    operator: In
                    values:
                      - <name-of-our-deployment>-follower-2
                      - <name-of-our-deployment>-leader-2
              topologyKey: 'kubernetes.io/hostname'

    follower:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: 'kubernetes.io/arch'
                    operator: In
                    values:
                      - 'amd64'

[xiaozhuang-a]

I think this feature can be implemented based on webhooks

[xiaozhuang-a]

#1174