Add skip-reconcile annotation support for controlled operator pause during migrations

[Kaushik-Vijayakumar-1]

Problem Statement

When migrating Redis clusters between Kubernetes clusters, the operator's reconciliation loop can interfere with the migration process by attempting to maintain the desired state (sentinel count, slave count) in the source cluster. This can potentially lead to split-brain scenarios during migrations.

Proposed Solution

Add support for a skip-reconcile annotation that allows operators to temporarily pause reconciliation for specific Redis Failover resources. This gives operators more control during maintenance windows and migrations.

Example usage:

apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: redisfailover-sample
  annotations:
    skip-reconcile: "true"
spec:
  # ... rest of the spec

Benefits

• Provides fine-grained control over operator reconciliation at the resource level
• Facilitates safer maintenance operations and migrations
• Prevents potential split-brain scenarios during cluster migrations
• Allows for temporary pause without needing to delete/modify the operator

Implementation Details

The implementation checks for the annotation at the start of the reconciliation loop:

if rf.Annotations != nil {
    skipReconcile, ok := rf.Annotations["skip-reconcile"]
    if ok && skipReconcile == "true" {
        r.logger.Infoln("skip-reconcile set to true. Skipping reconcile for", rf.Name)
        return nil
    }
}

[github-actions]

This issue is stale because it has been open for 45 days with no activity.