HPA Race Condition with Astarte CRD Deployment

[guicrocetti]

Issue Description

When deploying Horizontal Pod Autoscaler (HPA) configurations with Astarte Custom Resources, a race condition occurs where the HPA cannot properly initialize pod metrics in time. This results in the HPA setting pod replicas to 0, creating an undesirable state.

Scenarios Affected

1. When HPA is deployed before Astarte CRDs
2. When HPA is deployed immediately after Astarte CRDs

Current Behavior

• HPA sets pod replicas to 0 when it cannot fetch pod metrics initially
• Creates a race condition between:
  • HPA trying to maintain 0 replicas
  • Operator attempting to increase replica count
• Results in service disruption and unstable pod counts

Technical Details

• Occurs when HPA minimum replicas is set to 1
• HPA status shows both DesiredReplicas and CurrentReplicas as 0
• Metrics state remains unknown during initialization period

Suggested Solution

Implement a protection mechanism in the operator that would:

1. Detect problematic HPA state:

if hpaStatus.DesiredReplicas == 0 && hpaStatus.CurrentReplicas == 0 {
    // Handle edge case
}

2. Take corrective action:

• Delete the problematic HPA
• Force a minimum replica count of 1
• Log an alert for operators to investigate

Expected Benefits

• Prevent service disruption
• Maintain minimum availability
• Provide clear logging for troubleshooting