ArgoCD: Fluentd helm chart deployment error due to deprecated HorizontalPodAutoScaler api

[ricsanfre]

ArgoCD not able to get status of logging application with the following error:

HPA is configured only when deploying fluentd. The manifest used by fluentd helm chart (hpa.yml) is using a deprecated version of autscaling API (autoscaling/v2) since v1.23+

kubectl describe HorizontalPodAutoscaler logging-fluentd -n logging
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler

kubectl get HorizontalPodAutoscaler logging-fluentd -o yaml -n logging

Even when original manifest file is using a deprecated version (autoscaling/v2beta2), kubernetes is applying the highest API version available for autoscaling which is autoscaling/v2.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  creationTimestamp: "2023-01-14T09:09:03Z"
  labels:
    app.kubernetes.io/instance: logging
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluentd
    app.kubernetes.io/version: v1.14.6
    argocd.argoproj.io/instance: logging
    helm.sh/chart: fluentd-0.3.9
  name: logging-fluentd
  namespace: logging
  resourceVersion: "14605"
  uid: 8b32c4f6-261b-4cfd-bced-f491eff04d75
spec:
  maxReplicas: 100
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: logging-fluentd

It seems that ArgoCD is not able to calculate the differences between what is really applied to the cluster (using autoscaling/v2 api) and the original manifest file (using autoscaling/v2beta2)

[ricsanfre]

Workaround

1. Disable HPA resource created by fluentd helm chart

   Include in values.yml:

     autoscaling:
       enabled: false

2. Define HPA resource using autoscaling v2 API

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     labels:
       app.kubernetes.io/instance: logging
       app.kubernetes.io/name: fluentd
     name: logging-fluentd
     namespace: {{ .Release.Namespace }}
   spec:
     maxReplicas: 100
     metrics:
     - resource:
         name: cpu
         target:
           averageUtilization: 80
           type: Utilization
       type: Resource
     minReplicas: 1
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: logging-fluentd

New out-of-sync issue after applying the workaround

Loki's Statefulset resources (loki-read, loki-write) remains as Ouf-of-sync

Issue already tracked in argoproj/argo-cd#11143

[ricsanfre]

There is an open issue in fluentd chart to solve initial issue of deprecated autoscaling API. fluent/helm-charts#345

[ricsanfre]

ArgoCD sync initial issue with Loki's statefulset resources has been solved. Loki's helm chart has been updated adding missing default keys in Statefulset manifest files: See grafana/loki#10452

[ricsanfre]

Keeping this issue for tracking fluentd helm chart issue

[ricsanfre]

Original fluend helm chart issue fixed with release 0.5.2