Application stuck in Progressing status

[burner1024]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Deployment stuck in Progressing state, with "HEALTH DETAILS Waiting for rollout to finish: 0 of 2 updated replicas are available..."

$ k get deployments.apps 
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
karpenter-root   2/2     2            2           54m

$ k get deployments.apps -o yaml | grep -i generation
    generation: 1
    observedGeneration: 1

$ k get pods
NAME                              READY   STATUS    RESTARTS   AGE
karpenter-root-648b85ffcc-qbn7f   1/1     Running   0          54m
karpenter-root-648b85ffcc-zv5gd   1/1     Running   0          54m

$ k get rs
NAME                        DESIRED   CURRENT   READY   AGE
karpenter-root-648b85ffcc   2         2         2       55m

$ k rollout status deployment
deployment "karpenter-root" successfully rolled out

To Reproduce

Install Karpenter.

Expected behavior

Deployment not stuck.

Screenshots

Version

argocd version
argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.9.0+0e67ed8

Logs

$ k get ev
No resources found in karpenter namespace.

 k -n argocd get ev
LAST SEEN   TYPE     REASON               OBJECT                       MESSAGE
58m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
55m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
52m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
49m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
47m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
46m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
43m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
40m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
37m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
34m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
31m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
28m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
26m         Normal   OperationStarted     application/karpenter-root   admin initiated partial sync to HEAD (95a4851ecb58f94419d19d01fea8f095ccc8490a)
26m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
26m         Normal   OperationCompleted   application/karpenter-root   Partial sync operation to 95a4851ecb58f94419d19d01fea8f095ccc8490a succeeded
26m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
26m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
25m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
22m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
19m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
16m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
13m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
10m         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
7m24s       Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
4m24s       Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy
84s         Normal   ResourceUpdated      application/karpenter-root   Updated health status: Progressing -> Healthy

[via-justa]

If you deploy to https://kubernetes.default.svc
it may be related to #17016
The workaround solved it for me.

[burner1024]

My argo is 2.9 and there's a dozen of other applications that don't exibit this issue, so doesn't look that way.

[burner1024]

status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2024-02-14T12:52:24Z"
    lastUpdateTime: "2024-02-14T12:52:24Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-02-14T12:52:13Z"
    lastUpdateTime: "2024-02-14T12:52:24Z"
    message: ReplicaSet "karpenter-root-648b85ffcc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

I kinda suspect it's due to order of conditions.
In other, "healthy" deployments in the cluster, I see "Progressing" as the first item, and "Available" second.
No idea how to rearrange them though, seems to be managed by k8s itself, changes just get reverted.
And re-creating keeps the same order.

[burner1024]

With upgrade to 2.11.0, it's now stuck in Degraded instead of Progressing.

...
26m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
26m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
23m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
21m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
20m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
17m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
16m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
14m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
11m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
11m         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
8m36s       Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
5m58s       Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
5m36s       Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
2m36s       Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy
52s         Normal   ResourceUpdated   application/karpenter-root   Updated health status: Degraded -> Healthy