-
Notifications
You must be signed in to change notification settings - Fork 880
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ProgressDeadlineExceeded: ReplicaSet has timed out progressing #3457
Comments
We have seen this happen with blue green strategy (no analysis templates were used).
It seems like the informer cache is not updated soon enough. |
We are also seeing this with a blue/green Rollout with AnalysisTemplate set. Restarting resolves the issue, but it resurfaces at irregular intervals. I have the slight assumption that this always happens when the Rollout did progress to all pods being ready fine at some point, but then some pods go killed (due to being evicted or due to an imminent node shutdown -- we are using spot nodes) and that killing may have happened after the progress deadline duration of the Rollout. |
We have no AnalysisTemplates
In our case, deleting all old replicasets has fixed this status, starting from the oldest. Perhaps only the most recent was needed, but wanted to be safe. |
Checklist:
Describe the bug
We have a number of rollouts that have analysist templates attached. The rollout shows that the replicas set has timed out but the replicas set is showing that it's fine. Restarting the replicas set seems to fix the problem. We also used to have 5 revisionHistoryLimit and found that removing one of the unused replicasets would sometimes get the rollout to see that everything was fine.
To Reproduce
Expected behavior
Rollout to reflect the replicaset's current state
Screenshots
From rollout:
From Replicaset
Version
image: 'quay.io/argoproj/argo-rollouts:v1.6.6'
Logs
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
The text was updated successfully, but these errors were encountered: