fix: Change pod readiness check mechanism #249
Conversation
lukapetrovic-git
changed the title
tasks/change_config.yml
Outdated
| args: | ||
| executable: /bin/bash | ||
| failed_when: "all_pods_ready.rc not in [ 0, 1 ]" | ||
| failed_when: "all_pods_ready.rc != 0" |
There was a problem hiding this comment.
I am not sure why return code 1 was considered ok here, so i made this change, if there is something im not seeing, please comment @MonolithProjects
There was a problem hiding this comment.
It was like that because if grep found no matches (all pods in Ready state), the command would return 1 and the task would fail. But with your approach it's fine to change this. I guess now you can even remove this line.
|
Another question i have regarding this task, why are pods running in kube-system exempt from the check (metadata.namespace!=kube-system)? |
tasks/change_config.yml
Outdated
| args: | ||
| executable: /bin/bash | ||
| failed_when: "all_pods_ready.rc not in [ 0, 1 ]" | ||
| failed_when: "all_pods_ready.rc != 0" |
There was a problem hiding this comment.
It was like that because if grep found no matches (all pods in Ready state), the command would return 1 and the task would fail. But with your approach it's fine to change this. I guess now you can even remove this line.
tasks/rolling_restart.yml
Outdated
| args: | ||
| executable: /bin/bash | ||
| failed_when: "all_pods_ready.rc not in [ 0, 1 ]" | ||
| failed_when: "all_pods_ready.rc != 0" |
There was a problem hiding this comment.
This is not needed anymore
Hmm actually this is something i overlooked. It does not make much sense to me to exclude the pods in this namespace from the check. |
|
Thanks for the review, i made the changes that were requested. I don't work too much with Github, so i'm not sure if i should resolve the changes you requested. Cheers! |
That's fine, i will do it. Thanks! |
Description
If needed i can open an issue for this as well. The following happens:
The check if all pods are ready fails in some of my clusters due to grep catching pods that it is not supposed to, for example:
In the situation above i have pods that as part of their name have init, and the check never passes, so i changed it to check the metadata of the pod itself and figure out its phase https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase.
Type of change
How Has This Been Tested?
Tested on Ubuntu 22.04, RKE2 v1.27.12+rke2r1 on a dev cluster and one production cluster where the problems were happening.