-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove nodeAffinity from typha #9826
remove nodeAffinity from typha #9826
Conversation
Hi @ozdanborne. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I believe the problem lies with the autoscaler's treatment of pod priority, as described in its FAQ. The priority cutoff was zero as recently as version 1.11, but they've since dropped it to -10. I suspect that the Typha pods hadn't been preventing scale downs because their priority was zero. Now that their priority is significantly higher, well above the default -10 cutoff, they prevent the autoscaler from scaling down. The "cluster-autoscaler.kubernetes.io/safe-to-evict" annotation is supposed to allow overriding that policy, so long as the Typha pods don't violate any of the other requirements expressed here. In some circumstances we can't move a Typha pod due its use of host ports, precluding multiple running on the same node. We should be able to move them back onto master machines, though. As for why I had seen these pods keeping worker nodes alive when it appeared the pods could have run elsewhere instead, I can't go back to recover that environment today to investigate further. |
/ok-to-test |
@ozdanborne Thanks for doing this. I agree with the approach. Let's see what happens when this is removed and fix it. Btw, nice job with Calico 3.16. Looking forward to give it a try! |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hakman, ozdanborne The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This PR removes the unnecessary nodeAffinity from the Typha deployment in calico and canal manifests.
Some background / history on how we got to this point:
nodeSelector: {kubernetes.io/role: master}
is added to Canal in Canal v3.10 manifest for k8s v1.15+ #7917nodeSelector
change is "copied" into the Calico manifest in Update Calico to v3.10.2 #8104 .Why it was added in the first place
I believe that the nodeSelector should never have been added in the first place. @tmjd has been asking questions on those threads and as best as we can tell, the node affinity / selectors were added to mitigate an issue where Typha was not being evicted from a node when the node group was being scaled down to zero. As per @seh:
As per @KashifSaadat
I don't disagree that something may have been funky with evictions. But I do disagree with using affinity to avoid it. We should fix that eviction issue instead! Even after pouring through these issues, It's not clear to me what the eviction problem is, though it sounds like we might not be expecting it anymore?