-
Notifications
You must be signed in to change notification settings - Fork 742
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubelet will start scheduling pods before amazon-vpc-cni-k8s daemon set is fully functional causing workloads to error with "failed to assign an IP address to container" #330
Comments
Related to #282 |
I've added some improved logging and a potential fix in #367. Punting this issue to 1.5 until I can make sure it has been resolved. |
Possible workaround: The only thing I can think of is tainting the node from the very beginning with a NoSchedule action, thus preventing pods from being scheduled. The aws-node(cni plugin) and kube-proxy have tolerations so they will be scheduled regardless. Later on, a few seconds/minutes after the node has joined the cluster and once aws-node is running on the new node they can remove the taint, allowing other pods to be scheduled. To taint the node one can use Kubelet register-with-taints flag. In the EC2 userdata it will look like this:
Later on, to manually remove the taint, use:
One could think of adding a delay in the userdata and removing the taint from there, the problem is the node doesn't have permission to remove taints so removing the taint has to be done manually by the operator. |
Thanks @lutierigb for the workaround. I did little research about the issue. The best possible approach to avoid manual steps would be to include grpc health check at 127.0.0.1:50051 and make changes to manifest file to do live-probe/readiness-probe. @mogren any thoughts ? |
@nithu0115 Sounds good to me, worth testing. Thanks! |
Hi! We're hitting this issue quite regularly with CNI plugin version 1.4.1, since our bursty Notably this issue became more of a problem for us when we upgraded to Kubernetes 1.13, since our Job definition specifies a I tested briefly using CNI plugin 1.5.0 (under Kubernetes 1.13), and couldn't seem to reproduce the race condition, though didn't investigate too much further for now, since we're going to stick to Kubernetes 1.12 until the Kubernetes regression is fixed anyway. |
For me the Liveness / readiness probe provided by @nithu0115 is not working. |
@dng-dev This works:
You need to set the host to localhost. Since it only listens on localhost |
I will make a PR |
@stijndehaes thanks for the advice, i also found that the issue is to set the host. But some was-node pods stays unhealthy. Its a new eks cluster.
|
@dng-dev not sure what is happing. I am having no problems atm. But I am using 1.4.1 have you looked into the logs? |
@dng-dev, I have unreferenced commits from my local repository which was adding confusion. I am still testing if tcpsocket liveness and readiness probes is better approach or otherwise. I have run into similar problems if aws-node pod takes more than usual time to start and the goes into crashloop back off with liveliness probe failed message. |
My PR is not enough just did a test launching a 100 pods an adding an extra 8 machines. Still got errors on some on the nodes. |
Maybe we should use this: https://github.com/grpc-ecosystem/grpc-health-probe/ ? |
Ok adding a long timeout does not work. As kubernetes does not care if this pod is ready or not. The real problem is that k8s thinks the CNI is correctly installed before it is running see: #282 |
@stijndehaes @dng-dev @uthark I wrote a quick fix by adding healthz to IPamD and client to poll IPamD endpoint https://raw.githubusercontent.com/nithu0115/amazon-vpc-cni-k8s/342626c1c885b63254cd114f3e82fe91ab63eb03/config/v1.5/aws-k8s-cni.yaml Would you mind giving it a spin and see if this helps ? |
I tried deploying this fix and got new errors specifically on new hosts
|
Hi I am seeing exact same error jobs pod. They get stuck in container creating state |
@kimi1987 @ericlarssen-wf Can you try with the recently released v1.5.2? |
lgtm, was able to verify that this v.1.5.2 worked |
@mogren Ill try the new version today and will let you know. |
Tried the new version I dont see the issue for now. Ill do some more runs. Also, AWS doc says new version is 1.5.2 but I created new cluster today and I had to manually change the container image to 1.5.2 it was 1.5.1 |
@kimi1987 Thanks for testing. Yes, the default for new clusters is still v1.5.1. I'm working on a v1.5.3 with some additional fixes for some rare cases. |
v1.5.3 has a fix for this on new nodes, and the fix has been merged back to master. |
Issue description
Pods are failing to create with sanbox error. We are using latest CNI (1.3.0) and there sufficient IPs available for pods
Analysis
I went through the Kubelet and aws-node logs. This happens as soon as the Worker Nodes joins the cluster and if there are any pending work loads which needs to be scheduled as soon as the Worker Nodes goes to Ready state. I see Kubelet failed to start pod because of no available IP address which tally with the ipamd daemon not having IP address in its warm pool.
This is what I found so far about issue: As soon as the aws-node CNI was initialized at 2019-01-29T01:57:01Z, Kubelet gave the aws-node cni todo pods networking to assign a private ip addresses. However, aws-node CNI took over 7seconds to add ip addresses to warm pool which happened around 2019-01-29T01:57:08Z. This caused the pods to fail in those 7 seconds as they could not get a IP address with error - "failed to assign an IP address to container".
Below are the last occurrences of the error where it is complaining about "failed to assign an IP address to container".
Kubelet logs
ipamd logs
The text was updated successfully, but these errors were encountered: