cni failed to set up pod

[damianoneill]

Hi, default install. I installed helm and then tried to provision the stable/nginx-ingress chart.

NODE_MEMORY_SIZE_GB=3 NODE_CPUS=2 NODE_COUNT=3 make up -j4
kubectl create serviceaccount tiller --namespace kube-system
kubectl create -f /tmp/rbac-config.yaml 

$ cat /tmp/rbac-config.yaml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system 

helm init --service-account tiller
helm upgrade --install nginx-ingress --namespace ingress --set controller.kind=DaemonSet --set controller.daemonset.useHostPort=true stable/nginx-ingress

I'm seeing the following in the controller pods, they are premantly in ContainerCreating status.

 Warning  FailedCreatePodSandBox  39s                 kubelet, node1     Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "6d408c26782737725756dcdbaf543206433cbaab253de34fabef7eb737a6955e" network for pod "nginx-ingress-controller-5zp68": NetworkPlugin cni failed to set up pod "nginx-ingress-controller-5zp68_ingress" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "6d408c26782737725756dcdbaf543206433cbaab253de34fabef7eb737a6955e" network for pod "nginx-ingress-controller-5zp68": NetworkPlugin cni failed to teardown pod "nginx-ingress-controller-5zp68_ingress" network: failed to get IP addresses for "eth0": <nil>]
  Normal   SandboxChanged          35s (x12 over 46s)  kubelet, node1     Pod sandbox changed, it will be killed and re-created.

[galexrt]

@damianoneill I was unable to reproduce on current master.

Could this be cause by the issue(s) you had with #2?

[damianoneill]

@galexrt I don't think so, I had applied the fix before running above. I'm working through a Kube training package on safari, it's referencing your project for multi node examples. I'll spend Monday trying to understand whats going on and come back with an update at the end of the day.

Thanks for your support.

[galexrt]

@damianoneill Thanks! When you are looking on Monday, please make sure to look into the kubelet logs (journalctl -u kubelet.service -xe) to see what the kubelet + CNI may is complaining about.

Hoping that these logs will tell what the issue, please post the error here then so I can see what to fix. 🙂

[damianoneill]

Output attached.

journalctl-kubelet.txt

[damianoneill]

@galexrt for the CNI issue, do you think it's related to containernetworking/cni#531 ? Looking at the VM that gets created for fedora, ipv6 is enabled by default.

[root@node1 ~]# sysctl -a | grep net.ipv6.conf | grep disable
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.disable_policy = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.disable_policy = 0
net.ipv6.conf.docker0.disable_ipv6 = 0
net.ipv6.conf.docker0.disable_policy = 0
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.disable_policy = 0
net.ipv6.conf.eth1.disable_ipv6 = 0
net.ipv6.conf.eth1.disable_policy = 0
net.ipv6.conf.flannel/1.disable_ipv6 = 0
net.ipv6.conf.flannel/1.disable_policy = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.disable_policy = 0
[root@node1 ~]#

[shamimgeek]

I am also hitting this issue

[sakhtar@new-host k8s-vagrant-multi-node]$ kubectl get po -n kube-system -o wide
NAME                             READY   STATUS              RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
coredns-fb8b8dccf-vrwpg          0/1     ContainerCreating   0          10m   <none>          master   <none>           <none>
coredns-fb8b8dccf-xzrcz          0/1     ContainerCreating   0          10m   <none>          master   <none>           <none>
etcd-master                      1/1     Running             0          10m   192.168.26.10   master   <none>           <none>
kube-apiserver-master            1/1     Running             0          10m   192.168.26.10   master   <none>           <none>
kube-controller-manager-master   1/1     Running             0          10m   192.168.26.10   master   <none>           <none>
kube-flannel-ds-amd64-nvvqn      1/1     Running             0          10m   192.168.26.12   node2    <none>           <none>
kube-flannel-ds-amd64-thsx9      1/1     Running             0          10m   192.168.26.13   node3    <none>           <none>
kube-flannel-ds-amd64-z6kh5      1/1     Running             0          10m   192.168.26.10   master   <none>           <none>
kube-flannel-ds-amd64-znnf6      1/1     Running             0          10m   192.168.26.11   node1    <none>           <none>
kube-proxy-4twp8                 1/1     Running             0          10m   192.168.26.12   node2    <none>           <none>
kube-proxy-d4z98                 1/1     Running             0          10m   192.168.26.11   node1    <none>           <none>
kube-proxy-k9rwp                 1/1     Running             0          10m   192.168.26.13   node3    <none>           <none>
kube-proxy-qvslj                 1/1     Running             0          10m   192.168.26.10   master   <none>           <none>
kube-scheduler-master            1/1     Running             0          10m   192.168.26.10   master   <none>           <none>
[sakhtar@new-host k8s-vagrant-multi-node]$ 

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Warning  FailedScheduling        6m33s (x2 over 6m33s)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling        6m29s (x3 over 6m31s)  default-scheduler  0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling        6m24s (x2 over 6m24s)  default-scheduler  0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled               5m48s                  default-scheduler  Successfully assigned kube-system/coredns-fb8b8dccf-vrwpg to master
  Warning  FailedCreatePodSandBox  5m47s                  kubelet, master    Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "e1fe43d423f1af48837bb0bce6f75b5038b34e5f70f9b32a56fe7604e6149422" network for pod "coredns-fb8b8dccf-vrwpg": NetworkPlugin cni failed to set up pod "coredns-fb8b8dccf-vrwpg_kube-system" network: failed to find plugin "loopback" in path [/opt/cni/bin], failed to clean up sandbox container "e1fe43d423f1af48837bb0bce6f75b5038b34e5f70f9b32a56fe7604e6149422" network for pod "coredns-fb8b8dccf-vrwpg": NetworkPlugin cni failed to teardown pod "coredns-fb8b8dccf-vrwpg_kube-system" network: failed to find plugin "portmap" in path [/opt/cni/bin]]
  Normal   SandboxChanged          46s (x25 over 5m47s)   kubelet, master    Pod sandbox changed, it will be killed and re-created.

Apr 01 14:16:13 master kubelet[3622]: E0401 14:16:13.419697    3622 pod_workers.go:190] Error syncing pod 53bf96e4-54c1-11e9-9e08-0800276a31d1 ("coredns-fb8b8>
Apr 01 14:16:17 master kubelet[3622]: E0401 14:16:17.748889    3622 container_manager_linux.go:445] cpu and memory cgroup hierarchy not unified.  cpu: /system>
Apr 01 14:16:17 master kubelet[3622]: E0401 14:16:17.750335    3622 container_manager_linux.go:490] failed to find cgroups of kubelet - cpu and memory cgroup >
Apr 01 14:16:21 master kubelet[3622]: W0401 14:16:21.465057    3622 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace f>
Apr 01 14:16:21 master kubelet[3622]: E0401 14:16:21.465611    3622 cni.go:352] Error deleting kube-system_coredns-fb8b8dccf-xzrcz/233dd576174cce4a6ada5d6d201>
Apr 01 14:16:21 master kubelet[3622]: E0401 14:16:21.480035    3622 remote_runtime.go:132] StopPodSandbox "233dd576174cce4a6ada5d6d20199372692a0e9cd2f3e608b00>
Apr 01 14:16:21 master kubelet[3622]: E0401 14:16:21.480145    3622 kuberuntime_manager.go:846] Failed to stop sandbox {"docker" "233dd576174cce4a6ada5d6d2019>
Apr 01 14:16:21 master kubelet[3622]: E0401 14:16:21.480386    3622 kuberuntime_manager.go:641] killPodWithSyncResult failed: failed to "KillPodSandbox" for ">
Apr 01 14:16:21 master kubelet[3622]: E0401 14:16:21.480443    3622 pod_workers.go:190] Error syncing pod 53bdcbe7-54c1-11e9-9e08-0800276a31d1 ("coredns-fb8b8>
Apr 01 14:16:28 master kubelet[3622]: W0401 14:16:28.409310    3622 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace f>
Apr 01 14:16:28 master kubelet[3622]: E0401 14:16:28.410043    3622 cni.go:352] Error deleting kube-system_coredns-fb8b8dccf-vrwpg/e1fe43d423f1af48837bb0bce6f>
Apr 01 14:16:28 master kubelet[3622]: E0401 14:16:28.413830    3622 remote_runtime.go:132] StopPodSandbox "e1fe43d423f1af48837bb0bce6f75b5038b34e5f70f9b32a56f>
Apr 01 14:16:28 master kubelet[3622]: E0401 14:16:28.413892    3622 kuberuntime_manager.go:846] Failed to stop sandbox {"docker" "e1fe43d423f1af48837bb0bce6f7>
Apr 01 14:16:28 master kubelet[3622]: E0401 14:16:28.413991    3622 kuberuntime_manager.go:641] killPodWithSyncResult failed: failed to "KillPodSandbox" for ">
Apr 01 14:16:28 master kubelet[3622]: E0401 14:16:28.414026    3622 pod_workers.go:190] Error syncing pod 53bf96e4-54c1-11e9-9e08-0800276a31d1 ("coredns-fb8b8>
Apr 01 14:16:33 master kubelet[3622]: W0401 14:16:33.430643    3622 cni.go:309] CNI failed to retrieve network namespace path: cannot find network namespace f>
Apr 01 14:16:33 master kubelet[3622]: E0401 14:16:33.431096    3622 cni.go:352] Error deleting kube-system_coredns-fb8b8dccf-xzrcz/233dd576174cce4a6ada5d6d201>
Apr 01 14:16:33 master kubelet[3622]: E0401 14:16:33.442604    3622 remote_runtime.go:132] StopPodSandbox "233dd576174cce4a6ada5d6d20199372692a0e9cd2f3e608b00>
Apr 01 14:16:33 master kubelet[3622]: E0401 14:16:33.442829    3622 kuberuntime_manager.go:846] Failed to stop sandbox {"docker" "233dd576174cce4a6ada5d6d2019>
Apr 01 14:16:33 master kubelet[3622]: E0401 14:16:33.443060    3622 kuberuntime_manager.go:641] killPodWithSyncResult failed: failed to "KillPodSandbox" for ">
Apr 01 14:16:33 master kubelet[3622]: E0401 14:16:33.443162    3622 pod_workers.go:190] Error syncing pod 53bdcbe7-54c1-11e9-9e08-0800276a31d1 ("coredns-fb8b8>
[root@master ~]# 

[galexrt]

@damianoneill @shamimgeek I have pushed some changes to master which fix the CNI plugin path issue.
Could you please test with the latest master and check if that issue is still happening.

[galexrt]

Closing as no response back and I have not been able to reproduce the issue since.