Super slow access to service IP from host (& host-networked pods) with Flannel CNI

[fengye87]

Ref: kubernetes/kubernetes#87233 (comment)

The k/k guys believed this is a Flannel's issue, so re-post here.

[mariusgrigoriu]

We are seeing multiple reports that flannel + kube 1.17 don't play well:

• kubernetes 1.17.0 - kubeadm init - kube-controller-manager status is ContainerCreating kubernetes/kubernetes#86961 (comment)
• ClusterIP services not accessible when using flannel CNI from host machines in Kubernetes #1243

@tomdee can you look at these?

[mikebryant]

I think I've been hitting this issue yesterday/today

Some tests I was doing, from one host (not in a container)

• curl http://pod-on-other-node worked
• curl http://service-ip worked after a delay of 1 second. tcpdump showed a retransmission which got through. This was reproducible

I've just swapped to the host-gw backend and everything's working normally

flannel: 0.11.0
kubernetes: 1.17.2, installed using kubeadm
on a baremetal switched network.

[mariusgrigoriu]

Something we noticed is that the number of conntrack insert_failed was dramatically higher while running kube 1.17.

[thibautvincent]

We experienced the same issue today. Fixed this by using the solution of @mikebryant. Is there any permanent solution on the way?

[MansM]

@tomdee as you are the last remaining maintainer, who should I ping/tag to get this looked at.

[tkislan]

Just FIY, this is not related only to 1.17 .. Because of these issues here, I've tried to downgrade from 1.17.3 to 1.16.8, but same result
First of all, route is missing from service cidr to cni0 interface gateway, so I had to manually add it in order for it to even resolve

ip route add 10.96.0.0/12 via 10.244.3.1

And after that, even traceroute is super slow

traceroute <service>.<namespace>.svc.cluster.local
traceroute to <service>.<namespace>.svc.cluster.local (10.106.49.44), 30 hops max, 38 byte packets
 1  10.244.3.1 (10.244.3.1)  3097.057 ms !H  3097.946 ms !H  3119.540 ms !H

[mariusgrigoriu]

Just curious, how many folks experiencing this issue are using hyperkube?

[mengmann]

I'm having this issue with vxlan backend both with flannel version 0.11 and 0.12 aswell.
Affected kubernetes versions 1.16.X, 1.17.x and 1.18.x.

Finally setting up a static route on my nodes to service network through cni0 interface helped me instantly:
ip route add 10.96.0.0/12 dev cni0

os: CentOS 7
install method: kubeadm
underlying plattform: Virtualbox 6

[pytimer]

Finally setting up a static route on my nodes to service network through cni0 interface helped me instantly:
ip route add 10.96.0.0/12 dev cni0

Fixed this problem by using the solution of @mengmann in kubernetes version v1.17.2 .

[blueabysm]

I think I've been hitting this issue yesterday/today

Some tests I was doing, from one host (not in a container)

• curl http://pod-on-other-node worked
• curl http://service-ip worked after a delay of 1 second. tcpdump showed a retransmission which got through. This was reproducible

I've just swapped to the host-gw backend and everything's working normally

flannel: 0.11.0
kubernetes: 1.17.2, installed using kubeadm
on a baremetal switched network.

Exactly the same issue here

[skamboj]

I think I've been hitting this issue yesterday/today
Some tests I was doing, from one host (not in a container)

• curl http://pod-on-other-node worked
• curl http://service-ip worked after a delay of 1 second. tcpdump showed a retransmission which got through. This was reproducible

I've just swapped to the host-gw backend and everything's working normally
flannel: 0.11.0
kubernetes: 1.17.2, installed using kubeadm
on a baremetal switched network.

Exactly the same issue here

Not sure if its the same issue but we noticed an additional delay of 1 second when upgrading from kubernetes 1.15.3 to 1.18.1. We seem to trace the problem to the --random-fully flag introduced by this PR. See the issue here

[blueabysm]

I think I've been hitting this issue yesterday/today
Some tests I was doing, from one host (not in a container)

• curl http://pod-on-other-node worked
• curl http://service-ip worked after a delay of 1 second. tcpdump showed a retransmission which got through. This was reproducible

I've just swapped to the host-gw backend and everything's working normally
flannel: 0.11.0
kubernetes: 1.17.2, installed using kubeadm
on a baremetal switched network.

Exactly the same issue here

Not sure if its the same issue but we noticed an additional delay of 1 second when upgrading from kubernetes 1.15.3 to 1.18.1. We seem to trace the problem to the --random-fully flag introduced by this PR. See the issue here

I'm currently working with kubernetes 17.3(some nodes 17.4). Fortunately there are not so many apps running on my new-built cluster, so I migrated them this week and changed the network fabric to calico according to this article. Now erverything works perfect. 😄

[Raven888888]

@rbrtbnfgl What do you think about this issue?

I am experiencing the same slowness when accessing service external to the cluster (flannel cni).
Ping inside pod for "google.com" sometimes results in

bad address

but occasionally success after 15 seconds...

I am almost at the point of switching to calico as it claims to solve the problem...

[rbrtbnfgl]

Which version of Flannel are you using? This is a very old issue I think it will be better if you create a new one with your setup config. It could be a problem with the UDP checksum #1679

[Raven888888]

@rbrtbnfgl
docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2, which i believe is the latest already. The only thing I have changed is:

net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "VNI" : 4096,
        "Port": 4789
      }
    }

Because my cluster is a mix of windows (worker only) and linux nodes, and the port numbers need to change.

I did try running

 iptables -t nat -I FLANNEL-POSTRTG -m mark --mark 0x4000/0x4000 -j RETURN

on all the master and worker nodes (no reboot or restart any services whatsoever, just run the command purely), but still does not resolve the issue.

The tricky bit is, the problem is not consistent. I'd say, 80% of the time ping will either give "bad address" or take > 15s to resolve, 20% of the time it works reasonably ok.

NB: ping ip always work, it is the DNS resolution, or the communication from pod to dns resolver via flannel, that seems to be causing the issue.

[rbrtbnfgl]

Is the issue only on the pods on the windows nodes or also with the ones on linux?

[Raven888888]

I have resolved my issue, although I am not sure how/why it causes issues to my cluster.

Root cause:
Coredns pods are all running in the same master node.
I suspect it may be due to the installation step sequence that mess it up? My steps:

1. Install 1 linux master node with HA and external etcd, then kubeadm init cluster.
2. By now it should already have coredns running, since there is only 1 master node, all coredns pods running in it.
3. Install flannel cni.
4. Add 2 more linux master nodes.
5. Add 3 more linux worker nodes.
6. Observe the issue above in any of the worker nodes.

Solution:
Run the following from master node

kubectl -n kube-system rollout restart deployment coredns

Perhaps after step 3, coredns is meant to be refreshed/restarted?

Anyway, thanks @rbrtbnfgl

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.