Excessive "Failed to receive from netlink: no buffer space available" errors

[zihaoyu]

We run flannel for our Kubernetes clusters, but not as a CNI plugin yet. Still the old way - run flannel, output the properties file, let docker read --bip flag from the properties file.

We noticed a large amount of the following errors when the cluster is under load:

Jul 14 15:11:46 ip-10-72-134-42.ec2.internal flannel-wrapper[1528]: E0714 15:11:46.419175    1528 device.go:222] Failed to receive from netlink: no buffer space available"

Expected Behavior

We should see fewer errors in flannel logs?

Current Behavior

We see a lot of such errors. Below is a Kibana screenshot.

Possible Solution

Steps to Reproduce (for bugs)

1. Scale Kubernetes cluster to ~300 minions.
2. Increase load/traffic on the cluster
3. Check flannel logs.

Context

We are seeing network timeouts in almost all of our microservices in the cluster, not sure if they are related, but highly susceptible.

Your Environment

• Flannel version: v0.7.0
• Backend used (e.g. vxlan or udp): vxlan
• Etcd version: v3.1.9
• Kubernetes version (if used): v1.5.7
• Operating System and version: CoreOS 1353.7.0 stable
• Link to your project (optional):

[tomdee]

It looks like it might be worth trying to increase the buffer size of the netlink socket
From https://www.netfilter.org/documentation/FAQ/netfilter-faq-4.html

these are standard Netlink sockets, and you can tune their receive buffer sizes via /proc/sys/net/core, sysctl, or use the SO_RCVBUF socket option on the file descriptor.

The file descriptor is available by calling GetFd() on nlsock here - https://github.com/coreos/flannel/blob/master/backend/vxlan/device.go#L214