docker.sock fd leak when using DSR

[johanot]

It looks like kube-router is leaking socket connections to docker.sock, when applying DSR=tunnel to a service.

Reproduce

1. Create a service
2. Annotate with kube-router.io/service.dsr=tunnel
3. Assign X external IP's to the service to enable DSR
4. Set kube-router ipvs-sync period low (e.g. 5 seconds) - it will make the leak run faster

Indications

sudo lsof 2>/dev/null |grep kube-rou
kube-rout 27802             root   11u     unix 0xffffa069d9e6f800       0t0  318621301 type=STREAM
kube-rout 27802             root   12u     unix 0xffffa069d9e6a400       0t0  318621302 type=STREAM
kube-rout 27802             root   13u     unix 0xffffa069a6dc2800       0t0  318585826 type=STREAM
kube-rout 27802             root   14u     unix 0xffffa088a11c6400       0t0  318563258 type=STREAM
kube-rout 27802             root   15u     unix 0xffffa069a6dc0800       0t0  318585827 type=STREAM
kube-rout 27802             root   16u     unix 0xffffa089e0757800       0t0  318567344 type=STREAM
kube-rout 27802             root   17u     unix 0xffffa069a6dc4400       0t0  318585828 type=STREAM
...

... kube-router eventually dies when reaching open-files ulimit.

Suggestions

Invoke client.close on the docker client at some point?

Env info

kube-router
0.1.0-rc2

OS/Kernel
Linux container-p01 4.15.7 #1-NixOS SMP Wed Feb 28 09:21:39 UTC 2018 x86_64 GNU/Linux

docker server
Version: 17.09.1-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.9.2
Git commit: 19e2cf6259bd7f027a3fff180876a22945ce4ba8
Built: Thu Jan 1 00:00:01 1970
OS/Arch: linux/amd64
Experimental: false

[murali-reddy]

@johanot thanks for trying DSR and reporting this issue. Do you see this issue specific to DSR.

Reason i ask it, there is an issue #314

I will try to reproduce anyway.

[johanot]

@murali-reddy Yes.. Only with DSR. When I remove the DSR annotation, the fd count is more or less constant, but not shrinking - indicating that the connections are never closed.

kube-router/app/controllers/network_services_controller.go

Line 694 in a466ace

containerSpec, err := dockerClient.ContainerInspect(context.Background(), containerId)

is the only place where the docker-socket is used, right? As far as I understand, all other container info is obtained through kube-apiserver?

[murali-reddy]

is the only place where the docker-socket is used, right? As far as I understand, all other container info is obtained through kube-apiserver?

Yes

@johanot some how it does not reproduce for me. Can you please try cloudnativelabs/kube-router-git:dsr-leak image. It has fix #343.

[johanot]

@murali-reddy Done testing. LGTM. Number of open connections is now stable. Thank for the quick response and good work!