[BUG]There is a memory leak in the yurt-tunnel-server, consuming too much memory

[huweihuang]

What happened:

There is a memory leak in the yurt-tunnel-server, consuming too much memory, Currently the memory over 80G.

 crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                        ATTEMPT             POD ID              POD
554efaae142a4       acb209d36497c       4 months ago        Running             yurt-tunnel-server          0                   b4f2de2cd768d       yurt-tunnel-server-558cd556d8-mbbvf


crictl stats
CONTAINER           CPU %               MEM                 DISK                INODES
554efaae142a4       2.28                82.48GB             53.25kB             15

Environment:

version: yurt-tunnel-server : v1.1.0

docker.io/openyurt/yurt-tunnel-server                             v1.1.0              acb209d36497c       15.3MB

others

/kind bug

[River-sh]

Please provide the cluster environment, some special configurations, and pprof information

[huweihuang]

Are there any input errors in this command? Error returning unrecognized profile format

go tool pprof http://HOST_IP/debug/pprof/profile
Fetching profile over HTTP from http://HOST_IP/debug/pprof/profile
http://HOST_IP/debug/pprof/profile: parsing profile: unrecognized profile format

[rambohe-ch]

Are there any input errors in this command? Error returning unrecognized profile format

go tool pprof http://HOST_IP/debug/pprof/profile
Fetching profile over HTTP from http://HOST_IP/debug/pprof/profile
http://HOST_IP/debug/pprof/profile: parsing profile: unrecognized profile format

@huweihuang The default meta port for tunnel-server is 10265

[huweihuang]

(pprof) top 20
Showing nodes accounting for 47.47GB, 97.58% of 48.65GB total
Dropped 319 nodes (cum <= 0.24GB)
Showing top 20 nodes out of 31
      flat  flat%   sum%        cum   cum%
   25.91GB 53.27% 53.27%    25.91GB 53.27%  bufio.NewWriterSize (inline)
   12.91GB 26.54% 79.80%    12.91GB 26.54%  bufio.NewReaderSize (inline)
    1.67GB  3.43% 83.23%     1.74GB  3.59%  net/textproto.(*Reader).ReadMIMEHeader
    1.26GB  2.59% 85.83%     1.26GB  2.59%  runtime.malg
    1.22GB  2.51% 88.34%     5.72GB 11.75%  net/http.(*conn).readRequest
    0.86GB  1.77% 90.11%     3.39GB  6.96%  net/http.readRequest
    0.68GB  1.39% 91.50%    13.72GB 28.20%  sigs.k8s.io/apiserver-network-proxy/pkg/server.(*Tunnel).ServeHTTP
    0.58GB  1.20% 92.70%     0.86GB  1.78%  context.propagateCancel
    0.51GB  1.06% 93.76%     0.51GB  1.06%  net/http.(*Server).newConn (inline)
    0.47GB  0.96% 94.72%     1.33GB  2.74%  context.WithCancel
    0.46GB  0.94% 95.65%     0.46GB  0.94%  net/url.parse
    0.39GB  0.81% 96.46%     0.39GB  0.81%  net.newFD (inline)
    0.28GB  0.58% 97.04%     0.28GB  0.58%  context.(*cancelCtx).Done
    0.14GB  0.28% 97.32%    45.86GB 94.28%  net/http.(*conn).serve
    0.13GB  0.26% 97.58%     0.33GB  0.67%  net/textproto.(*Reader).ReadLine (inline)
         0     0% 97.58%    12.90GB 26.52%  bufio.NewReader (inline)
         0     0% 97.58%    12.96GB 26.64%  bufio.NewWriter (inline)
         0     0% 97.58%     1.21GB  2.48%  github.com/openyurtio/openyurt/pkg/yurttunnel/server.runProxier.func1
         0     0% 97.58%     0.69GB  1.42%  net.(*UnixListener).Accept
         0     0% 97.58%     0.69GB  1.42%  net.(*UnixListener).accept

[huweihuang]

goroutine info:

[huweihuang]

maybe fix by apiserver-network-proxy, commit: kubernetes-sigs/apiserver-network-proxy@c747eaf, issue: kubernetes-sigs/apiserver-network-proxy#270

[huweihuang]

more error logs:

tunnel.go:74] "currently no tunnels available" err="No backend available"
interceptor.go:136] fail to setup the tunnel: fail to setup TLS handshake through the Tunnel: write unix @->/tmp/interceptor-proxier.sock: write: broken pipe
server.go:696] "DIAL_RSP contains failure" err="dial tcp xxx:80: connect: connection refused"
interceptor.go:136] fail to read response from the tunnel: read unix @->/tmp/interceptor-proxier.sock: read: connection reset by peer

[huweihuang]

@rambohe-ch you can close the issue for fixed the issue by pr #1471