Memory leaking leads to OOMs

[evgkrsk]

Somehow all spegel's pods in my test cluster have linearly increasing memory usage that already killed one of pods twice in 1 day due to OOM (limit is 320Mi).

How can I help to debug such issue?

[evgkrsk]

Seems like disabling tags resolution helps to prevent OOMs:

[phillebaba]

That is an interesting finding. I can't really see how resolving tags would effect memory usage. Could I ask which version of Spegel you are using and what Kubernetes flavor you are deploying on.

[evgkrsk]

v0.0.14 image and vanilla k8s-1.23 on bare-metal.

[phillebaba]

@evgkrsk I have not come up with a method of reproducing this. I just released v0.0.15 which contains the fix libp2p/go-libp2p-kad-dht#960 which seems to have caused memory leaks during lookups. Could you try this new version and see if you experience the same issues?

[evgkrsk]

I have upgraded my test cluster to v0.0.16 and so far memory usage decreased by an order of magnitude (300Mi -> 50Mi). Lets wait a bit to get more statistics...

[phillebaba]

@evgkrsk any updates?

[evgkrsk]

Issue resolved :-)