Memory Leak?

[christroutner]

I'm running several JavaScript libp2p nodes using js-libp2p v1.2.1 and helia v2.1.0. I know these version are a couple months old, but they are running reliably.

What I've noticed is that the memory usage starts at about 500MB and then creeps up to several GB (like around 12GB) over the course of a couple days. This makes me think there might be a memory leak.

I would expect the memory to increase a little bit as the node makes contact with other nodes and opens pubsub channels between them. But the size and shape of the memory increase seems to indicate that there is a memory leak.

Has anyone else experienced this? Are there updates in newer versions that address this memory usage?

My node cluster is relatively small at about 10 nodes. They don't connect to the wider IPFS network. But they do open pubsub channels between each node.

I run the nodes inside Docker containers, so I can set memory limits. However, this kills the node and restarts it when it hits that limit, which is not an ideal solution.

[christroutner]

I just checked a couple desktops running my IPFS nodes and they all seem to reach a steady state of around 7GB of memory per node.

[christroutner]

Here is the memory profile over 24 hours. This is a classic memory leak pattern with the memory usage increasing slowly, up and to the right.

[2color]

Can you please share how you're instantiating js-libp2p?

[christroutner]

This createNode() function is how I'm instantiating js-libp2p.

[SgtPooki]

@christroutner i think the main thing we're looking for is the code you're using to configure yours

[christroutner]

I've been digging into this memory leak. I still have more questions than answers.

There are two possible root causes:

1. It could be that js-libp2p increases its memory as it makes connections to nodes, opens pubsub channels, etc during normal operation.
2. It could also be that the helia-coord library that I use to coordinate my nodes is adding unresolved promises over time and that is increasing the memory.

Those are not mutually exclusive. They could both be happening at the same time.

I wish I could read documentation or best-practice examples on how to handle promises for connecting to nodes, opening pubsub channels, getting files, and pinning files. It's a known issue in JS that promises that never resolve can not be cancelled. I'm not sure of the proper way to issue promise-based commands to libp2p, if they never resolve.

[SgtPooki]

One thing you should do is check out memlab. Create some heapdumps and then run it through memlab. if there is a memory-leak, or steadily increasing object in rss, it will show you exactly what it is.

[christroutner]

Here are some data points I retrieved from one of the nodes after monitoring it each day for several days:

5/8 6:30AM: 143MB, 2-5% CPU
5/9 8:45AM: 214MB, 10% CPU
5/10 11AM: 280MB, 5-10% CPU
5/11 10AM: 370MB, 10-25% CPU
5/13 9:30AM: 444MB, 5-35% CPU
5/14 7:45AM: 480MB, 5-35% CPU

[SgtPooki]

With helia and js-libp2p, you will want to use non-default blockstore and datastore, as they're in-memory by default. Since they're memory only, they will steadily increase the memory size your process uses.

If you're not already using non-memory stores, that's the first thing I would switch

[SgtPooki]

we use blockstore-fs and datastore-level in helia-http-gateway: https://github.com/ipfs/helia-http-gateway/blob/87deb922f6137b0a288954e6de149107e0f7a648/src/get-custom-helia.ts#L5-L6