Linear memory growth in bootstrapped application process

[ManuGowda]

Expected behavior

The Bootstrapped application process memory should not grow in linear over a period of time.
The GC should be able to clean up memory on time.

Actual behavior

Note: the memory growth occurs over a period of time (6 days in this case)

Possible memory bloat/growth in bootstrapped application process.

• The rss, heapTotal, heapUsed memory usage grows over period of time (over a period of 6 days), however, the rss memory came down to 121.86 MB after 6th day, initially the rss growth started from 100MB to 500MB. The heapTotal, heapUsed continues to grow without any reduction.

15:23:49 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 265.21 MB",
 "heapTotal": "Memory: heapTotal 262.23 MB",
 "heapUsed": "Memory: heapUsed 244.24 MB",
 "external": "Memory: external 3.17 MB",
 "arrayBuffers": "Memory: arrayBuffers 1.21 MB"
}
15:23:49 INFO lisk-framework: Forged new block (module=lisk:app)
{
 "id": "310cd2cb90996c01ec3672de1d72134ef5185b1839cd5699bb90b8ef2c6cf956",
 "generatorAddress": "7a5ef1fc22c8a30a6ed2cd6657a0e07af7fb7f78",
 "seedReveal": "d7bccabb920173ec7ecf9dda43cfc740",
 "height": 19783,
 "slot": 27186,
 "reward": "500000000"
}

09:39:39 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 337.52 MB",
 "heapTotal": "Memory: heapTotal 331.14 MB",
 "heapUsed": "Memory: heapUsed 326.11 MB",
 "external": "Memory: external 8.21 MB",
 "arrayBuffers": "Memory: arrayBuffers 6.24 MB"
}
09:39:39 INFO lisk-framework: Forged new block (module=lisk:app)
{
 "id": "1b5091f9cb38ecdca2b1bb99a0b25803392e1a36a47f0b248ba133a965be5022",
 "generatorAddress": "3424a55d41c89317c131be6d9e8e889b4b18afb6",
 "seedReveal": "aed898e8221123e1f9121084ddd3543c",
 "height": 25963,
 "slot": 33761,
 "reward": "500000000"
}

09:24:09 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 553.05 MB",
 "heapTotal": "Memory: heapTotal 544.19 MB",
 "heapUsed": "Memory: heapUsed 521.29 MB",
 "external": "Memory: external 9.42 MB",
 "arrayBuffers": "Memory: arrayBuffers 7.45 MB"
}
09:24:09 INFO lisk-framework: Forged new block (module=lisk:app)
{
 "id": "0247bf6ab0d2aa07df59b08504350b516d78d30675d3a094ce0b668145bc301a",
 "generatorAddress": "403a9828db9afd04a5437fc85f32d3d12558e64d",
 "seedReveal": "649aa3ec344976077bafa54c0c1d4cf5",
 "height": 45027,
 "slot": 59588,
 "reward": "500000000"
}
09:59:09 INFO lisk-framework: New block added to the chain (module=lisk:app)
{
 "id": "3f5b210914d57db642f3863361ea7424a87b36c0e3ddf431d47cdb3f5c4294e0",
 "height": 103799,
 "numberOfTransactions": 0
}
09:59:09 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 121.86 MB",
 "heapTotal": "Memory: heapTotal 1154.59 MB",
 "heapUsed": "Memory: heapUsed 1094.5 MB",
 "external": "Memory: external 14.45 MB",
 "arrayBuffers": "Memory: arrayBuffers 12.49 MB"
}

Steps to reproduce

• Install Lisk Commander npm install -g lisk-commander
• Bootstrap an application lisk init /tmp/test_memory
• Add logs/profiling to watch memory growth, e.g process.memoryUsage
• Observe the memory growth

Which version(s) does this affect? (Environment, OS, etc...)

5.1.x, Node.js version 12.22.3

[ManuGowda]

rss – resident set size refers to the amount of space occupied in the main memory for the process, which includes code segment, heap, and stack. If your RSS is going up, there is a likelihood your application is leaking memory
heapTotal – total amount of memory available for JavaScript objects
heapUsed – total amount of memory occupied by JavaScript objects
external – amount of memory consumed by off-heap data (buffers) used by Node.js. This is where objects, strings, and closures are stored

[ManuGowda]

I guess looking at more data i see the rss memory usage is going down later point in time, which means the memory leak could not be present i guess. the heap usage is growing steadily though, need more debugging.

15:23:49 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 265.21 MB",
 "heapTotal": "Memory: heapTotal 262.23 MB",
 "heapUsed": "Memory: heapUsed 244.24 MB",
 "external": "Memory: external 3.17 MB",
 "arrayBuffers": "Memory: arrayBuffers 1.21 MB"
}
15:23:49 INFO lisk-framework: Forged new block (module=lisk:app)
{
 "id": "310cd2cb90996c01ec3672de1d72134ef5185b1839cd5699bb90b8ef2c6cf956",
 "generatorAddress": "7a5ef1fc22c8a30a6ed2cd6657a0e07af7fb7f78",
 "seedReveal": "d7bccabb920173ec7ecf9dda43cfc740",
 "height": 19783,
 "slot": 27186,
 "reward": "500000000"
}

09:39:39 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 337.52 MB",
 "heapTotal": "Memory: heapTotal 331.14 MB",
 "heapUsed": "Memory: heapUsed 326.11 MB",
 "external": "Memory: external 8.21 MB",
 "arrayBuffers": "Memory: arrayBuffers 6.24 MB"
}
09:39:39 INFO lisk-framework: Forged new block (module=lisk:app)
{
 "id": "1b5091f9cb38ecdca2b1bb99a0b25803392e1a36a47f0b248ba133a965be5022",
 "generatorAddress": "3424a55d41c89317c131be6d9e8e889b4b18afb6",
 "seedReveal": "aed898e8221123e1f9121084ddd3543c",
 "height": 25963,
 "slot": 33761,
 "reward": "500000000"
}

09:24:09 INFO lisk-framework: Memory Usage... (module=lisk:app)
{
 "rss": "Memory: rss 553.05 MB",
 "heapTotal": "Memory: heapTotal 544.19 MB",
 "heapUsed": "Memory: heapUsed 521.29 MB",
 "external": "Memory: external 9.42 MB",
 "arrayBuffers": "Memory: arrayBuffers 7.45 MB"
}
09:24:09 INFO lisk-framework: Forged new block (module=lisk:app)
{
 "id": "0247bf6ab0d2aa07df59b08504350b516d78d30675d3a094ce0b668145bc301a",
 "generatorAddress": "403a9828db9afd04a5437fc85f32d3d12558e64d",
 "seedReveal": "649aa3ec344976077bafa54c0c1d4cf5",
 "height": 45027,
 "slot": 59588,
 "reward": "500000000"
}

[ManuGowda]