firod Memory leak

[pinpins]

Latest master compiled on Ubuntu 16.04

ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -20
27.3 7.8 16773524 74498 firod

27GB of usage.

Observation is that this can be connected with RPC usage, in case of daemon idling it only consumes 0.8GB, once RPC are triggered by pool, it spikes to > 25GB of usage.

[a-bezrukov]

Hi @pinpins,
What are the suspicious RPCs? How often they are invoked?

[pinpins]

The one most frequently executed is getblocktemplate. Also listtransactions, getblock, getinfo.

Then we have submitblock, but it is invoked rarely.

[pinpins]

Today crashed again ending with below, this is since I reduced workqueue param, thinking it may cause memory issues

2021-03-17 10:35:09 WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting
2021-03-17 10:35:10 WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

config used

rpcworkqueue=24
rpcthreads=64
rpctimeout=5
rpcclienttimeout=5
rpcservertimeout=5

[pinpins]

Additionally my debug.log is full of, like 95% of log entries

2021-03-17 10:29:45 CWalletTx::RelayWalletTransaction() --> invalid condition
2021-03-17 10:29:45 CWalletTx::RelayWalletTransaction() --> invalid condition

[a-bezrukov]

Thanks for your info, I am investigating.

[a-bezrukov]

What is the output free -g on your machine?

[pinpins]

free -g
              total        used        free      shared  buff/cache   available
Mem:             29          15          10           0           3          13
Swap:            25           6          19

[a-bezrukov]

Thanks!
So, firod uses 27.3% of 29 GB of mem, 8 GB approx. None of our instances use that much memory... I'm moving on.

[pinpins]

I also added size and vsize to compare with others

 ps -eo pmem,pcpu,size,vsize,pid,cmd | sort -k 1 -nr | head -20
21.6  9.8 12229572 12597356 41412 firod
 2.9 23.2 2031760 2910584 60134 PRiVCYd
 2.5  3.9 2056600 2844028 2311 sonod
 2.3  0.6 2953392 3400160 2192 shieldd

from ps man

   s     size         memory size in kilobytes
   v     vsize        total VM size in KiB

[pinpins]

Since last comment timing (6h passed)

ps -eo pmem,pcpu,size,vsize,pid,cmd | sort -k 1 -nr | head -20
32.4  7.0 15834308 16202092 41412 firod

[a-bezrukov]

I cannot see such behavior on any of or nodes, nor can I reproduce even 1 byte leak in my testing setup.

[pinpins]

can you reach me in discord to brainstorm or, if I should recompile perhaps, use different version etc? maybe to renew wallet?

[a-bezrukov]

Yep, sure

[a-bezrukov]

The fix will be included into the next release. PR: #1010