RPC Node Memory Leak Frequently

[KaiQiu9527]

Problem

Proposed Solution

Hi Dev Team:
I'm running a solana rpc node in Hong Kong Region on Huawei Cloud. It was running well on the first few hours, while it will became fall behind and the memory usage will grow rapidly until being killed for OOM. I have try different specifications, like 32u256g and 64u512g, they both crashed for OOM. Is there any way to find out the reason?

Specification:
Machine: m7n.16xlarge.8 (64U512g, 3rd Generation Intel® Xeon® Scalable Processor)
Disk: 2T ESSD(I/O throughput up to 1000MB/S)
Network: 10Gbps

validator.sh is:
#!/bin/bash
exec solana-validator
--identity /home/sol/validator-keypair.json
--known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2
--known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ
--known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ
--known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S
--full-rpc-api
--no-voting
--ledger /mnt/solana/ledger
--accounts /mnt/solana/accounts
--log /home/sol/solana-rpc.log
--rpc-port 8899
--rpc-bind-address 0.0.0.0
--private-rpc
--dynamic-port-range 8000-8020
--entrypoint entrypoint.mainnet-beta.solana.com:8001
--entrypoint entrypoint2.mainnet-beta.solana.com:8001
--entrypoint entrypoint3.mainnet-beta.solana.com:8001
--entrypoint entrypoint4.mainnet-beta.solana.com:8001
--entrypoint entrypoint5.mainnet-beta.solana.com:8001
--expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d
--wal-recovery-mode skip_any_corrupted_record
--limit-ledger-size
--maximum-local-snapshot-age 20000

[github-actions]

This repository is no longer in use. Please re-open this issue in the agave repo: https://github.com/anza-xyz/agave