solana-ledger-tool verify OOM's with master on mnb

[steviez]

Problem

Running solana-ledger-tool verify with the tip of master OOM's after processing a . For example, with a snapshot a from epoch 333, I see an OOM after processing < 250 slots. I believe this has been an issue for a little bit, as jwash also reported hitting some OOM's with master roughly 1 month back:
https://discord.com/channels/428295358100013066/439194979856809985/992161854686240778

mvines had a potential fix with #26349 that enabled AccountsBackgroundService when running verify; however, I still encountered the OOM with this.

Some things that have already been verified:

• roots are being made (BankForks::set_root() is getting called)
• banks are getting dropped (Bank::Drop() is getting called)

Proposed Solution

Debug and fix, several methods to approach

• Idea 1: Profile (heaptrack) and look for any obvious offender
• Idea 2: git bisect (1.10 does not show the issue)
  • The common base between v1.10.32 and tip of master is 1535748 which leaves about ~1700 commits (assuming the base is good and something that changed on master caused the OOM) ... not great but doable

Testing Results

The following are for attempting to process roughly 10k slots from a recent mainnet beta ledger.

9cdd84bb5e66d9db01ef9aa5c3e0277731c3f00e ==> No OOM // v1.10.32 tag
224550d65fca059a46f6293d41cee7055986a638 ==> OOM    // jwash recent improvement; also OOM'd with mvines PR on top

[steviez]

Luckily I pinged @jeffwashington and he had just put in a fix that makes things better: 224550d

• Without this change, I was sometimes seeing as few as ~200 processed before OOM
• With this change, my single run made it through ~5500 processed slots before OOM

A definite improvement, but I think there is still something worth investigating here

[jeffwashington]

maybe related to 83e0412
specifically removal of exhaustively_free_unused_resource

[steviez]

Unfortunately, doing a git bisect doesn't have a good path. There was some fairly trivial patches to get compatibility with the new snapshot / ledger on older commits; however, processing sputters out from TX's not being handled the same in older version. I would essentially have to bisect to find commits that would create compatibility ... we could do this if we get desperate, but going to pursue other angles at the moment.

For completeness incase we come back, here were the commits to get snapshot / ledger compatible:

• If a commit is unable to open snapshot, cherry-pick 8caced6
• If a commit is unable to open rocksdb, use ldb to drop optimistic_slots column

[jeffwashington]

I have done all this already and have the branches. I am pretty sure this is it:

…

83e0412 - Run real snapshot packager while processing blockstore at validator startup (3 months ago) <Michael Vines> On Aug 3, 2022, at 10:48 PM, steviez ***@***.***> wrote: Unfortunately, doing a git bisect doesn't have a good path. There was some fairly trivial patches to get compatibility with the new snapshot / ledger on older commits; however, processing sputters out form some TX's. I would essentially have to bisect to find commits that would create compatibility ... we could do this if we get desperate, but going to pursue other angles at the moment. For completeness incase we come back, here were the commits to get snapshot / ledger compatible: If a commit is unable to open snapshot, cherry-pick 8caced6 <8caced6> If a commit is unable to open rocksdb, use ldb to drop optimistic_slots column — Reply to this email directly, view it on GitHub <#26895 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASCZMGAP3AIBYDKQ2XJR7D3VXM4SVANCNFSM55MW464Q>. You are receiving this because you were mentioned.

[jeffwashington]

I was after a different issue, but ledger tool perf was the main thing.

…

On Aug 3, 2022, at 10:59 PM, Jeff Washington ***@***.***> wrote: I have done all this already and have the branches. I am pretty sure this is it: > 83e0412 - Run real snapshot packager while processing blockstore at validator startup (3 months ago) <Michael Vines> > On Aug 3, 2022, at 10:48 PM, steviez ***@***.*** ***@***.***>> wrote: > > > Unfortunately, doing a git bisect doesn't have a good path. There was some fairly trivial patches to get compatibility with the new snapshot / ledger on older commits; however, processing sputters out form some TX's. I would essentially have to bisect to find commits that would create compatibility ... we could do this if we get desperate, but going to pursue other angles at the moment. > > For completeness incase we come back, here were the commits to get snapshot / ledger compatible: > > If a commit is unable to open snapshot, cherry-pick 8caced6 <8caced6> > If a commit is unable to open rocksdb, use ldb to drop optimistic_slots column > — > Reply to this email directly, view it on GitHub <#26895 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASCZMGAP3AIBYDKQ2XJR7D3VXM4SVANCNFSM55MW464Q>. > You are receiving this because you were mentioned. >

[steviez]

specifically removal of exhaustively_free_unused_resource

In support of this, I scraped incidence rates from solana-ledger-tool verify runs of the datapoint metrics that from the gc functions that were previously called by exhaustively_free_unused_resource():

v1.10.32
  accounts_db-flush_accounts_cache ==> 183 instances
  clean_accounts ==> 215 instances
  shrink_stats ==> 278 instances
tip-of-master
  accounts_db-flush_accounts_cache ==> 0 instances
  clean_accounts ==> 0 instances
  shrink_stats ==> 0 instances
tip-of-master + mvines make ABS run in solana-ledger-tool verify
  accounts_db-flush_accounts_cache ==> 0 instances
  clean_accounts ==> 1 instances
  shrink_stats ==> 2 instances

So, these cleanup functions are not getting called in tip of master, or even with the mvines PR