solana-ledger-tool verify command is too slow

[mvines]

solana-ledger-tool verify takes ~5minutes to run during a testnet sanity check (cc: #2314). Culprit is very likely process_blocktree().

Decide if we should just disable ledger verification as a part of ./net.sh sanity or try to optimize.

Example: https://buildkite.com/solana-labs/testnet-management/builds/23148#4b97278e-1ad9-4eeb-9028-5a0467d81f85/228

[mvines]

#3192 switches process_blocktree() to use par_process_entries() under the hood, but that didn't seem to produce an meaningful speed up in a quick test of processing a ledger output by a couple minutes of ./run.sh.

@sambley - how hard is it to create those profiling flame graphs? Any docs on how? I think the next step here is to grab a profile of process_blocktree() and see where the time is going

[sambley]

This link has the required information to generate the profiling information
https://blog.anp.lol/rust/2016/07/24/profiling-rust-perf-flamegraph/

[sambley]

@mvines, let me know if you want to take a look at it

[sambley]

Looks like significant amount of time is spent in iterating over StatusCache::has_signature(). In this run it was taking around 55 secs to complete the verify.
flame.pdf

[sambley]

This change has an update to increase NUM_TICKS_PER_SECOND to 100 from 10 which seems to improve the peformance a bit
766f3ec

[sambley]

Looks like the bottleneck is inside accounts squash, looking into it