rpc: performance fix for getProgramAccounts

[ckamm]

Problem

The result accounts were pushed into a fresh vector one-by-one, which produced significant slowdowns for very large responses (visible when exceeding thousands of accounts).

Summary of Changes

Pre-allocate the right capacity - the result length is known.

[ckamm]

Optionally these could also be rewritten to use iterators, like

           keyed_accounts
                .into_iter()
                .map(|(pubkey, account)| {
                    Ok(RpcKeyedAccount {
                         pubkey: pubkey.to_string(),
                         account: encode_account(&account, &pubkey, encoding, data_slice_config)?,
                    })
                })
                .collect::<Result<Vec<_>>>()?

[jstarry]

Optionally these could also be rewritten to use iterators, like

           keyed_accounts
                .into_iter()
                .map(|(pubkey, account)| {
                    Ok(RpcKeyedAccount {
                         pubkey: pubkey.to_string(),
                         account: encode_account(&account, &pubkey, encoding, data_slice_config)?,
                    })
                })
                .collect::<Result<Vec<_>>>()?

I typically prefer this approach, do you mind updating?

[ckamm]

@jstarry Not at all, that's what I did initially, then undid to keep the patch solely about the performance aspect. :)

[jstarry]

lgtm once the checks are fixed, thanks for this!

[CriesofCarrots]

Looks like you just need to remove the Ok(..?) wrappers in two of your iterators, then gtg.
Thanks for this!

[ckamm]

Looks like you just need to remove the Ok(..?) wrappers in two of your iterators, then gtg.

Could you elaborate? I'm intentionally making the map() produce an iterator over Result<_>, so I can deal with the potential error inside the closure. Is there a better way to write it?

[jstarry]

[brianlong]

When this is ready, please prepare a version of v1.6.25 with these changes for staging & user testing. Apps are eager to test this out on the large mainnet account set for Serum Program V3.

[CriesofCarrots]

Thanks again!

[codecov]

Codecov Report

Merging #19941 (36681ce) into master (28673d5) will decrease coverage by 0.0%.
The diff coverage is 61.8%.

@@            Coverage Diff            @@
##           master   #19941     +/-   ##
=========================================
- Coverage    82.6%    82.6%   -0.1%     
=========================================
  Files         478      478             
  Lines      133443   133545    +102     
=========================================
- Hits       110345   110343      -2     
- Misses      23098    23202    +104