RPC uses base58 for large binary leading to slow response

[ryoqun]

Problem

RPC responses encodes account.data with base58:

solana/client/src/rpc_response.rs

Line 78 in ad7b113

data: bs58::encode(account.data.clone()).into_string(),

base58 isn't suited for large arbitrary-sized binary blobs. it's only for hash: #9490 (comment)

And it's very easy to attack dos. Also, viewing program account data (which tends to be large by definition) takes long time to retrieve via RPC.

$ time curl -s -X POST -H 'Content-Type: application/json' -d '{"id":1,"jsonrpc":"2.0","method":"getAccountInfo","params":["SysvarS1otHistory11111111111111111111111111",{"commitment":"max"}]}' http://tds.solana.com -o - | wc -c
179217

real    0m24.143s
user    0m0.005s
sys     0m0.007s

$ time curl -s -X POST -H 'Content-Type: application/json' -d '{"id":1,"jsonrpc":"2.0","method":"getAccountInfo","params":["SysvarRecentB1ockHashes11111111111111111111",{"commitment":"max"}]}' http://tds.solana.com -o - | wc -c
8389

real    0m0.377s
user    0m0.005s
sys     0m0.005s

$ time curl -s -X POST -H 'Content-Type: application/json' -d '{"id":1,"jsonrpc":"2.0","method":"getAccountInfo","params":["SysvarC1ock11111111111111111111111111111111",{"commitment":"max"}]}' http://tds.solana.com -o - | wc -c
239

real    0m0.304s
user    0m0.002s
sys     0m0.010s

As predicted theoretically, this is quadratic time increase to the size of account.data:

$ pry
[9] pry(main)> 179217.0 / 8389
=> 21.363332935987604
[11] pry(main)> (377 - 304) * 21.363332935987604 ** 2
=> 33316.615571771246    # =~ 24.143s

Proposed Solution

Use base64? But compat... ;)

FYI: @CriesofCarrots

Discovered via #9490.

[mvines]

This would likely reduce the load on our RPC servers significantly.

[garious]

If a new API endpoint because we want a new encoding, why base64 and not unicode via String::from_utf8?

[garious]

For this endpoint, how about we return an error for large accounts, and perhaps that error includes a hash of the account data?

[mvines]

I was thinking of allowing the caller to supply their preferred encoding as an option to some of the existing RPC methods

[garious]

Umm, so caller has the option to DOS?

[mvines]

Imposing a max response length for base58 at the same time would solve that and allow for a smooth migration. Large base58 responses already take forever or time out already. Run solana account TokenSVp5gheXUvJ6jGWGeCsgPKgnE3YgdGKRVCMY9o, ~10 seconds to respond in base58

[garious]

Got it, yeah, perfect

[mvines]

Priority here should be on adding an optional user-provided encoding to getAccountInfo to improve response time. Streamlining the other RPC methods that return base58 can be secondary