Allow programs to realloc their accounts within limits

[jackcmay]

Problem

Programs can only allocate account data via SystemInstruction::Allocate which only supports new account allocations (account owned by system program and alloc from 0 to x bytes).

Summary of Changes

Allow programs to realloc the data size of accounts they own. They are still limited by the BPF serialization cap of 10k and the maximum account size of 10M.

• This also allows programs to realloc by 10k multiple times to support >10k account sizes.
• And allows programs to dealloc their account data to size 0 for the purpose of closing accounts safely.

This is a proof-of-concept meant to start a conversation as to the safety of allowing these types of operations so these changes don't contain any featurization or testing yet.

Thoughts?

Fixes #14694

[jackcmay]

@jstarry @mvines @t-nelson 👀

[codecov]

Codecov Report

Merging #19475 (80e599c) into master (6f08f9d) will decrease coverage by 0.0%.
The diff coverage is 71.6%.

@@            Coverage Diff            @@
##           master   #19475     +/-   ##
=========================================
- Coverage    82.8%    82.8%   -0.1%     
=========================================
  Files         487      487             
  Lines      135451   135528     +77     
=========================================
+ Hits       112237   112265     +28     
- Misses      23214    23263     +49     

[t-nelson]

This will make a bunch of people very happy 🙂

[mvines]

Dumb sleepy Saturday question: How does a BPF program take advantage of this again?

[jackcmay]

Dumb sleepy Saturday question: How does a BPF program take advantage of this again?

Helpers and some tests added

[jstarry]

As discussed offline, are there any remaining blockers from allowing a program at depth 2+ to reallocate a non-empty account?

solana/programs/bpf_loader/src/syscalls.rs

Lines 2420 to 2423 in 622a6fb

	if !account_ref.data.is_empty() {
	// Only support for `CreateAccount` at this time.
	// Need a way to limit total realloc size across multiple CPI calls
	ic_msg!(

Seems like the issue is that we would need to prevent sequentially invoked CPI's from each bumping the data size?

[jackcmay]

Working on it, looks doable 😁

…

[jackcmay]

No featurization yet but lots of test cases, any other test scenarios folks can think of for account reallocs?

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[jstarry]

This account.data could have already changed in an earlier CPI so we can't trust it as the original data length. The original data length has to come from the time of serializing account parameters.

[jstarry]

@jackcmay I still think we need to zero in this sdk method. If a program deallocs and then reallocs, it shouldn't be already filled with data.

[jackcmay]

@jstarry I'd like to avoid calling memset here, for most it will be a waste of compute units. How about clearly documenting the behavior instead?

[jstarry]

@jstarry I'd like to avoid calling memset here, for most it will be a waste of compute units. How about clearly documenting the behavior instead?

Ah sure, can we have an opt-in bool param to fn realloc called zero_freed_data so that devs have to think about whether they want it?

[jstarry]

orig_data_lens change looks good!

[jstarry]

@jackcmay still working on adding a zero_freed_data param here?

[jackcmay]

@jstarry

Yup, almost done, but question, what do you think of:

pub fn realloc(&self, new_len: usize, zero_init: bool) -> Result<(), ProgramError> {

vs

pub fn realloc(&self, new_len: usize) -> Result<(), ProgramError>
pub fn realloc_unset(&self, new_len: usize) -> Result<(), ProgramError>

[Lichtso]

Looking good so far.
This is only about BPF programs, not native_invoke, right?

pub fn realloc(&self, new_len: usize, zero_init: bool) -> Result<(), ProgramError>

I like the parameter flag better than two methods as you might decide to add even more parameters later on.

Btw, account realloc in ABI v2 is not designed yet and something I would like to discuss on Monday.