svm: interleave transaction validation and processing

[2501babe]

Problem

simd83 intends to relax entry-level constraints, namely that a transaction which takes a write lock on an account cannot be batched with any other transaction which takes a read or write lock on it

before the locking code can be changed, however, svm must be able to support such batches. presently, if one transaction were to write to an account, and a subsequent transaction read from or wrote to that account, svm would not pass the updated account state to the second transaction; it gets them from accounts-db, which is not updated until after all transaction execution finishes

previously, in #3146, we attempted to implement a jit account loader with intermediate account cache to pass updated accounts forward to future transaction in the batch. however, due to existing patterns in how the program cache is used, this has proven quite difficult to get right, and the volume of code changes required is quite large

Summary of Changes

this pr extracts all code from #3146 which does not modify behavior. we would like to commit it standalone for ease of review, and focus specifically on the cache behaviors in a following pr

the existing transaction processing loop in master validates all transaction fee-payers together, loads all transaction accounts together, and then executes each transaction in serial. the new transaction processing loop validates one fee-payer, loads accounts for one transaction, and then executes that transaction before proceeding to the next. this prepares us for an svm where accounts can change in between transactions

we also validate nonce accounts before each transaction, because these accounts can also change in the future svm. a future pr may choose to eliminate most nonce validation code from the check_transactions, but that is out of scope here

futhermore, we create a new type, AccountLoader, which encapsulates the batch-local program cache, account overrides, and the accounts-db callback. it provides the method load_account, which is opaque to its caller, returning a LoadedTransactionAccount according to the exact same rules as the current load_transaction_account

this pr changes nothing about account loading, but introducing AccountLoader prepares us to add the internal account cache in support of simd83

[2501babe]

the first commit is the two files copied exactly from the previous pr, and the second commit deletes the intermediate account cache. this way you can look at the full changeset, or the difference between what you already reviewed, easily

[jstarry]

This change has the side effect of collecting rent from writable accounts that come from account overrides. Probably fine? I'm not too sure yet

[2501babe]

this was intentional:

• the current code actually collects rent from an override if its the fee-payer, otherwise it doesnt. so this is at least consistent
• with simd83, we have to consider what happens to a writable overridden account after the first transaction: does it stay frozen, and all changes to it are ignored? or does it behave like an ordinary account that happens to have an overridden starting value? the latter is more in line with its current behavior, where it is freely allowed to change in the transaction, and then the mutated value is retained in the LoadedTransactionAccount returned from the svm entrypoint
• if we let an overriden account change intrabatch then rent collection should happen on first use, otherwise we collect on second use, which is odd
• account overrides is only intended to be used for simulation. actually giving it a writable account for a real execution would be highly unwise because it would make its way all the way back to accounts-db to be committed

in fact, account overrides is an overly general mechanism for transaction simulation to pass svm a SlotHistory sysvar from the previously frozen bank, and nothing else. andrew was suggesting we could change it if/when we change the load_and_execute_sanitized_transactions interface

[jstarry]

Great breakdown, completely onboard now

[jstarry]

nit: can you introduce a new function called validate_transaction which wraps validate_transaction_nonce and validate_transaction_fee_payer? There's a lot of nesting here that makes the code hard to read IMO

[jstarry]

I love that all the inspect_account call sites are inside AccountLoader now

[2501babe]

im putting this back in draft because alexander and alessandro are planning on fundamentally changing how the program cache interacts with account loading and putting it in 2.1 so we need to coordinate with them

[2501babe]

ok. this will be back in review soon. i spent a while talking with them and alexander thought he had an elegant fix to decouple account loading from program cache but it is now scuttled due to the bugs in how the account loader uses the program cache. but he is making some changes to the program cache tests that i want to rebase on

[2501babe]

once i rebase on alexander, i am removing this get_account_shared_data() call:

• its wrong. the changes to the flow means if it were ever hit, account loading would believe there is a new account to create rather than aborting
• it is impossible to ever trigger because program cache is built directly from accounts-db

alexander is removing the test that artificially triggers this condition

[2501babe]

instead of following the "pub first priv after" rule which made things hard to remember, the fields are in the order of load priority, overrides > program cache > accounts-db, with constructor args in the same order

the next pr immediately after this one adds account cache and removes overrides from the struct. when disable_account_loader_special_case activates we also remove the two program cache items from the struct

[2501babe]

ok! this is now ready for review again and i think we can finish all the account loader work next week. nothing i was trying to do in this pr has changed conceptually, but i had to rebase on five or six different commits that all changed the underlying code in master, so another look through with fresh eyes is necessary

nothing in this pr is intended to affect behavior at all (other than rent collection on overrides, which does not affect anything since in the next pr we will enforce that SlotHistory is the only allowable override). it is simply:

• refactors to introduce AccountLoader, which in the next pr will contain an account cache
• refactors to interleave transaction processing tasks. ie, before it was "validate all fee-payers, load all transactions, execute all transactions" and now it is "validate one fee-payer, load one transaction, execute it, repeat"
• nonce validation, which essentially duplicates batch-level checks, most of which can safely be removed at a later date

this pr will be immediately followed by #3453, which is based on this one and can be made ready for review once this one lands. that pr adds the account cache and fully supports simd83 account loading. it is also delightfully small, i think the actual code changes are like 100 lines and the vast majority is new tests

i will also do a third pr getting loaders out of the account keys without changing size calculations, but thats basically just a cleanup

to recap our exciting journey since this was put back into draft last week:

• alexander was hoping to decouple account loading from the program cache in 2.1 on monday, but this fell through due to transaction sizes depending on tombstone logic
• we rebased on two commits changing transaction size unit tests to use the program cache more correctly
• we rebased on two commits that mess around with how lamports_per_signature is used for fee-payer validation
• we rebased on a commit which adds feature remove_accounts_executable_flag_checks, which will eventually disable checking the execute bit on programs
• we rebased on a commit which reverts program cache owner checking logic to how it was in 1.18, which is anything using the (old) new program_accounts hashmap
• we rebased on a commit which adds the feature disable_account_loader_special_case which will get the program cache out of account loading once and for all

the merges were fairly involved and i have looked at the code itself and diffs against both master and the previous state of this branch to be sure i didnt misplace anything, but yea. i think we will be out of the weeds into greener pastures very soon