Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check bank capitalization #11927

Merged
merged 11 commits into from
Sep 11, 2020
Merged

Conversation

ryoqun
Copy link
Member

@ryoqun ryoqun commented Aug 31, 2020

Problem

  • No check for bank capitalization.

  • Also, the current offline bank capitalization check doesn't catch overflows.

Summary of Changes

On accounts hashing, load the total accounts capitalization and compare with bank.capitalization.

Fixes #

context

reboot of #10937

@ryoqun ryoqun changed the title [wip] Check bank capitalization Check bank capitalization Aug 31, 2020
@ryoqun
Copy link
Member Author

ryoqun commented Sep 1, 2020

btw, because the testnet ledger was restarted, it's now safe to start to enforce this everytime validator starts. (thanks @mvines).

@@ -2843,22 +2845,7 @@ impl Bank {
}

pub fn calculate_capitalization(&self) -> u64 {
self.get_program_accounts(None)
Copy link
Member Author

@ryoqun ryoqun Sep 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been solely used for the ledger-tool capitalization subcommand.

Move to AccountsDB:: and use this from ledger-tool and snapshot codepaths

@codecov
Copy link

codecov bot commented Sep 1, 2020

Codecov Report

Merging #11927 into master will increase coverage by 0.0%.
The diff coverage is 90.0%.

@@           Coverage Diff            @@
##           master   #11927    +/-   ##
========================================
  Coverage    82.0%    82.0%            
========================================
  Files         337      337            
  Lines       79220    79369   +149     
========================================
+ Hits        65015    65160   +145     
- Misses      14205    14209     +4     

runtime/src/accounts_db.rs Outdated Show resolved Hide resolved
@aeyakovenko
Copy link
Member

@ryoqun Let's say we had an inflation bug, like an overflow in the contract, would this catch it? I want to make sure that we can compute the max possible supply in the snapshot and fail if it exceeds it.

@ryoqun
Copy link
Member Author

ryoqun commented Sep 1, 2020

@ryoqun Let's say we had an inflation bug, like an overflow in the contract, would this catch it? I want to make sure that we can compute the max possible supply in the snapshot and fail if it exceeds it.

@aeyakovenko Yeah, I've tested this locally. These catches them as the last protection mechanism.

For both inflation and overflow in contracts, these will be protected by their specific protections, btw. So, I've manually disabled them while testing locally. I've added some sanity assertion:

solana/runtime/src/bank.rs

Lines 932 to 940 in 11ac4eb

if let Some(rewards) = self.rewards.as_ref() {
assert_eq!(
validator_rewards_paid,
u64::try_from(rewards.iter().map(|(_pubkey, reward)| reward).sum::<i64>()).unwrap()
);
}
// verify that we didn't pay any more than we expected to
assert!(validator_rewards >= validator_rewards_paid);
for inflation and this is the protection for contracts:
let (mut pre_sum, mut post_sum) = (0_u128, 0_u128);
{
let program_id = instruction.program_id(&message.account_keys);
let mut work = |unique_index: usize, account_index: usize| {
// Verify account has no outstanding references and take one
let account = accounts[account_index]
.try_borrow_mut()
.map_err(|_| InstructionError::AccountBorrowOutstanding)?;
pre_accounts[unique_index].verify(&program_id, rent, &account)?;
pre_sum += u128::from(pre_accounts[unique_index].lamports());
post_sum += u128::from(account.lamports);
Ok(())
};
instruction.visit_each_account(&mut work)?;
}
// Verify that the total sum of all the lamports did not change
if pre_sum != post_sum {
return Err(InstructionError::UnbalancedInstruction);
}

@ryoqun
Copy link
Member Author

ryoqun commented Sep 5, 2020

@sakridge Could you review this? I have still something to do (add test for overflow; benchmark checked u64 vs u128 for summing) but mostly this should be done.

What I have added to your original pr:

  • Really Inhibit automatic restart at blockstore_processor after capitalization mismatch is detected (otherwise, I observed the cluster would manage to progress while being tripped at every snapshot interval...)
  • Don't miss overflowing capitalization (so that I tweaked the data passing for the verification loops).

@ryoqun ryoqun force-pushed the bank-capitalization branch from 0a64fb0 to d4d1831 Compare September 7, 2020 09:15
@ryoqun
Copy link
Member Author

ryoqun commented Sep 7, 2020

I have still something to do (add test for overflow; benchmark checked u64 vs u128 for summing)

All of these are now done. For benchmarking, I saw no noticeable difference and doing the cap calculation should be no harm because it's so small compared to hashing:

slower:

$ sudo chrt --rr 99 bash -c "RUST_LOG=debug  /tmp/solana-accounts-bench-slower --num_slots 4 --num_accounts 80000 > /tmp/result 2>&1" && cat /tmp/result | grep cap.*took
[2020-09-07T09:19:30.308469764Z DEBUG solana_runtime::accounts_db] sort took 558us hash took 24ms cap took 756us
[2020-09-07T09:19:30.366677331Z DEBUG solana_runtime::accounts_db] sort took 7ms hash took 19ms cap took 740us
[2020-09-07T09:19:30.417183194Z DEBUG solana_runtime::accounts_db] sort took 516us hash took 18ms cap took 786us
[2020-09-07T09:19:30.467481852Z DEBUG solana_runtime::accounts_db] sort took 627us hash took 18ms cap took 775us
[2020-09-07T09:19:30.518508803Z DEBUG solana_runtime::accounts_db] sort took 535us hash took 19ms cap took 764us
[2020-09-07T09:19:30.569019096Z DEBUG solana_runtime::accounts_db] sort took 558us hash took 18ms cap took 804us
[2020-09-07T09:19:30.620938190Z DEBUG solana_runtime::accounts_db] sort took 611us hash took 19ms cap took 760us
[2020-09-07T09:19:30.671447692Z DEBUG solana_runtime::accounts_db] sort took 569us hash took 19ms cap took 777us
[2020-09-07T09:19:30.722204665Z DEBUG solana_runtime::accounts_db] sort took 677us hash took 18ms cap took 769us
[2020-09-07T09:19:30.772587342Z DEBUG solana_runtime::accounts_db] sort took 602us hash took 18ms cap took 749us
[2020-09-07T09:19:30.823144280Z DEBUG solana_runtime::accounts_db] sort took 624us hash took 18ms cap took 719us
[2020-09-07T09:19:30.873514619Z DEBUG solana_runtime::accounts_db] sort took 573us hash took 18ms cap took 785us
[2020-09-07T09:19:30.924470309Z DEBUG solana_runtime::accounts_db] sort took 611us hash took 19ms cap took 777us
[2020-09-07T09:19:30.974693557Z DEBUG solana_runtime::accounts_db] sort took 584us hash took 18ms cap took 757us
[2020-09-07T09:19:31.025382499Z DEBUG solana_runtime::accounts_db] sort took 564us hash took 18ms cap took 751us
[2020-09-07T09:19:31.075546537Z DEBUG solana_runtime::accounts_db] sort took 599us hash took 18ms cap took 739us
[2020-09-07T09:19:31.125940844Z DEBUG solana_runtime::accounts_db] sort took 571us hash took 18ms cap took 766us
[2020-09-07T09:19:31.176431051Z DEBUG solana_runtime::accounts_db] sort took 639us hash took 18ms cap took 763us
[2020-09-07T09:19:31.227311366Z DEBUG solana_runtime::accounts_db] sort took 627us hash took 19ms cap took 748us
[2020-09-07T09:19:31.277557409Z DEBUG solana_runtime::accounts_db] sort took 640us hash took 19ms cap took 796us

faster:

$ sudo chrt --rr 99 bash -c "RUST_LOG=debug  /tmp/solana-accounts-bench-faster --num_slots 4 --num_accounts 80000 > /tmp/result 2>&1" && cat /tmp/result | grep cap.*took
[2020-09-07T09:19:42.578398620Z DEBUG solana_runtime::accounts_db] sort took 709us hash took 24ms cap took 759us
[2020-09-07T09:19:42.636548401Z DEBUG solana_runtime::accounts_db] sort took 7ms hash took 19ms cap took 706us
[2020-09-07T09:19:42.687420942Z DEBUG solana_runtime::accounts_db] sort took 610us hash took 18ms cap took 739us
[2020-09-07T09:19:42.738622939Z DEBUG solana_runtime::accounts_db] sort took 680us hash took 19ms cap took 711us
[2020-09-07T09:19:42.788517799Z DEBUG solana_runtime::accounts_db] sort took 645us hash took 18ms cap took 729us
[2020-09-07T09:19:42.839177491Z DEBUG solana_runtime::accounts_db] sort took 683us hash took 19ms cap took 722us
[2020-09-07T09:19:42.889007691Z DEBUG solana_runtime::accounts_db] sort took 610us hash took 18ms cap took 733us
[2020-09-07T09:19:42.940074238Z DEBUG solana_runtime::accounts_db] sort took 690us hash took 18ms cap took 712us
[2020-09-07T09:19:42.990138219Z DEBUG solana_runtime::accounts_db] sort took 659us hash took 19ms cap took 716us
[2020-09-07T09:19:43.040929840Z DEBUG solana_runtime::accounts_db] sort took 683us hash took 18ms cap took 737us
[2020-09-07T09:19:43.090966708Z DEBUG solana_runtime::accounts_db] sort took 636us hash took 18ms cap took 717us
[2020-09-07T09:19:43.141026689Z DEBUG solana_runtime::accounts_db] sort took 665us hash took 18ms cap took 753us
[2020-09-07T09:19:43.191746274Z DEBUG solana_runtime::accounts_db] sort took 612us hash took 18ms cap took 710us
[2020-09-07T09:19:43.242019966Z DEBUG solana_runtime::accounts_db] sort took 646us hash took 19ms cap took 716us
[2020-09-07T09:19:43.293032927Z DEBUG solana_runtime::accounts_db] sort took 691us hash took 19ms cap took 711us
[2020-09-07T09:19:43.343507060Z DEBUG solana_runtime::accounts_db] sort took 600us hash took 19ms cap took 739us
[2020-09-07T09:19:43.394155445Z DEBUG solana_runtime::accounts_db] sort took 715us hash took 18ms cap took 718us
[2020-09-07T09:19:43.444164476Z DEBUG solana_runtime::accounts_db] sort took 692us hash took 18ms cap took 737us
[2020-09-07T09:19:43.494308321Z DEBUG solana_runtime::accounts_db] sort took 593us hash took 18ms cap took 716us
[2020-09-07T09:19:43.544581596Z DEBUG solana_runtime::accounts_db] sort took 658us hash took 19ms cap took 730us

For completeness and pure curiosity, I've went into details and found that u128 version is slightly (about +1%) faster actually:

https://docs.google.com/spreadsheets/d/1nsnNwkaTQ4pjpQKPwEhYcxhuhU0YCnvYm81Nmvf9zjA/edit#gid=0

@ryoqun ryoqun force-pushed the bank-capitalization branch from f9b58ac to 4ba5a8c Compare September 8, 2020 15:46
@ryoqun
Copy link
Member Author

ryoqun commented Sep 8, 2020

it seems that #11996 has worked. testnet stopped to overflow.

@sakridge But, now this is blocked by #12112

@ryoqun
Copy link
Member Author

ryoqun commented Sep 8, 2020

Instead of #12112, #12111 could unblock this pr more faster....

runtime/src/accounts.rs Outdated Show resolved Hide resolved
@ryoqun ryoqun requested a review from sakridge September 11, 2020 10:02
@ryoqun
Copy link
Member Author

ryoqun commented Sep 11, 2020

@sakridge Thanks for reviewing. I think I've addressed all your review comments.

Copy link
Member

@sakridge sakridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks really good, thanks!

@ryoqun ryoqun removed the v1.2 label Sep 11, 2020
@ryoqun ryoqun merged commit de4a613 into solana-labs:master Sep 11, 2020
mergify bot pushed a commit that referenced this pull request Sep 11, 2020
* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <[email protected]>
(cherry picked from commit de4a613)

# Conflicts:
#	Cargo.lock
#	accounts-bench/Cargo.toml
mergify bot added a commit that referenced this pull request Sep 11, 2020
* Check bank capitalization (#11927)

* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <[email protected]>
(cherry picked from commit de4a613)

# Conflicts:
#	Cargo.lock
#	accounts-bench/Cargo.toml

* Fix conflict 1/2

* Fix conflict 2/2

Co-authored-by: Ryo Onodera <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants