Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add filler accounts to bloat validator and predict failure #20491

Merged
merged 8 commits into from
Oct 11, 2021

Conversation

jeffwashington
Copy link
Contributor

@jeffwashington jeffwashington commented Oct 6, 2021

Problem

For ledger-tool verify or a validator, you can specify --accounts-filler-count 10000000 and we will add 10M dummy accounts in the append vecs and acct idx to clog things up. These accounts will not be used to calculate hashes and they won't have rent collected from them. But, they will bloat the appendvecs, acct idx, rent collection and all other algorithms while running against mnb or whatever. The idea is that if we keep a validator like this running with extra accounts, then when it ooms or otherwise fails, it gives us an indication that other validators may be nearing failure.

Note this currently only works when starting from a snapshot.

Summary of Changes

Fixes #

@jeffwashington jeffwashington changed the title Canary add filler accounts to bloat validator and predict failure Oct 6, 2021
@jeffwashington jeffwashington requested a review from carllin October 6, 2021 23:21
@jeffwashington
Copy link
Contributor Author

@carllin - I just got this working fully. Maybe some cosmetic things wrong with it. Please take a look so I can get pass 1 of your feedback. One variation: I considered making these dummy accounts 'executable' so that we don't collect rent from them and I don't have to plumb through pubkey checking.

@jeffwashington
Copy link
Contributor Author

runtime/src/accounts_db.rs Outdated Show resolved Hide resolved
@@ -6645,7 +6792,9 @@ impl AccountsDb {
for slot_stores in self.storage.0.iter() {
for (id, store) in slot_stores.value().read().unwrap().iter() {
// Should be default at this point
assert_eq!(store.alive_bytes(), 0);
if self.filler_account_count == 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might also be good to sanity check in generate_index that none of the accounts in the snapshot match the filler_account_suffix

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this if was left over from previous attempts and has been removed.
I added an assert

runtime/src/accounts_db.rs Outdated Show resolved Hide resolved
runtime/src/accounts_db.rs Outdated Show resolved Hide resolved
runtime/src/accounts_db.rs Outdated Show resolved Hide resolved
@@ -1972,6 +1992,10 @@ impl AccountsDb {
let mut purges_zero_lamports = HashMap::new();
let mut purges_old_accounts = Vec::new();
for pubkey in pubkeys {
if self.is_filler_account(pubkey) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm when rent runs and updates these accounts, won't we need to run clean in order to remove the old update from the old storage entries?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I think you're right. Not sure why I left this in there.

Comment on lines +5611 to +5619
if self.is_filler_account(loaded_account.pubkey()) {
None
} else {
// Cache only has one version per key, don't need to worry about versioning
Some((*loaded_account.pubkey(), loaded_account.loaded_hash()))
}
},
|accum: &DashMap<Pubkey, (u64, Hash)>, loaded_account: LoadedAccount| {
let loaded_write_version = loaded_account.write_version();
let loaded_hash = loaded_account.loaded_hash();
if self.is_filler_account(loaded_account.pubkey()) {
return;
}
Copy link
Contributor

@carllin carllin Oct 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, glad to see this worked for getting around the updates caused by rent.

@codecov
Copy link

codecov bot commented Oct 10, 2021

Codecov Report

Merging #20491 (c51b88b) into master (e030221) will decrease coverage by 0.0%.
The diff coverage is 66.1%.

@@            Coverage Diff            @@
##           master   #20491     +/-   ##
=========================================
- Coverage    82.0%    82.0%   -0.1%     
=========================================
  Files         494      494             
  Lines      137478   137611    +133     
=========================================
+ Hits       112786   112871     +85     
- Misses      24692    24740     +48     

@jeffwashington jeffwashington marked this pull request as ready for review October 11, 2021 01:03
@@ -6426,7 +6427,9 @@ impl AccountsDb {
let accounts = storage.all_accounts();
accounts.into_iter().for_each(|stored_account| {
let this_version = stored_account.meta.write_version;
match accounts_map.entry(stored_account.meta.pubkey) {
let pubkey = stored_account.meta.pubkey;
assert!(!self.is_filler_account(&pubkey));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also sanity checked that this check only runs when the filler accounts are passed in, so shouldn't trigger during normal validator operation.

@jeffwashington jeffwashington merged commit a8e000a into solana-labs:master Oct 11, 2021
dankelleher pushed a commit to identity-com/solana that referenced this pull request Nov 24, 2021
…bs#20491)

* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
frits-metalogix added a commit to identity-com/solana that referenced this pull request Nov 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants