This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Suggestions for #9081 (Store voters in unsorted bags) #9328
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
A0-please_review
5 tasks
kianenigma
reviewed
Jul 12, 2021
kianenigma
reviewed
Jul 12, 2021
kianenigma
reviewed
Jul 12, 2021
frame/staking/src/voter_bags.rs
Outdated
Comment on lines
471
to
474
| debug_assert!( | ||
| prev.prev().is_some() || self.head.as_ref() == Some(&prev.voter.id), | ||
| "node.prev.prev should be Some OR node.prev should be the head" | ||
| ); |
There was a problem hiding this comment.
What if I ask you to check that pre.prev().prev().is_some() is Some or tail?
I think what we should do is to write a
fn sanity_check_voter_list() {
// iterate all voters and ensure only those that are head don't have a prev
// iterate all voters and ensure only those that are tail don't have a next
// iterate all voters and ensure that there are no loops
// iterate all voters and ensure their count is in sync with `VoterCount`
// ensure VoterCount itself is always `== CounterForValidators + CounterForNominators`
// anything else..
}
and sprinkle this as debug assertions after each mutating operation on the public interface of the bags module.
Probably we can retire some existing ones in favour of this.
kianenigma
reviewed
Jul 12, 2021
kianenigma
reviewed
Jul 12, 2021
kianenigma
reviewed
Jul 12, 2021
kianenigma
reviewed
Jul 12, 2021
frame/staking/src/voter_bags.rs
Outdated
| @@ -498,7 +536,10 @@ pub struct Node<T: Config> { | |||
| impl<T: Config> Node<T> { | |||
| /// Get a node by bag idx and account id. | |||
| pub fn get(bag_upper: VoteWeight, account_id: &AccountIdOf<T>) -> Option<Node<T>> { | |||
| // debug_assert!(bag_upper is in Threshold) | |||
| // debug_assert!( // TODO: figure out why this breaks test take_works | |||
kianenigma
approved these changes
Jul 12, 2021
There was a problem hiding this comment.
We can merge this PR into the base once the comments are addressed. We need more testing PRs on top of #9081, but preferably let's merge them frequent to avoid merge issues.
Co-authored-by: Kian Paimani <[email protected]>
emostov
commented
Jul 18, 2021
| @@ -0,0 +1,75 @@ | |||
| use remote_externalities::{Builder, Mode, OnlineConfig}; | |||
There was a problem hiding this comment.
Right now I am planning on moving these tests to the polkadot repo so they are concrete over the exact runtime types.
Polkadot branch name: zeke-companion-for-9081-tests
There was a problem hiding this comment.
Will need to remove the new deps in Cargo.toml as well
emostov
commented
Jul 19, 2021
frame/staking/src/voter_bags.rs
Outdated
| // maybe_len.unwrap_or_default() as u32, | ||
| // crate::CounterForNominators::<T>::get() + crate::CounterForValidators::<T>::get(), | ||
| // "voter count must be sum of validator and nominator count", | ||
| // ); |
There was a problem hiding this comment.
needs fixing. Any idea why?
emostov
commented
Jul 19, 2021
|
|
||
| // TODO bags do not get cleaned up from storages | ||
| // - is this ok? I assume its ok if this is not cleaned just because voters are removed | ||
| // but it should be cleaned up if we migrate thresholds |
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
kianenigma
reviewed
Jul 20, 2021
| ExtBuilder::default().validator_pool(true).build_and_execute(|| { | ||
| // initialize the voters' deposits | ||
| let mut balance = 0; // This will be 10 on the first loop iteration because 0 % 3 == 0 | ||
| for (idx, voter_id) in GENESIS_VOTER_IDS.iter().enumerate() { |
There was a problem hiding this comment.
why are we doing this? GENESIS_VOTER_IDS should already be in the correct bag.
There was a problem hiding this comment.
I am thinking its easier to read the test when you can see up front how they are distributed in the bags. I need to double check how the genesis voters are distributed, but the scheme here guarantees that the last node taken is mid-bag, which is an edge case we talked about explicitly testing
There was a problem hiding this comment.
I see your point, but when you have 100+ tests, it is quite verbose to setup things manually for each tests. Instead, it is more reasonable to assume some fixed genesis stakers (and the programmer should simply have this in their mind) and write all tests preferably based on that.
ghost
pushed a commit
that referenced
this pull request
Sep 17, 2021
…#9507) * remove extra whitespace Co-authored-by: Kian Paimani <[email protected]> * only emit rebag event on success * add doc explaining the term voter * revamp/simplify rebag test * ensure genesis accounts are placed into the correct nodes/bags * bond_extra implicitly rebags * types at top; doc public type * start sketching out adjustable thresholds * add integrity test for voter bag threshold requirements * get rid of BagIdx This reorganizes bag storage such that bags are always referred to by their upper threshold. This in turn means that adding and removing bags is cheaper; you only need to migrate certain voters, not all of them. * implement migration logic for when the threshold list changes * start sketching out threshold proc macros * further refine macro signatures * WIP: implement make_ratio macro * start rethinking the process of producing threshold lists The macro approach seems to be a non-starter; that only really works if we're throwing around numeric literals everywhere, and that's just not nice in this case. Instead, let's write helper functions and make it really easy to generate the tables in separate, permanent files, which humans can then edit. * write helper functions to emit voter bags module * WIP: demo generating voter bags for a realistic runtime This isn't yet done, becuase it seems to take a Very Long Time to run, and it really shouldn't. Need to look into that. Still, it's a lot closer than it was this morning. * rm unnecessary arg_enum * fix voter bags math Turns out that when you're working in exponential space, you need to divide, not subtract, in order to keep the math working properly. Also neaten up the output a little bit to make it easier to read. * add computed voter bags thresholds to node * fixup some docs * iter from large bags to small, fulfuilling the contract * make tests compile * add VoterBagThresholds to some configs * ensure that iteration covers all voters even with implied final bag * use sp_std::boxed::Box; * fix unused import * add some more voter bags tests * file_header.txt * integrity test to ensure min bag exceeds existential weight * add more debug assertions about node list length * rm unused imports * Kian enters * Update frame/election-provider-support/src/onchain.rs Co-authored-by: Zeke Mostov <[email protected]> * Suggestions for #9081 (Store voters in unsorted bags) (#9328) * Add some debug asserts to node::get and remove_node * Improve the debug asserts in remove_node * improve debug asserts * Space * Remove bad assertions * Tests: WIP take_works * Take test * Doc comment * Apply suggestions from code review Co-authored-by: Kian Paimani <[email protected]> * Test storage is cleaned up; * formatting * Switch to simpler thresholds * Update the storage cleanup test * Remove hardcoded values from benchmark to make it more robust * Fix tests to acces bags properly * Sanity check WIP; tests failing * Update sanity checks to be more correct * Improve storage cleanup tests * WIP remote_ext_tests * Some notes on next steps * Remove some stuff that was for remote-ext tests * Some more cleanup to reduce diff * More :clean: * Mo cleanin * small fix * A lot of changes from kian Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: kianenigma <[email protected]> * merge fallout * Run cargo +nightly fmt * Fix a bunch of stuff, remove not needed runtime arg of make-bags * add logs * Glue the new staking bags to the election snapshot (#9415) * Glue the new staking bags to the election snapshot * add CheckedRem (#9412) * add CheckedRem * fix * Run fmt * Test comment Co-authored-by: Xiliang Chen <[email protected]> Co-authored-by: emostov <[email protected]> * Update node runtime with VoterSnapshotPerBlock * Unit test for pallet-staking unsorted bags feature (targets #9081) (#9422) * impl notional_bag_for_works * Add tests: insert_as_works & insert_works * Impl test: remove_works * Trivial cleaning * Add test: update_position_for_works * Write out edge case; probably can delete later * Add test: bags::get_works * Add test: remove_node_happy_path_works * Add test: remove_node_bad_paths_documented * WIP: voting_data_works * done * Improve test voting_data_works * Add comment * Fill out test basic_setup_works * Update: iteration_is_semi_sorted * Improve remove_works * Update update_position_for_works; create set_ledger_and_free_balance * Improve get_works * Improve storage clean up checks in remove test * Test: impl rebag_works + insert_and_remove_works * forgot file - Test: impl rebag_works + insert_and_remove_works * Small tweak * Update voter_bags test to reflect unused bags are removed * Unbond & Rebond: do_rebag * Prevent infinite loops with duplicate tail insert * Check iter.count on voter list in pre-migrate * undo strang fmt comment stuff * Add in todo Co-authored-by: kianenigma <[email protected]> * Try prepare for master merge * Reduce diff * Add comment for test to add * Add in code TODO for update_position efficiency updates * Initial impl compiles * impl StakingVoterListStub * Sample impl<T: Config> VoterListProvider<T> for VoterList<T> * impl VoterListProvider for voter-bags * WIP integrate pallet-voter-bags to staking mock * the trait `pallet_staking::pallet::pallet::Config` is not implemented for `mock::Test` * random * pushing my stuff * Mock working * WIP voter list tests * Add bag insert, remove tests * Add test for bag insert and remove * Add remaining tests for VoterList * Add tests for node * Add rebag works * Add rebag extrinsic tests * Rename to bags-list and name the list .. list! * Rename VoterBagThresholds => BagThresholds * Add test count_works * Test on_update_works * test sanity check * a round of test fixes * push a lot of changes * my last changes * all bags-list test work; fmt * Beautify some tests * Doc comment for bags-list * Add insert warnings * Setup initial benchmark * Wire up WeightInfo * is_terminal wip; everything broken! * Is terminal working * add TODOs for remove_node * clean up remoe_node * Fix all staking tests * retire VoterBagFor * commit * bring in stashed changes * save * bench pipeline works now, but I can't run stuff * sabe * benchmarks now run, but we have a failure * WIP: Wire up make_bags * bags-thresholds compiles * Fix most build issues * This will fix all the tests * move bag thresholds to bags-list * Move bag-thresholds bin to within pallet-bags * Remove some unnescary TODOs * Impl tets wrong_rebag_is_noop * assert remove is a noop with bad data * Assert integrity test panics * Return an error when inserting duplicates * Update to handle error in staking pallet when inserting to list * Test contains and on_insert error * Test re-nominate does not mess up list or count * Everything builds and works, only the benchmark... * fuck yeah benchmarks * more cleanup, more hardening. * use the bags list again * fix benhc * Some questions and changs for List::migration * Fix migration removed_bags and new_bags usage * Some trivial aesthetic changes * Some more trivial changes * tiny changes/ * mega rename * fix all tests and ci build * nit * Test and fix migration * nit * fmt * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fmt * remove unused * make a few things pub * make node also pub.. for remote-ext test * Fix all tests again * Force bag changes in relevant benchmarks (targets #9507) (#9529) * force rebag for unbond, rebond, and bond_extra * nit * Improve utils * fmt * nits * Move generate_bags to its own pallet * Get runtime-benchmarks feature setup with prepare_on_update_benchmark * Withdraw unbonded kill working * Nominate bench working * some cleanup * WIP * update to check head pre & post conditions * Add some post condition verification stuff for on_remove * Update nominate * fmt * Improvements * Fix build * fix build with polkadot companion * Update frame/bags-list/src/list/tests.rs Co-authored-by: Kian Paimani <[email protected]> * move generate-bag from frame to utils * wip * refactor WIP * WIP save * Refactor working * some variable renaming * WIP: prepare to remove head checks * Finish MvP refactor * Some cleanup * Soem more cleanup * save * fix a lot of stuff * Update client/db/src/bench.rs Co-authored-by: Shawn Tabrizi <[email protected]> * Apply suggestions from code review * Apply suggestions from code review * Fix some issues that came from trying to merge comments on github * some small changes * simplify it Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: kianenigma <[email protected]> Co-authored-by: Shawn Tabrizi <[email protected]> * Build works * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <[email protected]> * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <[email protected]> * Remove commented out debug assert * Remove some unused deps and some unused benchmarking stuff * Fix stakings ElectionDataProvider clear * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Improving staking pallet-bags-list migration * fix build and some comments; * comment * Reduce visibility in bags list components * make node.bag_upper only accesible to benchmarks * Address some feedback; comments updates * use nominator map comment * fix vec capacity debug assert * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <[email protected]> * clarify VoterSnapshotPerBlock * Reduce diff on create_validators by wrapping with_seed * Some small improvements to staking benches * Soem comment updates * fix vec capacity debug assert ... for real this time * Reduce ListBags viz * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Remove supports_eq_unordered & Support eq_unordered * Update utils/frame/generate-bags/src/lib.rs Co-authored-by: Guillaume Thiolliere <[email protected]> * Make total-issuance & minimium-balance CLI args; Dont use emptry ext * Improve docs for generate bags CLI args * Apply suggestions from code review Co-authored-by: Kian Paimani <[email protected]> * Apply suggestions from code review Co-authored-by: Kian Paimani <[email protected]> * Don't use default bags weight in node runtime * Feature gating sanity_check not working * Feature gate sanity check by creating duplicate fns * Fix line wrapping * Document VoteWeightProvider * Make bags ext-builder not a module * Apply suggestions from code review Co-authored-by: Kian Paimani <[email protected]> * use pallet_bags_list instead of crate in mock * Make get_bags test helper fn live in List * use remove_from_storage_unchecked for node removal * Remove count of ids removed in remove_many * Add node sanity check, improve list sanity check * Do a list sanity check after on_update * List::migrate: clean up debug assert, exit early when no change in thresholds * Improve public doc comments for pallet_bags_list::list::List * Improve public doc comments for pallet_bags_list::list::List * Update generate bags docs * Fix grammar in bags-list benchmark * Add benchmark case for `rebag` extrinsic * Add count parameter to List::clear; WIP adding MaxEncodedLen to list' * MaxEncodeLen + generate_storage_info not working for Bag or Node * Get MaxEncodeLen derive to work * Try to correctly feature gate SortedListProvider::clear * Use u32::MAX, not u32::max_value * Get up to nominators_quota noms * SortedListProvider::clear takes an Option * Eplicitly ignore SortedListProvider return value * Fix doc comment * Update node-runtime voter snapshot per block * Add test get_max_len_voters_even_if_some_nominators_are_slashed * Add test only_iterates_max_2_times_nominators_quota * Fix generate bags cargo.toml * use sp_std vec * Remove v8 migration hooks from pallet-staking * Update npos trait * Try respect line width * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Update frame/bags-list/src/benchmarks.rs * Unwrap try-runtime error; remove sortedlistprovider pre upgrade len check * trigger ci * restore * trigger ci * restore * trigger ci * revert * trigger ci * revert Co-authored-by: Peter Goodspeed-Niklaus <[email protected]> Co-authored-by: Kian Paimani <[email protected]> Co-authored-by: Peter Goodspeed-Niklaus <[email protected]> Co-authored-by: kianenigma <[email protected]> Co-authored-by: Xiliang Chen <[email protected]> Co-authored-by: Parity Benchmarking Bot <[email protected]> Co-authored-by: Shawn Tabrizi <[email protected]> Co-authored-by: Guillaume Thiolliere <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
target branch: prgn-nominator-unsorted-bags
target pr: #9081
cc @kianenigma