Extrinsic to restore corrupt staking ledgers #3706

gpestana · 2024-03-14T22:42:51Z

This PR adds a new extrinsic Call::restore_ledger gated by StakingAdmin origin that restores a corrupted staking ledger. This extrinsic will be used to recover ledgers that were affected by the issue discussed in #3245.

The extrinsic will re-write the storage items associated with a stash account provided as input parameter. The data used to reset the ledger can be either i) fetched on-chain or ii) partially/totally set by the input parameters of the call.

In order to use on-chain data to restore the staking locks, we need a way to read the current lock in the balances pallet. This PR adds a InspectLockableCurrency trait and implements it in the pallet balances. An alternative would be to tightly couple staking with the pallet balances but that's inelegant (an example of how it would look like in this branch).

More details on the type of corruptions and corresponding fixes https://hackmd.io/DLb5jEYWSmmvqXC9ae4yRg?view#/

We verified that the Call::restore_ledger does fix all current corrupted ledgers in Polkadot and Kusama. You can verify it here https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA.

Changes introduced

Adds Call::restore_ledger extrinsic to recover a corrupted ledger;
Adds trait frame_support::traits::currency::InspectLockableCurrency to allow external pallets to read current locks given an account and lock ID;
Implements the InspectLockableCurrency in the pallet-balances.
Adds staking locks try-runtime checks (Add try-state check for staking locks #3751)

Todo

benchmark Call::restore_ledger
throughout testing of all ledger recovering cases
consider adding the staking locks try-runtime checks to this PR (Add try-state check for staking locks #3751)
simulate restoring all ledgers (https://hackmd.io/Dsa2tvhISNSs7zcqriTaxQ?view) in Polkadot and Kusama using chopsticks -- https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA

Related to #3245
Closes #3751

gpestana · 2024-03-21T10:37:25Z

bot bench substrate-pallet --pallet=pallet_staking

…=dev --target_dir=substrate --pallet=pallet_staking

command-bot · 2024-03-27T02:38:20Z

@gpestana Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5668119 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5668119/artifacts/download.

Ank4n · 2024-03-27T09:45:46Z

substrate/frame/staking/src/pallet/impls.rs

-	/// storage.
+	/// * The ledger's controller and stash matches the associated `Bonded` tuple.
+	/// * Staking locked funds for every bonded stash should be the same as its ledger's total.
+	/// * Staking ledger and bond are not corrupted.
 	fn check_ledgers() -> Result<(), TryRuntimeError> {


How is the running time for this try state check? Don't mind putting this in release for now if it is not super long but want to point out that you are reading some storages multiple times and it can be optimized to be much faster.
Ideally we read any storage only once and keep it in memory for further checks.

substrate/frame/staking/src/pallet/mod.rs

gpestana · 2024-03-27T11:01:27Z

bot bench polkadot-pallet --runtime=westend --pallet=pallet_staking

command-bot · 2024-03-27T11:01:32Z

@gpestana https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5671993 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 2-d8bd9239-bd33-4fcf-bf1b-3829eddd07dc to cancel this command or bot cancel to cancel all commands in this pull request.

…nto HEAD

…=westend --target_dir=polkadot --pallet=pallet_staking

command-bot · 2024-03-27T12:12:06Z

@gpestana Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=westend --target_dir=polkadot --pallet=pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5671993 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5671993/artifacts/download.

Backport for 1.7: - #3639 - #3706 Relevant Issues: - #3245

Backports for 1.7: - #3639 - #3706 Relevant Issues: - #3245

This PR adds a new extrinsic `Call::restore_ledger ` gated by `StakingAdmin` origin that restores a corrupted staking ledger. This extrinsic will be used to recover ledgers that were affected by the issue discussed in paritytech#3245. The extrinsic will re-write the storage items associated with a stash account provided as input parameter. The data used to reset the ledger can be either i) fetched on-chain or ii) partially/totally set by the input parameters of the call. In order to use on-chain data to restore the staking locks, we need a way to read the current lock in the balances pallet. This PR adds a `InspectLockableCurrency` trait and implements it in the pallet balances. An alternative would be to tightly couple staking with the pallet balances but that's inelegant (an example of how it would look like in [this branch](https://github.com/paritytech/polkadot-sdk/tree/gpestana/ledger-badstate-clean_tightly)). More details on the type of corruptions and corresponding fixes https://hackmd.io/DLb5jEYWSmmvqXC9ae4yRg?view#/ We verified that the `Call::restore_ledger` does fix all current corrupted ledgers in Polkadot and Kusama. You can verify it here https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA. **Changes introduced** - Adds `Call::restore_ledger ` extrinsic to recover a corrupted ledger; - Adds trait `frame_support::traits::currency::InspectLockableCurrency` to allow external pallets to read current locks given an account and lock ID; - Implements the `InspectLockableCurrency` in the pallet-balances. - Adds staking locks try-runtime checks (paritytech#3751) **Todo** - [x] benchmark `Call::restore_ledger` - [x] throughout testing of all ledger recovering cases - [x] consider adding the staking locks try-runtime checks to this PR (paritytech#3751) - [x] simulate restoring all ledgers (https://hackmd.io/Dsa2tvhISNSs7zcqriTaxQ?view) in Polkadot and Kusama using chopsticks -- https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA Related to paritytech#3245 Closes paritytech#3751 --------- Co-authored-by: command-bot <>

Polkadot-Forum · 2024-08-26T15:51:02Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/recover-corrupted-staking-ledgers-in-polkadot-and-kusama/9796/1

gpestana added 2 commits March 14, 2024 17:42

init

03d821c

adds clean bad state bond

68df5d2

gpestana self-assigned this Mar 14, 2024

gpestana marked this pull request as draft March 14, 2024 22:42

gpestana requested a review from Ank4n March 14, 2024 22:43

gpestana added 12 commits March 14, 2024 23:47

adds more checks to the test

fc57112

do not clean validator ledgers

4038ca1

Adds non-destructive ledger recover call

af010de

Rethinks the clean/reset extrinsics

87ad1e5

Adds InspectLockableCurrency and impls it for Balances pallet

03cf845

uses InspectLockableCurrency in staking pallet to fetch lock of a stash

9eea773

Adds simple tests for reset ledger

fbbf7f8

removes call::force_clean and replaces it with fast_unstake

64cbee1

nits

a6cb392

nits2

8f52d0a

removes clean ledger

c3ed2c3

remove unecessary CannotForceCleanLedger error variant

11920e4

gpestana changed the title ~~wip: Extrinsic to clean corrupted ledger~~ Extrinsic to recover corrupted ledger Mar 21, 2024

gpestana changed the title ~~Extrinsic to recover corrupted ledger~~ Extrinsic to recover corrupt staking ledgers Mar 21, 2024

gpestana requested a review from kianenigma March 21, 2024 09:52

gpestana added the T1-FRAME This PR/Issue is related to core FRAME, the framework. label Mar 21, 2024

gpestana marked this pull request as ready for review March 21, 2024 09:54

gpestana requested a review from a team as a code owner March 21, 2024 09:54

gpestana added the D1-medium Can be fixed by a coder with good Rust knowledge but little knowledge of the codebase. label Mar 21, 2024

Merge branch 'master' into gpestana/ledger-badstate-clean

42525dc

gpestana added T2-pallets This PR/Issue is related to a particular pallet. and removed T1-FRAME This PR/Issue is related to core FRAME, the framework. D1-medium Can be fixed by a coder with good Rust knowledge but little knowledge of the codebase. labels Mar 21, 2024

Adds benchmarks

c9aba77

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

9426202

…=dev --target_dir=substrate --pallet=pallet_staking

Ank4n reviewed Mar 27, 2024

View reviewed changes

substrate/frame/staking/src/pallet/mod.rs Show resolved Hide resolved

Adds new total checked balance

10fc869

paritytech-review-bot bot requested a review from a team March 27, 2024 10:46

command-bot added 2 commits March 27, 2024 11:03

Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…

f27ef5b

…nto HEAD

".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…

a3100f8

…=westend --target_dir=polkadot --pallet=pallet_staking

kianenigma approved these changes Mar 27, 2024

View reviewed changes

gpestana added a commit that referenced this pull request Mar 27, 2024

Backport staking pallet fix and recover call

07a6733

Backport for 1.7: - #3639 - #3706 Relevant Issues: - #3245

gpestana mentioned this pull request Mar 27, 2024

Backport staking pallet fix and recover call #3869

Closed

Ank4n approved these changes Mar 27, 2024

View reviewed changes

gpestana added this pull request to the merge queue Mar 27, 2024

Ank4n mentioned this pull request Mar 27, 2024

Restore ledger with squashing of ledgers. #3868

Closed

Merged via the queue into master with commit bbdbeb7 Mar 27, 2024
133 of 135 checks passed

gpestana deleted the gpestana/ledger-badstate-clean branch March 27, 2024 17:44

gpestana added a commit that referenced this pull request Mar 27, 2024

Backport #3706

b5cd792

gpestana mentioned this pull request Mar 27, 2024

Release crates io v1.7.0 staking backports #3871

Merged

bkchr pushed a commit that referenced this pull request Mar 28, 2024

Release crates io v1.7.0 staking backports (#3871)

297f545

Backports for 1.7: - #3639 - #3706 Relevant Issues: - #3245

Ank4n mentioned this pull request Aug 4, 2024

Payees lazy migration to PayoutDestination, Split variant for payouts, remove Controller #1196

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extrinsic to restore corrupt staking ledgers #3706

Extrinsic to restore corrupt staking ledgers #3706

gpestana commented Mar 14, 2024 •

edited

Loading

gpestana commented Mar 21, 2024

command-bot bot commented Mar 27, 2024

Ank4n Mar 27, 2024

gpestana commented Mar 27, 2024

command-bot bot commented Mar 27, 2024 •

edited

Loading

command-bot bot commented Mar 27, 2024

Polkadot-Forum commented Aug 26, 2024

Extrinsic to restore corrupt staking ledgers #3706

Extrinsic to restore corrupt staking ledgers #3706

Conversation

gpestana commented Mar 14, 2024 • edited Loading

gpestana commented Mar 21, 2024

command-bot bot commented Mar 27, 2024

Ank4n Mar 27, 2024

Choose a reason for hiding this comment

gpestana commented Mar 27, 2024

command-bot bot commented Mar 27, 2024 • edited Loading

command-bot bot commented Mar 27, 2024

Polkadot-Forum commented Aug 26, 2024

gpestana commented Mar 14, 2024 •

edited

Loading

command-bot bot commented Mar 27, 2024 •

edited

Loading