Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SDK changes break simulator #227

Closed
mpoke opened this issue Jul 12, 2022 · 4 comments
Closed

SDK changes break simulator #227

mpoke opened this issue Jul 12, 2022 · 4 comments
Assignees
@mpoke @danwt
Labels
scope: cosmos-sdk Integration with Cosmos SDK type: bug Issues that need priority attention -- something isn't working

Comments

@mpoke
Copy link
Contributor

mpoke commented Jul 12, 2022 •
edited
Loading

This is visible in this PR cosmos/cosmos-sdk#12537, i.e.,

exit status 1
[W2] Seed 2: FAILED
To reproduce run: go test ./simapp -run TestFullAppSimulation -Enabled=true -NumBlocks=50 -Genesis= -Verbose=true -Commit=true -Seed=2 -Period=10 -ExportParamsPath /tmp/sim-logs-863298177/sim_params-2.json -ExportStatePath /tmp/sim-logs-863298177/sim_state-2.json -v -timeout 24h

Running the above command on the https://github.com/cosmos/cosmos-sdk/tree/interchain-security-rebase branch results in errors, i.e.,

Simulating... block 10/50, operation 0/0.I[2022-07-12|15:49:03.573] asserting crisis invariants                  module=x/crisis inv=0/11 name=bank/nonnegative-outstanding
I[2022-07-12|15:49:03.575] asserting crisis invariants                  module=x/crisis inv=1/11 name=bank/total-supply
I[2022-07-12|15:49:03.576] asserting crisis invariants                  module=x/crisis inv=2/11 name=distribution/nonnegative-outstanding
I[2022-07-12|15:49:03.577] asserting crisis invariants                  module=x/crisis inv=3/11 name=distribution/can-withdraw
I[2022-07-12|15:49:03.619] asserting crisis invariants                  module=x/crisis inv=4/11 name=distribution/reference-count
I[2022-07-12|15:49:03.620] asserting crisis invariants                  module=x/crisis inv=5/11 name=distribution/module-account
I[2022-07-12|15:49:03.621] asserting crisis invariants                  module=x/crisis inv=6/11 name=staking/module-accounts
simulation halted due to panic on block 10
Logs to writing to /Users/marius/.simapp/simulations/2022-07-12_15:49:03.log
--- FAIL: TestAppImportExport (1.88s)
panic: invariant broken: staking: bonded and not bonded module account coins invariant
        Pool's bonded tokens: 21724093905797stake
        sum of bonded tokens: 21724093905797
not bonded token invariance:
        Pool's not bonded tokens: 4625651167973stake
        sum of not bonded tokens: 4531562441725
module accounts total (bonded + not bonded):
        Module Accounts' tokens: 26349745073770stake
        sum tokens:              26255656347522


        CRITICAL please submit the following transaction:
                 tx crisis invariant-broken staking module-accounts [recovered]
        panic: invariant broken: staking: bonded and not bonded module account coins invariant
        Pool's bonded tokens: 21724093905797stake
        sum of bonded tokens: 21724093905797
not bonded token invariance:
        Pool's not bonded tokens: 4625651167973stake
        sum of not bonded tokens: 4531562441725
module accounts total (bonded + not bonded):
        Module Accounts' tokens: 26349745073770stake
        sum tokens:              26255656347522


        CRITICAL please submit the following transaction:
                 tx crisis invariant-broken staking module-accounts [recovered]
        panic: invariant broken: staking: bonded and not bonded module account coins invariant
        Pool's bonded tokens: 21724093905797stake
        sum of bonded tokens: 21724093905797
not bonded token invariance:
        Pool's not bonded tokens: 4625651167973stake
        sum of not bonded tokens: 4531562441725
module accounts total (bonded + not bonded):
        Module Accounts' tokens: 26349745073770stake
        sum tokens:              26255656347522


        CRITICAL please submit the following transaction:
                 tx crisis invariant-broken staking module-accounts

goroutine 24 [running]:
testing.tRunner.func1.2({0x101586440, 0x14001785290})
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1389 +0x1c8
testing.tRunner.func1()
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1392 +0x384
panic({0x101586440, 0x14001785290})
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/runtime/panic.go:838 +0x204
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed.func2()
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:147 +0xc8
panic({0x101586440, 0x14001785290})
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/runtime/panic.go:844 +0x258
github.com/cosmos/cosmos-sdk/x/crisis/keeper.Keeper.AssertInvariants({{0x140001d2c80, 0xb, 0x10}, {{0x1296df1d8, 0x14000ed22e0}, 0x1400011c050, {0x10174f670, 0x14000ed3bb0}, {0x10174f6c0, 0x14000ed3c10}, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/crisis/keeper/keeper.go:83 +0x528
github.com/cosmos/cosmos-sdk/x/crisis.EndBlocker({{0x10175d648, 0x140000400b8}, {0x101765630, 0x14000518780}, {{0x0, 0x0}, {0x1010b6a47, 0xe}, 0xa, {0x0, ...}, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/crisis/abci.go:20 +0x140
github.com/cosmos/cosmos-sdk/x/crisis.AppModule.EndBlock(...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/crisis/module.go:168
github.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x10175d648, 0x140000400b8}, {0x101765630, 0x14000518780}, {{0x0, 0x0}, {0x1010b6a47, 0xe}, 0xa, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/types/module/module.go:495 +0x34c
github.com/cosmos/cosmos-sdk/simapp.(*SimApp).EndBlocker(_, {{0x10175d648, 0x140000400b8}, {0x101765630, 0x14000518780}, {{0x0, 0x0}, {0x1010b6a47, 0xe}, 0xa, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/simapp/app.go:439 +0x64
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0x14000ee8ea0, {0x140000400b8?})
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/baseapp/abci.go:211 +0x180
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed({0x101766ab0?, 0x14000ee84e0}, {0x10174d438?, 0x14000010018}, 0x1?, 0x0?, 0x101744ad8, {0x1400002a780, 0x17, 0x18}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:186 +0x109c
github.com/cosmos/cosmos-sdk/simapp.TestAppImportExport(0x14000ee84e0)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/simapp/sim_test.go:112 +0x404
testing.tRunner(0x14000ee84e0, 0x101744a30)
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1439 +0x110
created by testing.(*T).Run
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1486 +0x300
FAIL    github.com/cosmos/cosmos-sdk/simapp     2.208s
FAIL
@mpoke mpoke added type: bug Issues that need priority attention -- something isn't working scope: cosmos-sdk Integration with Cosmos SDK labels Jul 12, 2022
@mpoke mpoke moved this to Todo in Replicated Security Jul 12, 2022
@mpoke mpoke mentioned this issue Jul 13, 2022
Closed
@mpoke
Copy link
Contributor Author

mpoke commented Jul 13, 2022

The issue was introduced somewhere between commits cosmos/cosmos-sdk@57f5633 and cosmos/cosmos-sdk@b293d64 (see https://github.com/cosmos/cosmos-sdk/commits/interchain-security-rebase). It's hard to identify the exact commit since the ones in between do not build.

@mpoke
Copy link
Contributor Author

mpoke commented Jul 13, 2022

The test fails because the ModuleAccountInvariants is broken, i.e., https://github.com/cosmos/cosmos-sdk/blob/c783aea68fbd856c2b188b2d467a7fa5cb4df1e6/x/staking/keeper/invariants.go#L80. Also, the test that is failing is TestAppImportExport, which first exports the state and then imports it back.

@danwt
Copy link
Contributor

danwt commented Jul 13, 2022

Found a problem

If we fix it will bring us closer but it might not be the only problem.

@mpoke mpoke mentioned this issue Jul 13, 2022
Merged
@danwt danwt mentioned this issue Jul 14, 2022
Merged
5 tasks
@mpoke mpoke assigned mpoke and unassigned sainoe Jul 28, 2022
@mpoke mpoke moved this from Todo to Waiting for review in Replicated Security Jul 28, 2022
@mpoke mpoke moved this from Waiting for review to In Progress in Replicated Security Jul 28, 2022
@mpoke
Copy link
Contributor Author

mpoke commented Aug 1, 2022

Fixed by cosmos/cosmos-sdk#12783

@mpoke mpoke closed this as completed Aug 1, 2022
Repository owner moved this from In Progress to Done in Replicated Security Aug 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mpoke mpoke

@danwt danwt

Labels
scope: cosmos-sdk Integration with Cosmos SDK type: bug Issues that need priority attention -- something isn't working
Projects
No open projects
Replicated Security
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mpoke @sainoe @danwt