Add accum tracking invariants

[cwgoes]

Ensure that all rewards could be withdrawn in a hypothetical scenario - this will double-check all our accum tracking logic & associated hooks are exactly correct.

[cwgoes]

Ref #2588 (comment)

[cwgoes]

OK, running the multisim produced two seemingly distinct failures, each replicated several times.

All can be replicated with scripts/multisim.sh 500 on the 0.25 release branch. Note that this will take a while to run.

Seeds 2, 4, 123, 124, 582, 2989, 4728, 891823782, and 89182391 passed the 500-block simulation.

With seeds 1, 7, 20, 32, 1893, 37827, 981928, 87821, 989182:

(all failed on different blocks but with the same failure)

E[10-25|11:28:41.386] ddi created: {01A1C140E0340A8E38B19A05520B6E51D0B57EFB A502D91204FE84A09A566DD21784DFC46E449260 113} module=x/distribution 
Simulating... block 113/500, operation 550/722. E[10-25|11:28:41.411] ddi created: {60C4B1521E05F4EAB9EFBEB005BA98AA9D456218 DCE8556859540CD0AD1C42B80034274D18D8946E 113} module=x/distribution 
Panic with err
 individual accum should never be greater than the total
goroutine 39 [running]:
runtime/debug.Stack(0xc42de37758, 0x2, 0x2)
	/snap/go/2130/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed.func3(0xc421164000, 0xc4200e56c0, 0xc4208012e0)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:119 +0xb6
panic(0xd22ba0, 0x10795d0)
	/snap/go/2130/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/x/distribution/types.ValidatorDistInfo.TakeFeePoolRewards(0xc423e0cea0, 0x14, 0x14, 0x71, 0xc42838f260, 0x1, 0x1, 0xc42838f340, 0x1, 0x1, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/types/validator_info.go:58 +0x530
github.com/cosmos/cosmos-sdk/x/distribution/types.DelegationDistInfo.WithdrawRewards(0xc423e0c9a0, 0x14, 0x14, 0xc423e0c9e0, 0x14, 0x14, 0x67, 0x71, 0xc42838eec0, 0xc42838efc0, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/types/delegator_info.go:41 +0x28b
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.getDelegatorRewardsAll.func1(0x4, 0x10861a0, 0xc42136a1c0, 0xc42136a1c0)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:142 +0x6c4
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.IterateDelegations(0x107f080, 0xc420801030, 0x107f0c0, 0xc420801040, 0xc420806540, 0x108bc00, 0xc42080ed20, 0x0, 0x0, 0xc420806540, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/sdk_types.go:118 +0x30b
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.getDelegatorRewardsAll(0x107f080, 0xc420801060, 0xc420806540, 0xc420806540, 0x107f080, 0xc4208010b0, 0x107f0c0, 0xc4208010c0, 0xc4208b93a0, 0x5, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:150 +0x240
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewardsAll(0x107f080, 0xc420801060, 0xc420806540, 0xc420806540, 0x107f080, 0xc4208010b0, 0x107f0c0, 0xc4208010c0, 0xc4208b93a0, 0x5, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:113 +0x120
github.com/cosmos/cosmos-sdk/x/distribution.handleMsgWithdrawDelegatorRewardsAll(0x10867e0, 0xc42144e870, 0xc42136b1c0, 0xc, 0xc420ab9c60, 0x14, 0x20, 0x107f080, 0xc420801060, 0xc420806540, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/handler.go:48 +0xf5
github.com/cosmos/cosmos-sdk/x/distribution.NewHandler.func1(0x10867e0, 0xc42144e870, 0xc42136b1c0, 0xc, 0x1088fe0, 0xc422504620, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/handler.go:17 +0x23b
github.com/cosmos/cosmos-sdk/x/distribution/simulation.SimulateMsgWithdrawDelegatorRewardsAll.func1(0xc420abef90, 0xc420902780, 0x10867e0, 0xc424766c60, 0xc42136b1c0, 0xb, 0xc420ad5000, 0xfa, 0xfa, 0xc420801310, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/distribution/simulation/msgs.go:57 +0x237
github.com/cosmos/cosmos-sdk/x/mock/simulation.createBlockSimulator.func2(0x2d2, 0xc420abef90, 0xc420902780, 0x10867e0, 0xc424766c60, 0xc42136b1c0, 0xb, 0xc420ad5000, 0xfa, 0xfa, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:223 +0x1a4
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0x1090e40, 0xc4208ae5a0, 0xc420902780, 0x1004008, 0xf17fe, 0xc4200e20e0, 0xe, 0xe, 0xc420019ef8, 0x0, ...)
	/root/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:158 +0x117c
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc4208ae5a0)
	/root/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:189 +0x2ac
testing.tRunner(0xc4208ae5a0, 0x1003fd8)
	/snap/go/2130/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/snap/go/2130/src/testing/testing.go:824 +0x2e0

Too many logs to display, instead writing to simulation_log_2018-10-25 11:28:41.txt

With seeds 9, 3012:
(all failed on separate blocks but with the same error)

I[10-25|11:27:20.044] validator cosmosvalcons1axhf84grzqpe4tra0ttt77wa86d3l0q4c4cu8s jailed module=x/stake 
E[10-25|11:27:20.044] allocation height: 125                       module=x/distribution 
Invariants broken after BeginBlock
expected loose tokens to equal total steak held by accounts - pool.LooseTokens: 8993955282023041, sum of account tokens: 8993958282023041
Too many logs to display, instead writing to simulation_log_2018-10-25 11:27:20.txt
--- FAIL: TestFullGaiaSimulation (375.01s)
	util.go:112: 
FAIL
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	375.158s

[rigelrozanski]

For reference here is the command which was generating the first bug outlined by @cwgoes :
< uses a seed of 1 >

go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=1 -v -timeout 24h

This has been fixed in #2597
with commit ce0528f

However the second issue which is the staking loose token invariance is now picked up by this command once again

[rigelrozanski]

RE: the coin invariance - looks like it breaks after begin-block:

~/go/src/github.com/cosmos/cosmos-sdk$ go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=7 -v -timeout 24h
=== RUN   TestFullGaiaSimulation
Starting SimulateFromSeed with randomness created with seed 7
Starting the simulation from time Sun Nov 27 17:22:59 UTC 6089, unixtime 130011758579
Simulating... block 76/400, operation 550/566. Invariants broken after BeginBlock
loose token invariance:
	pool.LooseTokens: 8132412500000000
	sum of account tokens: 8132413000000000
Too many logs to display, instead writing to simulation_log_2018-10-25 21:42:16.txt
--- FAIL: TestFullGaiaSimulation (437.04s)
    util.go:115:
FAIL
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	437.106s

[rigelrozanski]

Update going with Jae's fix from for accum invariance: b97a076

LooseToken invariance can still be reproduced with a seed of 9 aka running go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=9 -v -timeout 24h

---

Update is resolved with 99efde2

(go jae!)