Randomized fuzzer: expected to find slashing period, but none was found

[ValarDragon]

Steps to Reproduce

Set -SimulationNumBlocks=500 in one of the makefile simulator commands.
Comment out:

govsim.SimulateMsgDeposit(app.govKeeper, app.stakeKeeper),
govsim.SimulateMsgVote(app.govKeeper, app.stakeKeeper),

in $GOPATH/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go.

Bug

$ make test_sim_gaia_fast 
Running quick Gaia simulation. This may take several minutes...
=== RUN   TestFullGaiaSimulation
Starting SimulateFromSeed with randomness created with seed 42
Starting the simulation from time Sun Jan 31 13:59:47 UTC 26140, unixtime 762734152787
Simulating... block 260/500, operation 650/664.--- FAIL: TestFullGaiaSimulation (860.65s)
panic: expected to find slashing period, but none was found [recovered]
	panic: expected to find slashing period, but none was found

goroutine 14 [running]:
testing.tRunner.func1(0xc420980f00)
	/usr/local/go/src/testing/testing.go:742 +0x29d
panic(0xcf5900, 0x1042510)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/x/slashing.Keeper.getValidatorSlashingPeriodForHeight(0x1048020, 0xc4209517f0, 0xc420982380, 0x1055b80, 0xc420492e60, 0xc420982380, 0x1048020, 0xc420951820, 0xa, 0x104f500, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/slashing/slashing_period.go:46 +0x380
github.com/cosmos/cosmos-sdk/x/slashing.Keeper.onValidatorBeginUnbonding(0x1048020, 0xc4209517f0, 0xc420982380, 0x1055b80, 0xc420492e60, 0xc420982380, 0x1048020, 0xc420951820, 0xa, 0x104f500, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/slashing/hooks.go:20 +0x108
github.com/cosmos/cosmos-sdk/x/slashing.ValidatorHooks.OnValidatorBeginUnbonding(0x1048020, 0xc4209517f0, 0xc420982380, 0x1055b80, 0xc420492e60, 0xc420982380, 0x1048020, 0xc420951820, 0xa, 0x104f500, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/slashing/hooks.go:45 +0xbb
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.beginUnbondingValidator(0x1048020, 0xc4209517e0, 0xc420982380, 0x1048020, 0xc4209517c0, 0xfd2578, 0xc420982380, 0x104a360, 0xc420492eb0, 0x4, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:645 +0x825
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateBondedValidators(0x1048020, 0xc4209517e0, 0xc420982380, 0x1048020, 0xc4209517c0, 0xfd2578, 0xc420982380, 0x104a360, 0xc420492eb0, 0x4, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:516 +0x89e
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.UpdateValidator(0x1048020, 0xc4209517e0, 0xc420982380, 0x1048020, 0xc4209517c0, 0xfd2578, 0xc420982380, 0x104a360, 0xc420492eb0, 0x4, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/validator.go:286 +0x70f
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Slash(0x1048020, 0xc4209517e0, 0xc420982380, 0x1048020, 0xc4209517c0, 0xfd2578, 0xc420982380, 0x104a360, 0xc420492eb0, 0x4, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/slash.go:111 +0xe1a
github.com/cosmos/cosmos-sdk/x/gov.EndBlocker(0x104f500, 0xc421078510, 0xc4200476c0, 0x212, 0xc420982380, 0x1048020, 0xc420951820, 0x1048020, 0xc4209517c0, 0xfd2578, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/gov/handler.go:151 +0x1305
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc420844600, 0x104f500, 0xc421078510, 0xc4200476c0, 0x212, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:153 +0xcb
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x104f500, 0xc421078510, 0xc4200476c0, 0x212, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:115 +0xce
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc4208b7ba0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:656 +0x10a
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0x10596e0, 0xc420980f00, 0xc4208b7ba0, 0xfd0ae8, 0x2a, 0xc420493360, 0xa, 0xa, 0xc420997f08, 0x0, ...)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/random_simulate_blocks.go:115 +0xf58
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc420980f00)
	/home/valar/Code/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:171 +0x2ac
testing.tRunner(0xc420980f00, 0xfd0ac0)
	/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x2e0
FAIL	github.com/cosmos/cosmos-sdk/cmd/gaia/app	860.690s
Makefile:159: recipe for target 'test_sim_gaia_fast' failed
make: *** [test_sim_gaia_fast] Error 1

This bug is blocking a change I was making to governance simulation. (Making it use future operations instead of normal operations) I'll probably PR it anyway, since this being on develop indicates its not introduced by my PR, just that my PR changes the randomness so it occurs sooner.

As an additional comment, we should add an additional defer so we can recover in these situations and still print out the log.

[ValarDragon]

/cc @rigelrozanski @alexanderbez @jackzampolin @cwgoes

[cwgoes]

Blocks 0.25.

[ValarDragon]

If you set the number of blocks to 260 this fails on latest develop, with no operations needing to be commented out. Fails on block 252. (Took ~100 seconds to get there on my machine in benchmark mode. Testing mode is significantly slower, perhaps we shouldn't run the invariant every single time. Disabling invariants for the first 150 blocks made this take 10 minutes to get there in testing mode, but currently we don't print the log message in event of a panic.)

[ValarDragon]

To get logs for this one, use #2285, and set the make test_sim_gaia_fast to 300 blocks. It will fail on block 260. (Above where I wrote 252 was always wrong)

It now takes ~6-7 minutes.

[alexanderbez]

Closing this in favor of #2312.

[alexanderbez]

Actually, re-opening this to keep track of the issue.

[cwgoes]

#2430 will close, the height should have been stored as big endian.

[cwgoes]

Closed by #2430.