Investigate simulation failing after migration to CometBFT

[julienrbrt]

The migration to CometBFT has happened in the following PRs, in this following order:

• refactor: migrate store and api to CometBFT #14899 (only import change in api and store)
• refactor!: migrate Cosmos SDK to CometBFT #14897 (only import change the rest of the SDK) (sims start to fail from here)
• refactor: rename to CometBFT #14914 (only import renaming)

The SDK upgraded from Tendermint v0.37.0-rc2 to CometBFT v0.37.x branch, which contains the following changes: cometbft/cometbft@059798a...v0.37.x

1. After refactor!: migrate Cosmos SDK to CometBFT #14897, the sims started to fail on main: https://github.com/cosmos/cosmos-sdk/actions/runs/4116831858.

Giving the following error

Logs to writing to /home/julien/.simapp/simulations/2023-02-06_22:36:49.log
--- FAIL: TestFullAppSimulation (0.73s)
panic: Stored fee pool should not have been nil [recovered]
        panic: Stored fee pool should not have been nil [recovered]
        panic: Stored fee pool should not have been nil

goroutine 459 [running]:
testing.tRunner.func1.2({0x21b4dc0, 0x2ffad20})
        /usr/local/go/src/testing/testing.go:1396 +0x24e
testing.tRunner.func1()
        /usr/local/go/src/testing/testing.go:1399 +0x39f
panic({0x21b4dc0, 0x2ffad20})
        /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed.func2()
        /home/julien/projects/cosmos/cosmos-sdk/x/simulation/simulate.go:146 +0xf1
panic({0x21b4dc0, 0x2ffad20})
        /usr/local/go/src/runtime/panic.go:890 +0x262
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.GetFeePool({{0x300bd38, 0xc000455af0}, {0x7f82dc3d7068, 0xc000603170}, {0x7f82dc3d7c30, 0xc0004cd440}, {0x7f82dc3d7c68, 0xc000c81140}, {0x7f82dc3d7cb0, 0xc0004cd6e0}, ...}, ...)
        /home/julien/projects/cosmos/cosmos-sdk/x/distribution/keeper/store.go:53 +0x115
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens({{0x300bd38, 0xc000455af0}, {0x7f82dc3d7068, 0xc000603170}, {0x7f82dc3d7c30, 0xc0004cd440}, {0x7f82dc3d7c68, 0xc000c81140}, {0x7f82dc3d7cb0, 0xc0004cd6e0}, ...}, ...)
        /home/julien/projects/cosmos/cosmos-sdk/x/distribution/keeper/allocation.go:30 +0x238
github.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker({{0x30271b8, 0xc000058148}, {0x303a180, 0xc0023afec0}, {{0x0, 0x0}, {0x25dd5e2, 0xe}, 0x2, {0x0, ...}, ...}, ...}, ...)
        /home/julien/projects/cosmos/cosmos-sdk/x/distribution/abci.go:28 +0x17b
github.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)
        /home/julien/projects/cosmos/cosmos-sdk/x/distribution/module.go:178
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x30271b8, 0xc000058148}, {0x303a180, 0xc0023afec0}, {{0x0, 0x0}, {0x25dd5e2, 0xe}, 0x2, ...}, ...}, ...)
        /home/julien/projects/cosmos/cosmos-sdk/types/module/module.go:644 +0x23d
github.com/cosmos/cosmos-sdk/runtime.(*App).BeginBlocker(_, {{0x30271b8, 0xc000058148}, {0x303a180, 0xc0023afec0}, {{0x0, 0x0}, {0x25dd5e2, 0xe}, 0x2, ...}, ...}, ...)
        /home/julien/projects/cosmos/cosmos-sdk/runtime/app.go:120 +0x7d
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0x0, 0x0, 0x0}, {{0x0, 0x0}, {0x25dd5e2, 0xe}, 0x2, {0x0, ...}, ...}, ...})
        /home/julien/projects/cosmos/cosmos-sdk/baseapp/abci.go:202 +0x91b
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed({0x304fc80?, 0xc000b82820?}, {0x30078c0?, _}, _, _, _, {0xc00122d880, 0x2e, 0x38}, ...)
        /home/julien/projects/cosmos/cosmos-sdk/x/simulation/simulate.go:165 +0x14a5
cosmossdk.io/simapp.TestFullAppSimulation(0xc000b82820)
        /home/julien/projects/cosmos/cosmos-sdk/simapp/sim_test.go:90 +0x98f
testing.tRunner(0xc000b82820, 0x2c7fa68)
        /usr/local/go/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:1493 +0x35f
FAIL    cosmossdk.io/simapp     0.806s
FAIL

When running go test . -run TestFullAppSimulation -Enabled=true -NumBlocks=50 -Genesis= -Verbose=true -Commit=true -Seed=4 -Period=10 -ExportParamsPath /tmp/sim-logs-2102746239/sim_params-4.json -ExportStatePath /tmp/sim-logs-2102746239/sim_state-4.json -v -timeout 24h.

2. Additionally, any transactions and queries are failing with Error: rpc error: code = InvalidArgument desc = failed to load state at height 1; version does not exist (latest height: 1): invalid request at any height.
   Updating the store (https://github.com/cosmos/cosmos-sdk/blob/main/store/rootmulti/store.go#L518-L521) to display the error shows the following:

Display error in store

var err error
cacheStore, err = store.(*iavl.Store).GetImmutable(version)
if err != nil {
		rs.logger.Error("error with store", "error", err.Error(), "key", key.String())
		continue
}

Errors shows

E[2023-02-07|17:02:26.666] error with store                             module=server error="version does not exist" key="KVStoreKey{0xc000fa0ad0, params}"
E[2023-02-07|17:02:26.666] error with store                             module=server error="version does not exist" key="KVStoreKey{0xc000fcbf60, evidence}"
E[2023-02-07|17:02:26.666] error with store                             module=server error="version does not exist" key="KVStoreKey{0xc000fac340, feegrant}"
E[2023-02-07|17:02:26.666] error with store                             module=server error="version does not exist" key="KVStoreKey{0xc000fcb650, authz}"
E[2023-02-07|17:02:26.666] error with store                             module=server error="version does not exist" key="KVStoreKey{0xc000fff330, nft}"

With that changes, only these modules do not work, and the rest of SimApp works just fine.

---

As the migration has been done in steps, and the import changes needed to happen at the same time (like for gogoproto. I don't think reverting adds any value.