Upgrade to Gaia V6.0.0 startup error

[Black52]

4:00PM INF Starting Peer service impl="Peer{MConn{18.142.7.52:26656} c6f03336e99b15b104048a1af056063107389441 out}" module=p2p peer={"id":"c6f03336e99b15b104048a1af056063107389441","ip":"18.142.7.52","port":26656}
4:00PM INF Starting MConnection service impl=MConn{18.142.7.52:26656} module=p2p peer={"id":"c6f03336e99b15b104048a1af056063107389441","ip":"18.142.7.52","port":26656}
4:00PM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
4:00PM ERR WARNING: Attempt to allocate proposer rewards to unknown proposer cosmosvalcons14sk4vptumprktehmuvvf0yynarjy4gv08t64t4. This should happen only if the proposer unbonded completely within a single block, which generally should not happen except in exceptional circumstances (or fuzz testing). We recommend you investigate immediately. module=x/distribution
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x170aeda]

goroutine 74 [running]:
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokensToValidator(0x23b9ea0, 0xc0029c3860, 0x23f5f40, 0xc001173c30, 0x23f5f40, 0xc001173c30, 0xc0000116f8, 0x23b9ea0, 0xc0029c3890, 0x23b9f20, ...)
github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:107 +0x3a
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens(0x23b9ea0, 0xc0029c3860, 0x23f5f40, 0xc001173c30, 0x23f5f40, 0xc001173c30, 0xc0000116f8, 0x23b9ea0, 0xc0029c3890, 0x23b9f20, ...)
github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:95 +0x13b0
github.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker(0x23de320, 0xc0000400a8, 0x23f8ec0, 0xc008b03780, 0xb, 0x0, 0xc004db88d0, 0xb, 0x815370, 0xf8bb81c, ...)
github.com/cosmos/[email protected]/x/distribution/abci.go:32 +0x4a8
github.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)
github.com/cosmos/[email protected]/x/distribution/module.go:170
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(0xc000502ee0, 0x23de320, 0xc0000400a8, 0x23f8ec0, 0xc008b03780, 0xb, 0x0, 0xc004db88d0, 0xb, 0x815370, ...)
github.com/cosmos/[email protected]/types/module/module.go:450 +0x1f8
github.com/cosmos/gaia/v6/app.(*GaiaApp).BeginBlocker(...)
github.com/cosmos/gaia/v6/app/app.go:628
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc0000c41a0, 0xc008b04420, 0x20, 0x20, 0xb, 0x0, 0xc004db88d0, 0xb, 0x815370, 0xf8bb81c, ...)
github.com/cosmos/[email protected]/baseapp/abci.go:193 +0x7f8
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc0064da420, 0xc008b04420, 0x20, 0x20, 0xb, 0x0, 0xc004db88d0, 0xb, 0x815370, 0xf8bb81c, ...)
github.com/tendermint/[email protected]/abci/client/local_client.go:280 +0xfa
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc0005dcde0, 0xc008b04420, 0x20, 0x20, 0xb, 0x0, 0xc004db88d0, 0xb, 0x815370, 0xf8bb81c, ...)
github.com/tendermint/[email protected]/proxy/app_conn.go:81 +0x75
github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x23df220, 0xc006a06120, 0x23ec6c0, 0xc0005dcde0, 0xc000f34d20, 0x23f5ac0, 0xc0056c51a0, 0x4f5b97, 0xc00208ae00, 0x20, ...)
github.com/tendermint/[email protected]/state/execution.go:307 +0x51b
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc0072a9b90, 0xb, 0x0, 0xc00e3b97b0, 0x7, 0xc00e3b97c0, 0xb, 0x4f5b97, 0x81536f, 0xc00208ae00, ...)
github.com/tendermint/[email protected]/state/execution.go:140 +0x165
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000e7e8c0, 0x0)
github.com/tendermint/[email protected]/blockchain/v0/reactor.go:398 +0xa28
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
github.com/tendermint/[email protected]/blockchain/v0/reactor.go:110 +0x8c

[okwme]

Hello,
Can you give us some more info about what you're trying to do?
Gaia v6.0.0 is released but should not be run on the mainnet until the upgrade proposal passes and block height 8695000 is reached.
The binary should work on the vega-testnet which is currently live.
Please let us know if you're still having an issue considering the information above and if not please close the issue or provide more information.

[okwme]

(linking to similar question: #1082)

[gituser]

Hello, Can you give us some more info about what you're trying to do? Gaia v6.0.0 is released but should not be run on the mainnet until the upgrade proposal passes and block height 8695000 is reached. The binary should work on the vega-testnet which is currently live. Please let us know if you're still having an issue considering the information above and if not please close the issue or provide more information.

v6.0.0 isn't working on vega-testnet, after syncing to the block #7453750 with v5.0.5 and then upgrading to v6.0.0 gaia gives this error:

Error: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected 34B13A7BE8CE3C65A3D5A151B2C3148FEBCFADE45BE06953654E15ECE8AA7448, got 5F350D3D64C2467643B29A31F3F1E4F7951165F0172EE2788EB0BE925516032A

It seems there is no way to rollback this specific block you have to start from scratch again with v5.0.5, then with v6.0.0-rc3 instead of v6.0.0 .. :/

[yaruwangway]

Hi @gituser, can you give us more info ?

how do you upgrade ? You manually switch to gaia v6.0.0 or you use cosmovisor ? also, could you please try v6.0.0-rc3 directly to see if this version works?

[yaruwangway]

Presently, we only have rollback in tendermint, but the rollback feature is in discussion in sdk as well, Please check here:

cosmos/cosmos-sdk#10281

[gituser]

Hi @gituser, can you give us more info ?

how do you upgrade ? You manually switch to gaia v6.0.0 or you use cosmovisor ? also, could you please try v6.0.0-rc3 directly to see if this version works?

I'm manually switching over to v6.0.0 version.

I had to remove all synced data and start from scratch again, now trying with v6.0.0-rc3 so far it's still syncing.

Presently, we only have rollback in tendermint, but the rollback feature is in discussion in sdk as well, Please check here:

Binance Coin (which is based on Cosmos) got that for a long time I think it's called state_restore.

[manefesto]

Hi. I try start gaiad v6.0.0 with existing data from v5.0.8. Not working
Now I trying start v6.0.0-rc3.
And got error
ERR WARNING: Attempt to allocate proposer rewards to unknown proposer...

[gituser]

@manefesto you shouldn't use v6.0.0 on the mainnet yet

[manefesto]

@manefesto you shouldn't use v6.0.0 on the mainnet yet

Tell me please. Will be v5.0.8 still worked at this year ?

[gituser]

@manefesto current mainnet latest release is v5.0.8 it works just fine.

[manefesto]

@manefesto current mainnet latest release is v5.0.8 it works just fine.

At release says:

This is the final release for the Vega upgrade which is proposed to take place at height 8,695,000 which should occur approximately December 13th, 2021 at 14:00:00 UTC with a margin of error of ~33 hours to accommodate for any clock drift (as we saw in the Delta upgrade). The chain id will remain cosmoshub-4.

Now our nodes has height ~ 8628826

[melvjoshua]

Hi @okwme, what is the minimum requirement server for gaiad v6.0.0?
Because I got error fatal error: runtime: out of memory

My server specifications are Ubuntu 16.04, 4 CPU, 16 GB RAM

[okwme]

@melvjoshua we've found many nodes reporting problems with just 16 but that was our understanding of a good config. We've updated that to recommend 32gb going forward. docs will be updated soon.