uneven store heights prevent pruning and state sync #13933

faddat · 2022-11-20T07:24:25Z

Hi, this issue is to track a set of issues, and may become an epic.

Pruning and/ or state sync

dig
konstellation
- code too old for valid comparisons
akash
stride

guess

This is because of issues in the CMD folder, and possible validator misconfiguration.

I think that this can be fixed by assisting crypto.org with migrating, then compacting their archival node state, so that they can use fast node and pebble, which makes archives much more performant.

Some of these issues were introduced here:

https://github.com/cosmos/iavl/commits/master#:~:text=feat%3A%20Add%20skipFastStorageUpgrade%20to%20MutableTree%20to%20control%20fast%20s%E2%80%A6

faddat · 2022-11-21T15:01:10Z

faddat · 2022-11-21T15:08:09Z

chillyvee · 2022-11-21T22:11:33Z

Comdex v5.0.0 also has broken pruning
https://github.com/comdex-official/comdex/releases/tag/v5.0.0

chillyvee · 2022-11-21T22:53:40Z

Konstellation snashot/pruning broke because feegrant and authz (in rootmulti) are the wrong height (tree.ndb is nil)
Team notified, no response

chillyvee · 2022-11-21T23:18:00Z

Comdex snashot/pruning broke because lendV1 store (in rootmulti) is the wrong height (tree.ndb is nil)
Team notified, no repsonse

faddat · 2022-11-22T15:44:05Z

.... stride has this issue, as well, in v3.0.1

chillyvee · 2022-11-22T15:48:26Z

Stride just hit the same problem with their authz store being uninitialized and caused issues for validators.
vishal is aware, says it makes sense, and is trying to fix it

faddat · 2022-11-22T17:12:45Z

probably because of the uneven iavl state height issue that @chillyvee is mentioning here, state sync broke on stride:

git clone https://github.com/stride-labs/stride
git checkout v3.0.0
bash scripts/statesync.bash

I don't remember having issues with uneven state heights in the past when adding modules.

Reecepbcups · 2022-11-22T17:25:52Z

@faddat Juno was having block header apphash issue as well back on Nov 7th.

Golden , Windpower, TedCrypto, Coldchain - fixed when bumped GO 1.19.2 from 1.18.3

faddat · 2022-11-22T17:38:58Z

@Reecepbcups really good.

Let's break that out separately, though. I'll maybe do issue/pr for it, doing some 'splain... I think I know why that occurs....

The mixed go runtime issue is now here:

#13976

@chillyvee 's PR related to this issue is here:

#13935

chillyvee · 2022-11-22T19:24:01Z

another validator also reports statesync for konstellation still doesnt work (maybe a store with information with the wrong height?)

faddat · 2022-11-22T20:09:22Z

@chillyvee -- my feeling is that you've correctly identified the problem.

Not certain just yet.

Here's my stride working branch:

https://github.com/Stride-Labs/stride/tree/no-state-breaks

That said, I figure that the fix is somewhere between here, iavl, and ibc-go, or such.

faddat · 2022-11-23T13:14:35Z

ChihuahuaChain/chihuahua#78

Highlander-maker · 2022-11-23T15:25:51Z

This is a great thing to be doing. Happy to run some tests and provide feedback when we can. Looping in @qf3l3k to the chat.

chillyvee · 2022-11-24T19:27:24Z

@faddat

Likely root cause

It's likely along the lines of adding a KV Store Key without adding the keeper in the migration

https://github.com/Stride-Labs/stride/blob/v3.0.0/app/app.go#L328

	keys := sdk.NewKVStoreKeys(
		authzkeeper.StoreKey,

That will probably continue on to confuse all the other stores/exports/snapshots/etc

To operate correctly a chain MUST do all the following at the same time:

KVStoreKey
define keeper
app.SetStoreLoader(upgradetypes.UpgradeStoreLoader(upgradeInfo.Height, storeUpgrades))

To resolve current breakages:
Upgrade and formally add keeper + add stores in upgrade handler

To completely ruin your node:
Removing keys from keys := sdk.NewKVStoreKeys will cause an instant AppHash mismatch. Don't do it.

Current breakages

For chains like Konstellation/Stride where authz/feegrant/etc keepers are not initialized, statesync restore apphashes will not match. They should probably upgrade and officially issue a store upgrade

https://github.com/Stride-Labs/stride/pull/384/files#diff-aeec7ddb4cf8b7e9e6e57625770d369cce5f377cc350b42c3aca0427bba8dfbcR52-R55

Surprisingly not broken

Dig statesyncs with patches probably because the uninitialized icahost store at least has a keeper defined.
notional-labs/dig#426

Decline to fix

Probably possible to emulate a broken KV store during snapshot, but that seems like a waste of time if a proper upgrade can be issued instead. (but we tried to fix it anyway, see next message)

A better fix

Check that each key actually has a keeper before entering consensus. This will allow nodes to start, upgrade/migrate, then check kv-store matches before starting up. This is an exercise left to the reader :)

The failure mode in this case is that a chain node could halt, giving time for teams to release a new version with a migration to formally add the missing store.

Start node. Add NewKVStoreKeys() without a keeper
Execute new sanity check. Gracefully exit on mismatch of NewKVStoreKeys vs keepers
Chain team can issue patch
Start node again, execute additional migration to add store
Enter consensus

faddat · 2022-11-25T12:53:47Z

I like your proposal CV.

chillyvee · 2022-11-25T19:48:34Z

This is a potential fix for snapshotting until chains execute the final keeper + store migrations

Root cause

Snapshots are missing IAVL nodes when heights do not match.

Fix

Snapshots can now be created even with mismatched rootmulti/store heights

Snapshot Patch

After checkout of chain code

 go mod edit -replace github.com/cosmos/iavl=github.com/chillyvee/[email protected]
 go mod edit -replace github.com/cosmos/cosmos-sdk=github.com/chillyvee/[email protected]
 go mod tidy

Tested it against Konstellation v0.6.0 and it works.

Isolate from peers without snapshot patched

You will need to isolate your peers to avoid getting broken snapshots from others.

This means setting the following on the restoring node (not the source node)

[p2p]
persistent_peers = "only your patched snapshot source node"

pex = false

Tested

Konstellation v0.6.0 snapshot and restore now operates properly

Need assistance

Since stride 3.0.0/comdex 5.0.0 are also on cosmos-sdk v0.45.9, I think the patch will work there as well.

For now the code seems functional, but isn't clean enough to upstream.

Would appreciate some help testing before I submit the PRs.

chillyvee · 2022-11-26T17:22:51Z

For Konstellation

 go mod edit -replace github.com/cosmos/iavl=github.com/chillyvee/[email protected]
 go mod edit -replace github.com/cosmos/cosmos-sdk=github.com/chillyvee/[email protected]
 go mod tidy

For Comdex

git checkout https://github.com/chillyvee/comdex
cd comdex
git checkout cv5.0.0.fixsnap
go mod edit -replace github.com/cosmos/iavl=github.com/chillyvee/[email protected]
go mod edit -replace github.com/cosmos/cosmos-sdk=github.com/chillyvee/[email protected]
go mod tidy

For Stride

go mod edit -replace github.com/cosmos/iavl=github.com/chillyvee/[email protected]
go mod edit -replace github.com/cosmos/cosmos-sdk=github.com/chillyvee/[email protected]
go mod tidy

tac0turtle · 2022-11-27T12:22:55Z

super interesting breakdown!! thank you for this. It seems there is a massive ux issue that leads to broken issues because they are not well documented as well.

I like your approach, I believe this is the right move, we can add some of the footman preventers in previous releases as well. Lmk how I can help you

chillyvee · 2022-11-27T21:15:57Z

super interesting breakdown!! thank you for this. It seems there is a massive ux issue that leads to broken issues because they are not well documented as well.

I like your approach, I believe this is the right move, we can add some of the footman preventers in previous releases as well. Lmk how I can help you

Need another day or two to clean up some code and send in another few helpful PRs. Once you see those, we might find a pattern we can enhance :) Thank you tac0turtle!

chillyvee · 2023-01-12T06:08:43Z

Lum network broke on v1.3.0 due to 'feeibc' store height mismatch.

dsmello · 2023-01-16T10:40:41Z

Hello everyone, I'm having a similar issue at the starname chain (cosmos-SDK v0.45.9), but in my case, I got an empty app hash.
Do you think it's related? I'll add more detail when that starts in my case. And I can also try to implement the "sanity check" to the app object if anyone could describe what's needed.

The error at state sync starts at starname v0.10.14, where was:

cosmos SDK v0.42.5 → v0.42.9

wasmvm → v0.16.0

tendermint v0.34.10 → v0.34.11 → v0.34.12

And now we're using the cosmos v0.45.9

alexanderbez · 2023-01-16T15:15:44Z

@chillyvee for whatever reason I cannot find the comment and/or PR, but I swear I commented somewhere that the root fix to many of these issues wrt to pruning and upgrading with new stores is that when a store is loaded for the first time, i.e. it does not currently exist, we set it's initial version to the current version and not 0. Does this ring a bell?

chillyvee · 2023-01-17T05:19:42Z

I got an empty app hash.

@dsmello - Haven't seen an empty app hash before. Can you share some logs? Was there a recent upgrade and if so what was the tag of the version before, and the version of the upgrade? Is this the repo? https://github.com/iov-one/starnamed

chillyvee · 2023-01-17T05:29:28Z

the root fix
@alexanderbez

Prevention:

Don't make the mistake in the first place and add all new stores via migration to make the new store height match

Future: Prevent adding stores outside of a migration/enforce height matching

Fixing broken appstate:

Not possible to adjust heights because iavl enforces increment by 1 and prevents fixing
if the broken store is empty, you can delete the broken store and make the new one the right way

If you are using a cosmos-sdk module you will need to split this into 2 upgrades. 1 for the store delete, and 1 for the store add
If you are using an external module, then you can delete the old store and migrate in the new store

If the broken store is not empty, then there must also be data migration

In the case of a cosmos-sdk module - not sure where to store the data between the first and second fix migration
In the case of a custom module - maybe it's possible to copy data between the old and new store?

dsmello · 2023-01-17T10:59:22Z

Hi @chillyvee,
Yes, that's our repo.

The current version of starnamed is v0.11.6 (wasmd -> ; cosmos-SDK -> v0.45.9 ; tendermint -> v0.34.21)
We also had a State Sync issue at version v0.10.14, but at that version, the issue was a different appHash instead of an empty appHash.

We migrate from v0.10.13 -> v0.11.6 (Cosmos-SDK v0.42.5 -> v0.45.9), adding these modules (authzKeeper, feegrant, escrowtypes, and the burner module), the only one with doesn't have a store was the burner module.

This is the error :

ERR appHash verification failed actual= expected="�=[u}A��|�N�9\rd7\x7fl�\x1c�i��,��\x00]Kkj" module=statesync

I tried to do some debugging about and:

The error was rise when the vendor/github.com/tendermint/tendermint/statesync/syncer.go(verifyApp:478) runs:
resp, err := s.connQuery.InfoSync(proxy.RequestInfo) and the resp.LastBlockAppHash was empty.

RESPONSE: %+vdata:"starnamed" version:"v0.11" last_block_height:12634200  module=statesync
SNAPSHOT: %+v&{12645200 1 1 [185 116 185 254 4 253 237 156 245 219 141 194 180 138 121 209 0 236 44 176 80 47 252 87 27 253 64 19 134 157 60 243] [10 32 185 116 185 254 4 253 237 156 245 219 141 194 180 138 121 209 0 236 44 176 80 47 252 87 27 253 64 19 134 157 60 243] [133 61 91 117 125 65 241 186 124 193 78 140 57 13 100 55 127 108 129 28 215 105 235 173 44 163 155 0 93 75 107 106]}

And what logs did you need?

[p2p]
persistent_peers = "[email protected]:26656"

[statesync]
enable = true
rpc_servers = "http://35.210.33.5:26657,http://35.210.33.5:26657"
trust_height = 12562301
trust_hash = "6E4FBCF45010667D230382A150461E0DD4467A7E2BEA8840607135D766657E2C"
trust_period = "2000h0m0s"

alexanderbez · 2023-01-17T16:12:41Z

@chillyvee Ok so we have #14410 and the author doesn't seem to be too responsive. @catShaark if you're OK with it, I will open another PR.

I would like to see a single PR address these points.

chillyvee · 2023-01-17T16:49:46Z

I would like to see a single PR address these points.

@alexanderbez - Correct the idea of #14410 (or any replacement) will prevent the root cause of the issue we commonly see.

chillyvee · 2023-01-17T16:53:24Z

We also had a State Sync issue at version v0.10.14, but at that version, the issue was a different appHash instead of an empty appHash.

Overall an interesting situation. No additional logs needed but we need to pick a direction.

Didn't see your ID on the committers list for starnamed, so I assume you are not a developer for that repo and a validator only? For a full fix, it would be good to bring in one of the core developers.

How about taking a look at a copy of the data from a working node. Is anybody sharing pruned zip files of the data directory?

dsmello · 2023-01-17T20:48:16Z

I never added myself to the contributor's list and need to fix the git. I'll do that this week.

The current version is v0.11.6

Yes, you can use the following:

*I still needed to gain experience in debugging the cosmos. What I'm supposed to look at in the data folder?

chillyvee · 2023-01-19T21:35:09Z

@dsmello - The issue starname is facing is NOT the one in this thread.

It looks like your stores are not in the array (rs.stores is EMPTY) for snapshotting at this point in the code:
https://github.com/cosmos/cosmos-sdk/blob/v0.45.9/store/rootmulti/store.go#L694

It seems somehow starname is able to define all the stores and operate without setting all the related fields.

That means that any restore with statesync results in no data and no apphash.

dsmello · 2023-01-20T17:24:25Z

Hi @chillyvee, Thanks for the help.
I'll try to debug deeper at this point to see if I find something, and if the bug was related to the cosmos, I'll open a PR for that issue 😄. Monday, I open an issue with that bug to leave some kind of history for that kind of problem to others. Tks guys.

tac0turtle · 2023-02-05T12:49:35Z

Part of the issue is solved, notional submitted a fix to prevent chains from starting when adding a store at a different height. How can we help teams recover from this state they are in? Should we write a tool? I think people need to upgrade their chains to recover right?

faddat · 2023-02-06T19:49:53Z

I think that there is a related issue:

  "message": "\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).createQueryContext\n\tgithub.com/cosmos/[email protected]/baseapp/abci.go:670\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).handleQueryGRPC\n\tgithub.com/cosmos/[email protected]/baseapp/abci.go:592\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Query\n\tgithub.com/cosmos/[email protected]/baseapp/abci.go:441\ngithub.com/tendermint/tendermint/abci/client.(*localClient).QuerySync\n\tgithub.com/tendermint/[email protected]/abci/client/local_client.go:256\ngithub.com/tendermint/tendermint/proxy.(*appConnQuery).QuerySync\n\tgithub.com/tendermint/[email protected]/proxy/app_conn.go:159\ngithub.com/tendermint/tendermint/rpc/core.ABCIQuery\n\tgithub.com/tendermint/[email protected]/rpc/core/abci.go:20\ngithub.com/tendermint/tendermint/rpc/client/local.(*Local).ABCIQueryWithOptions\n\tgithub.com/tendermint/[email protected]/rpc/client/local/local.go:87\ngithub.com/cosmos/cosmos-sdk/client.Context.queryABCI\n\tgithub.com/cosmos/[email protected]/client/query.go:94\ngithub.com/cosmos/cosmos-sdk/client.Context.QueryABCI\n\tgithub.com/cosmos/[email protected]/client/query.go:57\ngithub.com/cosmos/cosmos-sdk/client.Context.Invoke\n\tgithub.com/cosmos/[email protected]/client/grpc_query.go:81\ngithub.com/cosmos/cosmos-sdk/x/bank/types.(*queryClient).AllBalances\n\tgithub.com/cosmos/[email protected]/x/bank/types/query.pb.go:917\ngithub.com/cosmos/cosmos-sdk/x/bank/types.request_Query_AllBalances_0\n\tgithub.com/cosmos/[email protected]/x/bank/types/query.pb.gw.go:139\ngithub.com/cosmos/cosmos-sdk/x/bank/types.RegisterQueryHandlerClient.func2\n\tgithub.com/cosmos/[email protected]/x/bank/types/query.pb.gw.go:684\ngithub.com/grpc-ecosystem/grpc-gateway/runtime.(*ServeMux).ServeHTTP\n\tgithub.com/grpc-ecosystem/[email protected]/runtime/mux.go:240\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/[email protected]/mux.go:210\ngithub.com/tendermint/tendermint/rpc/jsonrpc/server.maxBytesHandler.ServeHTTP\n\tgithub.com/tendermint/[email protected]/rpc/jsonrpc/server/http_server.go:256\ngithub.com/tendermint/tendermint/rpc/jsonrpc/server.RecoverAndLogHandler.func1\n\tgithub.com/tendermint/[email protected]/rpc/jsonrpc/server/http_server.go:229\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991\nfailed to load state at height 69; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 69): invalid request",

Related issue:

RPC requests during node startup trigger error #14620

alexanderbez · 2023-02-06T20:28:33Z

Part of the issue is solved, notional submitted a fix to prevent chains from starting when adding a store at a different height. How can we help teams recover from this state they are in? Should we write a tool? I think people need to upgrade their chains to recover right?

I think they would need a separate upgrade and corresponding handler to fix the broken state. Either, perhaps, by renaming it or by deleting it and creating it again?

chillyvee · 2023-02-19T11:16:25Z

This just hit kujira testnet v0.8.0 in the alliance store.

tac0turtle · 2023-02-20T09:59:33Z

da, i dont think we can backport the fix due to some chains having this issue. they will get hit with errors. we should write documentation around this

alexanderbez · 2023-02-20T20:04:40Z

Mhhh wait, why is this issue still open? Doesn't #14410 fix it? Also, why can't we backport?

tac0turtle · 2023-02-21T12:07:36Z

i believe a chain with uneven store heights wouldnt be able to start. Is that right?

alexanderbez · 2023-02-21T15:44:42Z

Ahh yes, the fix is NOT retroactive indeed. So existing chains that already have this issue will need to manually patch it in an upgrade.

chillyvee · 2023-02-21T15:51:22Z

i believe a chain with uneven store heights wouldnt be able to start. Is that right?

Chain runs, but pruning, snapshots and statesync are all broken without the patches we issued.

tac0turtle · 2023-02-21T16:34:59Z

ah okay, then we can backport, to help prevent future issues

chillyvee mentioned this issue Nov 21, 2022

Prevent over pruning option cosmos/iavl#624

Closed

faddat mentioned this issue Nov 22, 2022

Pruning issue Stride-Labs/stride#383

Closed

faddat mentioned this issue Nov 22, 2022

initial set of bumps akash-network/node#1719

Closed

10 tasks

faddat changed the title ~~Multi Chain Issue tracker for validator reports~~ uneven store heights prevent pruning and state sync Nov 23, 2022

asalzmann mentioned this issue Nov 26, 2022

v4 upgrade Stride-Labs/stride#393

Merged

19 tasks

asalzmann mentioned this issue Nov 27, 2022

v4/upgrade handler Stride-Labs/stride#384

Closed

19 tasks

tac0turtle added T:Bug T: UX T: Dev UX UX for SDK developers (i.e. how to call our code) labels Nov 27, 2022

asalzmann mentioned this issue Nov 27, 2022

v4 upgrade Stride-Labs/stride#398

Closed

19 tasks

julienrbrt mentioned this issue Jan 15, 2023

panic: cannot delete latest saved version #14625

Closed

tac0turtle added the T:Sprint label Jan 16, 2023

tac0turtle added this to Cosmos-SDK Jan 16, 2023

github-project-automation bot moved this to 📝 Todo in Cosmos-SDK Jan 16, 2023

Eengineer1 mentioned this issue Feb 6, 2023

fix(state-sync): Added state-sync fast-forward to latest state version cheqd/cheqd-node#537

Merged

tac0turtle closed this as completed Feb 21, 2023

github-project-automation bot moved this from 📝 Todo to 👏 Done in Cosmos-SDK Feb 21, 2023

tac0turtle removed this from Cosmos-SDK May 31, 2023

uneven store heights prevent pruning and state sync #13933

uneven store heights prevent pruning and state sync #13933

Comments

faddat commented Nov 20, 2022 • edited Loading

Pruning and/ or state sync

guess

faddat commented Nov 21, 2022

faddat commented Nov 21, 2022

chillyvee commented Nov 21, 2022

chillyvee commented Nov 21, 2022 • edited Loading

chillyvee commented Nov 21, 2022 • edited Loading

faddat commented Nov 22, 2022

chillyvee commented Nov 22, 2022 • edited Loading

faddat commented Nov 22, 2022

Reecepbcups commented Nov 22, 2022

faddat commented Nov 22, 2022 • edited Loading

chillyvee commented Nov 22, 2022

faddat commented Nov 22, 2022

faddat commented Nov 23, 2022

Highlander-maker commented Nov 23, 2022

chillyvee commented Nov 24, 2022 • edited Loading

Likely root cause

Current breakages

Surprisingly not broken

Decline to fix

A better fix

faddat commented Nov 25, 2022

chillyvee commented Nov 25, 2022 • edited Loading

Root cause

Fix

Snapshot Patch

Isolate from peers without snapshot patched

Tested

Need assistance

chillyvee commented Nov 26, 2022

tac0turtle commented Nov 27, 2022

chillyvee commented Nov 27, 2022

chillyvee commented Jan 12, 2023

dsmello commented Jan 16, 2023 • edited Loading

alexanderbez commented Jan 16, 2023

chillyvee commented Jan 17, 2023

chillyvee commented Jan 17, 2023

dsmello commented Jan 17, 2023 • edited Loading

alexanderbez commented Jan 17, 2023

chillyvee commented Jan 17, 2023

chillyvee commented Jan 17, 2023

dsmello commented Jan 17, 2023 • edited Loading

chillyvee commented Jan 19, 2023 • edited Loading

dsmello commented Jan 20, 2023

tac0turtle commented Feb 5, 2023

faddat commented Feb 6, 2023 • edited Loading

alexanderbez commented Feb 6, 2023

chillyvee commented Feb 19, 2023

tac0turtle commented Feb 20, 2023

alexanderbez commented Feb 20, 2023

tac0turtle commented Feb 21, 2023

alexanderbez commented Feb 21, 2023

chillyvee commented Feb 21, 2023

tac0turtle commented Feb 21, 2023

faddat commented Nov 20, 2022 •

edited

Loading

chillyvee commented Nov 21, 2022 •

edited

Loading

chillyvee commented Nov 21, 2022 •

edited

Loading

chillyvee commented Nov 22, 2022 •

edited

Loading

faddat commented Nov 22, 2022 •

edited

Loading

chillyvee commented Nov 24, 2022 •

edited

Loading

chillyvee commented Nov 25, 2022 •

edited

Loading

dsmello commented Jan 16, 2023 •

edited

Loading

dsmello commented Jan 17, 2023 •

edited

Loading

dsmello commented Jan 17, 2023 •

edited

Loading

chillyvee commented Jan 19, 2023 •

edited

Loading

faddat commented Feb 6, 2023 •

edited

Loading