RPC requests during node startup trigger error #14620

ttl33 · 2023-01-14T01:04:46Z

Summary of Bug

RPC requests to nodes when a node is just getting initialized/started trigger the following error:

"rpc error: code = Unknown desc = codespace sdk code 18: invalid request: failed to load state at height 0; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 0)",

The error is being thrown here when creating the context for the query.

It seems that the IAVL tree does not exist at height 0, so the height hasn't been created yet (here is the VersionExists method on the IAVL struct).

Cosmos SDK version previously returned an empty IAVL tree in this scenario. PR that introduced the change is #13355.

This might be related to this other issue.

Version

v0.47.0-alpha2

Steps to Reproduce

Create some task/job that call a node RPC endpoint
Start a new node

The text was updated successfully, but these errors were encountered:

facundomedica · 2023-01-14T01:13:46Z

To me it makes sense that it returns an error. Height 0 isn't a thing, and this error looks easily handleable on the client's side (I might be very wrong tho).

ttl33 · 2023-01-17T19:43:37Z

Yes, it can be handled on the client side, but my focus is mostly on how the request is being handled on the node (server) side.

It's odd that the node would throw an error if "height 0 is not a thing". I would assume that the node would reject the call until the setup has been complete. It feels like the node is opening up RPC endpoints before the node is fully ready, which doesn't quite feel right.

IMO, it's better for the node to be defensive and gracefully handle these requests before the node is ready to serve these requests. Let me know your thoughts!

alexanderbez · 2023-01-18T16:58:25Z

Which RPC endpoint are you referring to exactly? There's a bunch of moving pieces (Tendermint has plenty) and then the app itself, so it's not that straightforward.

ttl33 · 2023-01-19T02:01:09Z

We ran into errors when making gRPC query requests. I can't share the exact gRPC query method (given that our codebase is private for now), but it's essentially any of the module specific query methods like this one

alexanderbez · 2023-01-19T18:30:40Z

And you made these queries right as the app started up?

arhamchordia · 2023-02-02T07:09:08Z

I tried making the chain run using sdk v0.46.4. And got a similar error but at a different height. It occurs when I try to query the chain.
Error: rpc error: code = InvalidArgument desc = failed to load state at height 10; version does not exist (latest height: 10): invalid request
It fails while trying to createQueryContext(). Enters this function and fails for feegrant while trying to do this.

alexanderbez · 2023-02-02T16:20:37Z

@arhamchordia was this on an upgraded binary?

arhamchordia · 2023-02-03T08:58:15Z

@arhamchordia was this on an upgraded binary?

@alexanderbez I tried it on simd and there was no such error.
So, do you think it depends on how things are declared in app.go?

alexanderbez · 2023-02-03T20:31:47Z

No, when you see that error it's a function of your pruning settings and if the binary was upgraded or not. So did you upgrade the network and what are your pruning settings?

arhamchordia · 2023-02-06T06:40:19Z

No, when you see that error it's a function of your pruning settings and if the binary was upgraded or not. So did you upgrade the network and what are your pruning settings?

I tried starting it up from scratch with a fresh genesis. Also the pruning settings are set to "default".

After attempting to query multiple time. It ends up with an error and chain halts: maximum number of retries exceeded, last error: rpc error: code = Unknown desc = codespace sdk code 18: invalid request: failed to load state at height 30; version does not exist (latest height: 30)

Reecepbcups · 2023-02-06T18:54:59Z

We are seeing the same thing on Juno after bumping up to SDK 45.12 in our e2e. I noticed this as well before we bumped tendermint to the 35.25 fix too.

https://github.com/CosmosContracts/juno/actions/runs/4106733477/jobs/7085521033#step:9:890

Additional Context:

upgrade height 71
upgrade goes through, then 5 blocks later it says upgradeHeight-2 (69 in this case) is bad after it queries PEX

failed to load state at height 69; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 69)

I have replicated this locally without any custom modules too. So feel its something with the patch used to fix this issue

faddat · 2023-02-06T19:51:30Z

Yeah I recommend that we re-open this.

Something is off.

alexanderbez · 2023-02-06T20:27:22Z

@Reecepbcups what are the pruning settings in your instance? Are heights 70 and 71 queryable? What about 68?

Reecepbcups · 2023-02-07T06:21:20Z

@alexanderbez pruning = nothing, single and multi node setups

Other blocks around it are querying just fine (after I start the node back, since panic = exit)

I patched it in my fork like so (how it was in 45.11) and it works just fine.

Reecepbcups@9fa1c52

My theories:

the new change in 45.12 (error instead of nil) has an incorrect panic from a calling function somewhere but then:
the IAVL store is losing reference to versions or something? Really odd.

Looking into more this week

alexanderbez · 2023-02-07T15:20:34Z

the new change in 45.12 (error instead of nil) has an incorrect panic from a calling function somewhere but then:

This seems more probable to me. In fact, after some quick digging it is the case ->

CreateQueryContext -> CacheMultiStoreWithVersion

Now that it errors, CreateQueryContext will return that error. What we need to do is return a typed error and ignore it in CreateQueryContext. I'll open a PR.

the IAVL store is losing reference to versions or something? Really odd.

I'm a bit suspect on this, I doubt it.

alexanderbez · 2023-02-07T15:33:53Z

ref: #13355

So it seems this was intentional. To return nil and error, meaning the version does not exist. So why is it the case that on a height that pre-upgrade it does not exist?

Reecepbcups · 2023-02-07T15:38:03Z

@alexanderbez So we just had like 5 people look through this through the night
turns out out we were missing a storeKey in our upgrade handler. The reason being: we added this in a previous upgrade on mainnet (v8)
So in the localnet/testnet environment, it fails because the key has not been added yet

Maybe on IAVL error should mention version AND dump the diff of root keys before and after the upgrade? not sure if this is possible but will remove so much headache in the future

alexanderbez · 2023-02-07T16:30:34Z

You mean you forgot to set the added field? If so, we already patched that.

dzmitryhil · 2023-03-15T08:30:10Z

@Reecepbcups could you please elaborate a bit on the changes you made to fix that issue. We faced the similar.
After the upgrade where we introduce 2 new modules and update the cosmos from v0.45.11 to v0.45.14 we get the error on any RPC request.
Upgrade handler

app.UpgradeKeeper.SetUpgradeHandler("upgrade", func(ctx sdk.Context, plan upgradetypes.Plan, fromVM module.VersionMap) (module.VersionMap, error) {
			return app.mm.RunMigrations(ctx, app.configurator, fromVM)
		})

Error

github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).createQueryContext
	github.com/cosmos/[email protected]/baseapp/abci.go:674
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).handleQueryGRPC
	github.com/cosmos/[email protected]/baseapp/abci.go:596
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Query
	github.com/cosmos/[email protected]/baseapp/abci.go:445
github.com/tendermint/tendermint/abci/client.(*localClient).QuerySync
	github.com/tendermint/[email protected]/abci/client/local_client.go:256
github.com/tendermint/tendermint/proxy.(*appConnQuery).QuerySync
	github.com/tendermint/[email protected]/proxy/app_conn.go:159
github.com/tendermint/tendermint/rpc/core.ABCIQuery
	github.com/tendermint/[email protected]/rpc/core/abci.go:20
github.com/tendermint/tendermint/rpc/client/local.(*Local).ABCIQueryWithOptions
	github.com/tendermint/[email protected]/rpc/client/local/local.go:87
github.com/cosmos/cosmos-sdk/client.Context.queryABCI
	github.com/cosmos/[email protected]/client/query.go:94
github.com/cosmos/cosmos-sdk/client.Context.QueryABCI
	github.com/cosmos/[email protected]/client/query.go:57
github.com/cosmos/cosmos-sdk/client.Context.Invoke
	github.com/cosmos/[email protected]/client/grpc_query.go:81
github.com/CoreumFoundation/coreum/x/customparams/types.(*queryClient).StakingParams
	github.com/CoreumFoundation/coreum/x/customparams/types/query.pb.go:174
github.com/CoreumFoundation/coreum/x/customparams/types.request_Query_StakingParams_0
	github.com/CoreumFoundation/coreum/x/customparams/types/query.pb.gw.go:40
github.com/CoreumFoundation/coreum/x/customparams/types.RegisterQueryHandlerClient.func1
	github.com/CoreumFoundation/coreum/x/customparams/types/query.pb.gw.go:133
github.com/grpc-ecosystem/grpc-gateway/runtime.(*ServeMux).ServeHTTP
	github.com/grpc-ecosystem/[email protected]/runtime/mux.go:240
github.com/gorilla/mux.(*Router).ServeHTTP
	github.com/gorilla/[email protected]/mux.go:210
github.com/gorilla/handlers.(*cors).ServeHTTP
	github.com/gorilla/[email protected]/cors.go:54
github.com/tendermint/tendermint/rpc/jsonrpc/server.maxBytesHandler.ServeHTTP
	github.com/tendermint/[email protected]/rpc/jsonrpc/server/http_server.go:256
github.com/tendermint/tendermint/rpc/jsonrpc/server.RecoverAndLogHandler.func1
	github.com/tendermint/[email protected]/rpc/jsonrpc/server/http_server.go:229
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2122
net/http.serverHandler.ServeHTTP
	net/http/server.go:2936
net/http.(*conn).serve
	net/http/server.go:1995
failed to load state at height 46; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 46): invalid request

Reecepbcups · 2023-03-15T13:09:44Z

@dzmitryhil You have to add the storeKeys for those modules as well. This is an extra thing on top of the upgrade handler

https://github.com/CosmosContracts/juno/blob/main/app/upgrades/v13/constants.go#L22

(This will more than likely be in your app.go, we have a different format to make upgrades cleaner)

In our case, we missed icacontrollertypes.StoreKey, since it was already on mainnet from a previous release, but not on our reset testnet

dzmitryhil · 2023-03-15T16:08:23Z

Thanks a lot @Reecepbcups, it helped.

lakefishingman522 · 2023-06-27T20:00:26Z

I am using cosmos sdk v0.46.13 and when I am trying to send tx to tendermint, I got follow messages.

rpc error: code = Unknown desc = codespace sdk code 18: invalid request: failed to load state at height 1; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 1361)

is this related to my hardhat task script or my tendermint side? how to fix this?

violog · 2024-04-10T13:14:51Z

On command query gov proposals I received a similar error:

Error: rpc error: code = InvalidArgument desc = failed to load state at height 46; version does not exist (latest height: 46): invalid request

My app is built on top of Cosmos SDK v0.46.7. However, the SDK itself is not an issue. Try to find indirect dependency in your go.mod: github.com/syndtr/goleveldb. On versions:

v1.0.1-0.20210819022825-2ae1ddf74ef7 no issue
v1.0.1-0.20220721030215-126854af5e6d the issue is reproduced

Hope it saves you some extra hours of debug 🙌

tac0turtle added T:Bug T:Sprint labels Jan 17, 2023

tac0turtle added this to Cosmos-SDK Jan 17, 2023

github-project-automation bot moved this to 📝 Todo in Cosmos-SDK Jan 17, 2023

julienrbrt self-assigned this Jan 19, 2023

julienrbrt moved this from 📝 Todo to 💪 In Progress in Cosmos-SDK Jan 19, 2023

julienrbrt mentioned this issue Jan 19, 2023

fix: improve rpc message when app is at height 0 #14692

Merged

19 tasks

julienrbrt moved this from 💪 In Progress to 👀 Needs Review in Cosmos-SDK Jan 19, 2023

julienrbrt closed this as completed in #14692 Jan 19, 2023

github-project-automation bot moved this from 👀 Needs Review to 👏 Done in Cosmos-SDK Jan 19, 2023

faddat mentioned this issue Feb 6, 2023

uneven store heights prevent pruning and state sync #13933

Closed

pharr117 mentioned this issue Apr 3, 2023

cosmoshub-4 - initial support. bryanlabs/cosmos-tax-cli#334

Closed

taariq mentioned this issue Apr 14, 2023

Bug: Palomad starts with updated genesis, but blocks are not being created. No errors in logs, but no blocks palomachain/paloma#796

Closed

tac0turtle removed this from Cosmos-SDK May 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RPC requests during node startup trigger error #14620

RPC requests during node startup trigger error #14620

ttl33 commented Jan 14, 2023

facundomedica commented Jan 14, 2023

ttl33 commented Jan 17, 2023

alexanderbez commented Jan 18, 2023

ttl33 commented Jan 19, 2023 •

edited

Loading

alexanderbez commented Jan 19, 2023

arhamchordia commented Feb 2, 2023 •

edited

Loading

alexanderbez commented Feb 2, 2023

arhamchordia commented Feb 3, 2023

alexanderbez commented Feb 3, 2023

arhamchordia commented Feb 6, 2023 •

edited

Loading

Reecepbcups commented Feb 6, 2023 •

edited

Loading

faddat commented Feb 6, 2023

alexanderbez commented Feb 6, 2023

Reecepbcups commented Feb 7, 2023 •

edited

Loading

alexanderbez commented Feb 7, 2023

alexanderbez commented Feb 7, 2023

Reecepbcups commented Feb 7, 2023

alexanderbez commented Feb 7, 2023

dzmitryhil commented Mar 15, 2023

Reecepbcups commented Mar 15, 2023 •

edited

Loading

dzmitryhil commented Mar 15, 2023

lakefishingman522 commented Jun 27, 2023

violog commented Apr 10, 2024

RPC requests during node startup trigger error #14620

RPC requests during node startup trigger error #14620

Comments

ttl33 commented Jan 14, 2023

Summary of Bug

Version

Steps to Reproduce

facundomedica commented Jan 14, 2023

ttl33 commented Jan 17, 2023

alexanderbez commented Jan 18, 2023

ttl33 commented Jan 19, 2023 • edited Loading

alexanderbez commented Jan 19, 2023

arhamchordia commented Feb 2, 2023 • edited Loading

alexanderbez commented Feb 2, 2023

arhamchordia commented Feb 3, 2023

alexanderbez commented Feb 3, 2023

arhamchordia commented Feb 6, 2023 • edited Loading

Reecepbcups commented Feb 6, 2023 • edited Loading

faddat commented Feb 6, 2023

alexanderbez commented Feb 6, 2023

Reecepbcups commented Feb 7, 2023 • edited Loading

alexanderbez commented Feb 7, 2023

alexanderbez commented Feb 7, 2023

Reecepbcups commented Feb 7, 2023

alexanderbez commented Feb 7, 2023

dzmitryhil commented Mar 15, 2023

Reecepbcups commented Mar 15, 2023 • edited Loading

dzmitryhil commented Mar 15, 2023

lakefishingman522 commented Jun 27, 2023

violog commented Apr 10, 2024

ttl33 commented Jan 19, 2023 •

edited

Loading

arhamchordia commented Feb 2, 2023 •

edited

Loading

arhamchordia commented Feb 6, 2023 •

edited

Loading

Reecepbcups commented Feb 6, 2023 •

edited

Loading

Reecepbcups commented Feb 7, 2023 •

edited

Loading

Reecepbcups commented Mar 15, 2023 •

edited

Loading