Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug/portal_loop] runtime error: index out of range [2] with length 0 while deploying certain realm on poral-loop #2283

Closed
r3v4s opened this issue Jun 5, 2024 · 4 comments · Fixed by #2504
Assignees
Labels
🐞 bug Something isn't working 📦 ⛰️ gno.land Issues or PRs gno.land package related 📦 🤖 gnovm Issues or PRs gnovm related

Comments

@r3v4s
Copy link
Contributor

r3v4s commented Jun 5, 2024

Description

We're trying to deploy our realms against portal-loop, but pool realm is failing while gnokey maketx addpkg with below error.

$ gnokey maketx addpkg -pkgdir ./pool -pkgpath gno.land/r/demo/p2 -broadcast=true -gas-fee 1ugnot -gas-wanted 100000000 -insecure-password-stdin=true test1 -chainid portal-loop -remote https://rpc.gno.land:443
Enter password.

--= Error =--
Data: internal error
Msg Traces:
    0  /Users/n3wbie/gno_orig/tm2/pkg/crypto/keys/client/maketx.go:213 - deliver transaction failed: log:recovered: runtime error: index out of range [2] with length 0
stack:
goroutine 103295 [running]:
runtime/debug.Stack()
        /opt/hostedtoolcache/go/1.22.3/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).runTx.func1()
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:749 +0x205
panic({0xfb75c0?, 0xc01816b428?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/runtime/panic.go:770 +0x132
go/types.(*Checker).handleBailout(0xc00c057800, 0xc0076e5d40)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/check.go:367 +0x88
panic({0xfb75c0?, 0xc01816b428?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/runtime/panic.go:770 +0x132
github.com/gnolang/gno/gnovm/pkg/gnolang.(*StructValue).GetPointerToInt(...)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/values.go:464
github.com/gnolang/gno/gnovm/pkg/gnolang.(*StructValue).GetPointerTo(0x12aaa60?, {0x12aaa60?, 0xc00aa30280?}, {0x28?, 0xac?, 0x128?, {0xc016960340?, 0x2?}})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/values.go:460 +0xc5
github.com/gnolang/gno/gnovm/pkg/gnolang.(*TypedValue).GetPointerTo(0xc006286028, 0x0, {0x12aaa60, 0xc00aa30280}, {0x0?, 0x0?, 0x0?, {0xc016960340, 0x7}})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/values.go:1716 +0x96d
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).PopAsPointer(0xc00c7fb780?, {0x129b2f0?, 0xc010f48ba0?})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:1953 +0x270
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).doOpAssign(0xc00ea366c8)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/op_assign.go:35 +0x254
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).Run(0xc00ea366c8)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:1380 +0xd73
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runDeclaration(0xc00ea366c8, {0x129c998?, 0xc019f7d570})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:841 +0x265
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runFiles.func1(0xc01eb22608, {0x129c998, 0xc019f7d570})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:609 +0x334
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runFiles(0xc00ea366c8, {0xc00697a988, 0x1, 0x1})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:623 +0x78e
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).RunFiles(...)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:496
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).runMemPackage(0xc00ea366c8, 0xc0147b49c0, 0x1, 0x0)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:287 +0x22d
github.com/gnolang/gno/gnovm/pkg/gnolang.(*Machine).RunMemPackage(...)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/machine.go:255
github.com/gnolang/gno/gno.land/pkg/sdk/vm.(*VMKeeper).initBuiltinPackagesAndTypes.func1({0xc015226c79, 0xf}, {0x12aaa60, 0xc00aa30280})
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/builtins.go:40 +0x245
github.com/gnolang/gno/gnovm/pkg/gnolang.(*defaultStore).GetPackage(0xc00aa30280, {0xc015226c79, 0xf}, 0x18?)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/store.go:170 +0x125
github.com/gnolang/gno/gnovm/pkg/gnolang.(*defaultStore).getMemPackage(0xc00aa30280, {0xc015226c79, 0xf}, 0x0)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/store.go:558 +0xbd
github.com/gnolang/gno/gnovm/pkg/gnolang.(*defaultStore).GetMemPackage(0xedfd00?, {0xc015226c79?, 0xc015226c79?})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/store.go:545 +0x1a
github.com/gnolang/gno/gnovm/pkg/gnolang.(*gnoImporter).ImportFrom(0xc01330b5c0, {0xc015226c79, 0xf}, {0xc0076e52a8?, 0x22acd8?}, 0xf?)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/go2gno.go:540 +0x64
go/types.(*Checker).importPackage(0xc00c057800, {0x1287700, 0xc017779240}, {0xc015226c79, 0xf}, {0x127abc0, 0x1})
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/resolver.go:158 +0x53a
go/types.(*Checker).collectObjects.func1({0x1287880?, 0xc01423b6e0})
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/resolver.go:267 +0x125
go/types.(*Checker).walkDecl(0xc00c057800, {0x128e270?, 0xc01d7b9780}, 0xc0076e5c38)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/decl.go:412 +0x2e7
go/types.(*Checker).walkDecls(...)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/decl.go:399
go/types.(*Checker).collectObjects(0xc00c057800)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/resolver.go:254 +0xe2f
go/types.(*Checker).checkFiles(0xc00c057800, {0xc00aa30320, 0x12, 0x13})
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/check.go:405 +0x17c
go/types.(*Checker).Files(...)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/check.go:372
go/types.(*Config).Check(0xc0091b37a0, {0xc0152260a8?, 0x12?}, 0xc01d7b8200, {0xc00aa30320, 0x12, 0x13}, 0x0)
        /opt/hostedtoolcache/go/1.22.3/x64/src/go/types/api.go:458 +0x79
github.com/gnolang/gno/gnovm/pkg/gnolang.(*gnoImporter).parseCheckMemPackage(0xc01330b5c0, 0xc01d7b8100)
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/go2gno.go:574 +0x366
github.com/gnolang/gno/gnovm/pkg/gnolang.TypeCheckMemPackage(0xc01d7b8100, {0x7fe1a29010f0, 0xc00aa30280})
        /home/runner/work/gno/gno/gnovm/pkg/gnolang/go2gno.go:505 +0x14c
github.com/gnolang/gno/gno.land/pkg/sdk/vm.(*VMKeeper).AddPackage(_, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/keeper.go:166 +0x545
github.com/gnolang/gno/gno.land/pkg/sdk/vm.vmHandler.handleMsgAddPackage({_}, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/handler.go:44 +0xb8
github.com/gnolang/gno/gno.land/pkg/sdk/vm.vmHandler.Process({_}, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/gno.land/pkg/sdk/vm/handler.go:31 +0x358
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).runMsgs(_, {{0x12900f0, 0xc00d19b5f0}, 0x1, {0x128e890, 0xc0161eb150}, {0x1290470, 0xc00d5d09a0}, {0xc006887720, 0xb}, ...}, ...)
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:650 +0x2ed
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).runTx(0xc000582200, 0x1, {0xc004ba2000, 0x129b1, _}, {{0xc00c2ab950, 0x1, 0x1}, {0x5f5e100, {{0xc013722db1, ...}, ...}}, ...})
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:829 +0xae5
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).Simulate(...)
        /home/runner/work/gno/gno/tm2/pkg/sdk/helpers.go:17
github.com/gnolang/gno/tm2/pkg/sdk.handleQueryApp(0xc000582200, {0xc01330aca0?, 0x0?, _}, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, ...})
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:418 +0x25d
github.com/gnolang/gno/tm2/pkg/sdk.(*BaseApp).Query(0xc000582200, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, 0x0, 0x0})
        /home/runner/work/gno/gno/tm2/pkg/sdk/baseapp.go:391 +0x14f
github.com/gnolang/gno/tm2/pkg/bft/abci/client.(*localClient).QuerySync(0xc00abf9220?, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, 0x0, 0x0})
        /home/runner/work/gno/gno/tm2/pkg/bft/abci/client/local_client.go:180 +0x154
github.com/gnolang/gno/tm2/pkg/bft/appconn.(*query).QuerySync(0x7fe1a1d56528?, {{}, {0xc004ba2000, 0x129b1, 0x129b1}, {0xc013722c30, 0xd}, 0x0, 0x0})
        /home/runner/work/gno/gno/tm2/pkg/bft/appconn/app_conn.go:144 +0xdc
github.com/gnolang/gno/tm2/pkg/bft/rpc/core.ABCIQuery(0xc009d8a9c0?, {0xc013722c30, 0xd}, {0xc004ba2000, 0x129b1, 0x129b1}, 0x0?, 0x33?)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/core/abci.go:59 +0x132
reflect.Value.call({0xf0d780?, 0x114d600?, 0x1030d60?}, {0x1033772, 0x4}, {0xc017e95b80, 0x5, 0x5?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/reflect/value.go:596 +0xca6
reflect.Value.Call({0xf0d780?, 0x114d600?, 0x18d29?}, {0xc017e95b80?, 0x1030d60?, 0xe973a0?})
        /opt/hostedtoolcache/go/1.22.3/x64/src/reflect/value.go:380 +0xb9
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.RegisterRPCFuncs.makeJSONRPCHandler.func9({0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/handlers.go:185 +0x745
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.RegisterRPCFuncs.handleInvalidJSONRPCPaths.func10({0x128e800?, 0xc00606ca68?}, 0x0?)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/handlers.go:209 +0x42
net/http.HandlerFunc.ServeHTTP(0xc0008f5340?, {0x128e800?, 0xc00606ca68?}, 0x411dbb?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2166 +0x29
net/http.(*ServeMux).ServeHTTP(0xc008d4b0a0?, {0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2683 +0x1ad
github.com/gnolang/gno/tm2/pkg/bft/node.(*Node).startRPC.(*Cors).Handler.func5({0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /home/runner/go/pkg/mod/github.com/rs/[email protected]/cors.go:281 +0x184
net/http.HandlerFunc.ServeHTTP(0x0?, {0x128e800?, 0xc00606ca68?}, 0xc00140ba10?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2166 +0x29
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.maxBytesHandler.ServeHTTP({{0x1285280?, 0xc01b602de0?}, 0xd?}, {0x128e800, 0xc00606ca68}, 0xc00d8ec360)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/http_server.go:216 +0xfb
github.com/gnolang/gno/tm2/pkg/bft/rpc/lib/server.StartHTTPServer.RecoverAndLogHandler.func1({0x128e740, 0xc01977f420}, 0xc00d8ec360)
        /home/runner/work/gno/gno/tm2/pkg/bft/rpc/lib/server/http_server.go:189 +0x1f8
net/http.HandlerFunc.ServeHTTP(0x412165?, {0x128e740?, 0xc01977f420?}, 0xc01977f401?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2166 +0x29
net/http.serverHandler.ServeHTTP({0x128b898?}, {0x128e740?, 0xc01977f420?}, 0x6?)
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:3137 +0x8e
net/http.(*conn).serve(0xc014b2b440, {0x12900f0, 0xc021ace120})
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2039 +0x5e8
created by net/http.(*Server).Serve in goroutine 74
        /opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:3285 +0x4b4

--= /Error =--

Steps to reproduce on LOCAL [couldn't really reproduce on local. it doesn't panic, but succeeds]

unable to reproduce this issue on local chain, seems to be happening only in portal-loop.

# clone fresh version of gno
$ git clone https://github.com/gnolang/gno gno_orig
$ cd ~/gno_orig

# gnokey install
$ make install


# gnoland install and start
$ cd gno.land
$ make install
$ gnoland start -lazy=true

## cloned gno repo every single commit from 0d299c37d0027edf16de1823b0be824030a2823a ~ 93299802139012ba8e80a99ee406319430ef7277 to re-produce on local but succeeded every commit.


# clone fresh version of gnoswap
$ git clone https://github.com/gnoswap-labs/gnoswap
$ cd gnoswap
$ git switch tmp/portal-loop-deploy-fail

# local deploy test
$ make -f __local/test/test_data.mk init

Tried below ways on portal-loop, but none of them succeeded

  1. Change pkg_path to deploy
  2. Reduce pool contract size (by removing comments, some common functions and etc...)
  3. Set gas-wanted to max
  4. Change client's env ( darwin-arm64, linux-amd64 )

Expected behaviour

pool should be deployed

Actual behaviour

panic

FYI

There is realm that imports pool realm, such as position

Since deploying pool has been failed on portal-loop, deploying position needs to be fail with message like unknown import found gno.land/r/demo/pool. However deploying position also panic with same error(index ouf of range)

@r3v4s r3v4s added help wanted 🐞 bug Something isn't working 📦 ⛰️ gno.land Issues or PRs gno.land package related labels Jun 5, 2024
@r3v4s r3v4s self-assigned this Jun 5, 2024
@zivkovicmilos zivkovicmilos added the 📦 🤖 gnovm Issues or PRs gnovm related label Jun 5, 2024
@thehowl
Copy link
Member

thehowl commented Jun 7, 2024

Okay, leaving an update on my current investigation.

Replication set-up

I'm running a gno.land node with this genesis.json. This is a trimmed-down version of the portal loop genesis.json. With this genesis.json, publishing succeeds.

To run using this genesis file, you'll need to change the validators key so that your local instance is a validator. You can get the fields you need from gnoland-data/secrets/priv_validator_key.json.

Then, I'm running the node with this command:

rm -rf gnoland-data/{db,wal} && gnoland secrets init -force ValidatorState && (go run ./cmd/gnoland start --lazy --skip-failing-genesis-txs | tee gnoland-node.txt)

This allows me to reset the node and start from this genesis. I get the logs redirected to gnoland-node.txt for inspection. Using the two commands at the beginning allows me not to have to update the validator address in genesis each time.

Findings

Running the replication step pointed above by @r3v4s:

make -f __local/test/test_data.mk init CHAINID=portal-loop

As I said in the previous section, publishing succeeds with the provided genesis.json. However, it won't if you first broadcast this transaction: feeds-add.json.

(This should work by simply doing gnokey broadcast of the file; otherwise, try to adjust the signatures)

After that, you get the panic pointed above. (This error happens also if the transaction is in the genesis).

The surprising, and most revealing, fact about this is that if you re-start the node (without clearing the database/state); then publishing works as normal.

Suspicions

Looking through some logs, my suspicions are:

  • That this has to do with encoding/base64, and what happens when handling a failing transaction.
  • That this is because of defaultStore.cacheNodes, which unlike the name would suggest is not a cache, but it is instead re-initialised on every node restart.
  • That the cause of this, is that having a failing transaction makes cacheNodes "dirty", leading to this panic, while there is nothing stored permanently on-chain which can cause the panic; thus why the problem doesn't show up when the node restart.

That's it for this week, I may continue this another time.

@thehowl
Copy link
Member

thehowl commented Jun 7, 2024

Note: I'm trying to get a txtar working before doing an actual fix. If my suspicion is correct, then we'll probably need to discuss either a) removing the caches from the Store (and just using the tm2 store cache), which does support transaction rollback properly, or b) creating a cache system in the Store that can properly handle rollbacks/commits.

@thehowl
Copy link
Member

thehowl commented Jun 10, 2024

#2319 I found a way to replicate the issue; don't know if I'll have time to fix it though

I'm thinking that attempting to fix it would mean trying to make Store properly transactional; so that'll take a while to get working

@thehowl
Copy link
Member

thehowl commented Jun 28, 2024

I'm prioritizing a fix for this one in the next week, to be ready by Gophercon

@moul moul closed this as completed in #2504 Jul 6, 2024
@moul moul closed this as completed in f28444a Jul 6, 2024
gfanton pushed a commit to gfanton/gno that referenced this issue Jul 23, 2024
…ang#2504)

Fixes gnolang#2283 (with a different approach from gnolang#2319)

This PR loads all the standard libraries into the database when
vm.Initialize is called. It doesn't fully fix the problems that gnolang#2319 is
trying to address, but it does fix the most immediate bug of not being
able to publish certain packages on portal loop.

With these changes, we don't need a PackageGetter on-chain anymore. All
packages are already loaded into the store, thus solving the problem
@jaekwon was talking about
[here](gnolang#2319 (comment))
at its root.

This PR has a problem, which is that adding loading the entire stdlibs
in the VM initialization step brings a huge overhead computationally;
this is not a problem on the node, but it is when testing as it needs to
happen very often. This translates to 2x slower txtar tests, compared to
master. On my PC, it adds a 2-3 second overhead whenever running
Initialize.

I tried working out on a system which could save the data to quickly
recover it, at least for some cases where we need to Initialize often;
[see this
diff](https://gist.github.com/thehowl/cb1ee79e63cf77d3f323730580eb2d18).
But I didn't get it to work so far; after copying the DB, attempting
initialization crashes because [ParseMemPackage is being handed a nil
package, for some
reason](https://gist.github.com/thehowl/d1efa51858d865fb5beb9c3a9cb0eeef).
@gfanton, any tips?? :)) I'd want to avoid lazy loading in the node, as
that's what got us here in the first place.

<details><summary>Contributors' checklist...</summary>

- [ ] Added new tests, or not needed, or not feasible
- [x] Provided an example (e.g. screenshot) to aid review or the PR is
self-explanatory
- [x] Updated the official documentation or not needed
- [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message
was included in the description
- [x] Added references to related issues and PRs
- [x] Provided any useful hints for running manual tests
- [x] Added new benchmarks to [generated
graphs](https://gnoland.github.io/benchmarks), if any. More info
[here](https://github.com/gnolang/gno/blob/master/.benchmarks/README.md).
</details>

---------

Co-authored-by: Manfred Touron <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working 📦 ⛰️ gno.land Issues or PRs gno.land package related 📦 🤖 gnovm Issues or PRs gnovm related
Projects
Development

Successfully merging a pull request may close this issue.

3 participants