feat(gnovm): improved native bindings (#859)

This PR provides an initial transition and groundwork to move from native bindings as "package injections", whereby an injector function is called after package parsing and loading to provide built-in natives, to the new design described in #814. ## Status The PR is currently ready for review. - [x] Fix all tests and ensure precompiling has no issues - [x] Find out if the errors that arise when running the tests with `DEBUG=1` are bugs introduced by this PR or not - [x] Create temporary exceptions for linked types (like `std.{Coins,Address,Realm}`), so that all normal native bindings run exclusively on the new system, awaiting `//gno:link` (see #814). - [x] Finding a permanent place for ExecContext (see comment on `gnovm/stdlibs/context.go`); suggestions welcome (tm2/vm?) - [x] Include code generation in Makefiles and add a CI check to make sure PRs do generate code when necessary - [x] And some unit tests for the new functionality / potential edge cases. - [x] make sure all injected functions have `// injected` as a comment Next steps, after merging this PR, include: - Supporting precompilation of native functions by linking directly to the Go source in stdlibs/, removing the necessity of `stdshim`. - Supporting the test-only tests/stdlibs/ directory in `gno doc`; as well as documenting native functions explicitly by adding the `// injected` comment after their declaration. - Making all of the native-only packages defined in tests/imports.go native bindings; so that the special packages and native values are restricted only to ImportModeNativePreferred, while everything else uses documentable native bindings also for tests. ## Implementation Summary A new API has been added to `gnolang.Store`: `NativeStore`. This is a simple function which resolves a package and a name of a function to a native function.[^1] This is used to resolve native bindings first and foremost in `preprocess.go`, which also adds information about the native binding in `NativePkg` and `NativeName`, two new fields in `*gnolang.FuncValue`. The purpose of `NativePkg` and `NativeName` is to 1. enable assigning native functions to other function types, including function parameters or func type variables and 2. being able to easily and effectively persist the FuncValue during realm de/serialization. This way, `op_call.go` can recover the value of `nativeBody` if it is nil. On the other side, examples of the new stdlibs can be seen in the `gnovm/stdlibs` directory, where everything except a few functions in package `std` have been ported to the new system. The code generator in `misc/genstd` parses the AST of both the Go and Gno files, and ultimately generates `native.go`, which contains the porting code from the Gno function call to Go and back. To support the test contexts, which needs to have different function implementation or entirely new functions, an additional stdlibs directory was added to the `tests` directory. This is still code generated with the same generator and follows the same structure. When stdlibs are loaded for tests, the code in tests/stdlibs/, if it exists, is loaded with "higher priority" compared to normal `stdlibs` code.[^2] ### Suggested order for reviewing (and some notes) 1. `pkg/gnolang` for the core of the changes, starting from `values.go`, `preprocess.go`, `op_call.go`, `store.go`, `realm.go` which show how NativePkg, NativeName and nativeBody interact * `gnovm/pkg/gnolang/op_eval.go` adds the type of the evaluated expression, as I think it's useful to understand in what terms something is being evaluated; specifically I added it to better distinguish between `DEBUG: EVAL: (*gnolang.CallExpr) std<VPBlock(3,0)>.TestCurrentRealm()` and `DEBUG: EVAL: (*gnolang.SelectorExpr) std<VPBlock(3,0)>.TestCurrentRealm`. I think it's useful, but we can remove it if we prefer to keep it simpler. * `gnovm/pkg/gnolang/op_expressions.go` adds a new debug print pretending we're popping and pushing the value stack. This is to make clear what's happening in the debug logs (we're really swapping the value at the pointer), and I've made it `v[S]` so it's clear that the log is not coming from `PushValue/PopValue`. * I removed a goto-loop in `gnovm/pkg/gnolang/values.go` because I don't see it as beneficial when it's essentially just doing a for loop; if the concern was inlining, even with goto the function is too "costly" for the go compiler to inline. 2. `misc/genstd` contains the code generator and the template it uses to generate the files, ie. `gnovm/stdlibs/native.go` and `gnovm/tests/stdlibs/native.go`. 3. `stdlibs` and `tests/stdlibs` changes show the new stdlib directories, which genstd uses to generate code. The `tests/stdlibs` is available only in test environments, and there are also some overriding definitions to those in normal stdlibs. * `gnovm/stdlibs/encoding/base64/base64.gno` removes a check on `strconv.IntSize`. This is because `strconv.IntSize` is dependent on the system and thus is non-deterministic behaviour on the Gnovm. I don't know if `int` in Gno is == Go's `int` or it is precisely defined to be 64-bits; in any case I don't think it should be dependent on the system as this wouldn't work in a blockchain scenario. 4. Changes to `tm2` and `gnovm/tests/imports.go` mostly relate to wiring up the new NativeStore to the gnovm where it's used. 5. Changes to `examples` and `tests/files` (except for float5) are all updates to golden tests. <details><summary>Checklists...</summary> <p> ## Contributors Checklist - [x] Added new tests, or not needed, or not feasible - [x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [x] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [x] Provided any useful hints for running manual tests ## Maintainers Checklist - [ ] Checked that the author followed the guidelines in `CONTRIBUTING.md` - [ ] Checked the conventional-commit (especially PR title and verb, presence of `BREAKING CHANGE:` in the body) - [ ] Ensured that this PR is not a significant change or confirmed that the review/consideration process was appropriate for the change </p> </details> [^1]: Currently, the only native bindings that are supported with the new system are top-level functions. I think we can keep it this way in the spirit of [KISS](https://en.wikipedia.org/wiki/KISS_principle); maybe think of extending it if we find appropriate usage. [^2]: See gnolang.RunMemPackageWithOverrides --------- Co-authored-by: Thomas Bruyelle <[email protected]> Co-authored-by: jaekwon <[email protected]>
gnolang · Dec 18, 2023 · 8457f7f · 8457f7f
1 parent 64b7d88
commit 8457f7f
Show file tree

Hide file tree

Showing 105 changed files with 4,392 additions and 1,239 deletions.
diff --git a/.gitattributes b/.gitattributes
@@ -1,3 +1,5 @@
 *.gno linguist-language=Go
 *.pb.go linguist-generated merge=ours -diff
 go.sum linguist-generated text
+gnovm/stdlibs/native.go linguist-generated
+gnovm/tests/stdlibs/native.go linguist-generated
diff --git a/.github/codecov.yml b/.github/codecov.yml
@@ -52,3 +52,6 @@ flag_management:
     - name: gno.land
       paths:
       - gno.land
+    - name: misc
+      paths:
+      - misc
diff --git a/.github/workflows/codegen.yml b/.github/workflows/codegen.yml
@@ -0,0 +1,36 @@
+name: code generation
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+    paths:
+      - 'gnovm/stdlibs/**'
+      - 'gnovm/tests/stdlibs/**'
+      - 'misc/genstd'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  generated:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: 1.21.x
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Check generated files are up to date
+        run: |
+          go generate -x ./...
+          if [ "$(git status -s)" != "" ]; then
+            echo "command 'go generate' creates file that differ from git tree, please run 'go generate' and commit:"
+            git status -s
+            exit 1
+          fi
+
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -0,0 +1,90 @@
+name: lint
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: 1.21.x
+
+      - name: Lint
+        uses: golangci/golangci-lint-action@v3
+        with:
+          # sync with misc/devdeps/go.mod
+          version: v1.54
+          args:
+            --config=./.github/golangci.yml
+  fmt:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: 1.21.x
+
+      - name: Install make
+        run: sudo apt-get install -y make
+
+      # prefill dependencies so that mod messages don't show up in make output
+      - name: Fetch dependencies
+        run: go mod download -modfile ./misc/devdeps/go.mod -x
+
+      # inspired by:
+      # https://github.com/Jerome1337/gofmt-action/blob/d5eabd189843f1d568286a54578159978b7c0fb1/entrypoint.sh
+      - name: Check gofumpt
+        run: |
+          output="$(GOFMT_FLAGS=-l make -s fmt)"
+          if [ ! -z "$output" ]; then
+            echo "The following files are not properly formatted; run 'make fmt' to format them."
+            echo "$output"
+            exit 1
+          else
+            echo 'Succeeded.'
+          fi
+  modtidy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: 1.21.x
+
+      - name: Check go.mods
+        run: |
+          set -xe
+          # Find all go.mod files
+          gomods=$(find . -type f -name go.mod)
+
+          # Calculate sums for all go.mod files
+          sums=$(sha256sum $gomods)
+
+          # Iterate over each go.mod file
+          for modfile in $gomods; do
+            dir=$(dirname "$modfile")
+
+            # Run go mod tidy in the directory
+            (cd "$dir" && go mod tidy -v) || exit 1
+          done
+
+          # Verify the sums
+          echo "$sums" | sha256sum -c
diff --git a/.github/workflows/misc.yml b/.github/workflows/misc.yml
@@ -1,90 +1,80 @@
+# tests the "misc" directory & tools
+# (not meant for miscellaneous workflows)
 name: misc
 
 on:
+  pull_request:
+    paths:
+      - "misc/genstd/**.go"
+      - "misc/Makefile"
+      - ".github/workflows/misc.yml"
+      # Until the codecov issue is resolved, it's essential to run the tests for gnovm, tm2, misc, and gno.land concurrently.
+      - "gnovm/**"
+      - "tm2/**"
+      - "gno.land/**"
+      - "examples/**"
+      - ".github/workflows/**"
   push:
     branches: [ "master" ]
-  pull_request:
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
 jobs:
-  lint:
+  build:
+    strategy:
+      fail-fast: false
+      matrix:
+        goversion:
+          - "1.21.x"
+        program:
+          - "genstd"
     runs-on: ubuntu-latest
+    timeout-minutes: 5
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-
       - name: Install Go
         uses: actions/setup-go@v4
         with:
-          go-version: 1.21.x
-
-      - name: Lint
-        uses: golangci/golangci-lint-action@v3
-        with:
-          # sync with misc/devdeps/go.mod
-          version: v1.54
-          args:
-            --config=./.github/golangci.yml
-  fmt:
-    runs-on: ubuntu-latest
-    steps:
+          go-version: ${{ matrix.goversion }}
       - name: Checkout code
         uses: actions/checkout@v4
 
-      - name: Install Go
-        uses: actions/setup-go@v4
-        with:
-          go-version: 1.21.x
-
-      - name: Install make
-        run: sudo apt-get install -y make
+      - name: go install
+        working-directory: misc
+        run: go install ./${{ matrix.program }}
 
-      # prefill dependencies so that mod messages don't show up in make output
-      - name: Fetch dependencies
-        run: go mod download -modfile ./misc/devdeps/go.mod -x
-
-      # inspired by:
-      # https://github.com/Jerome1337/gofmt-action/blob/d5eabd189843f1d568286a54578159978b7c0fb1/entrypoint.sh
-      - name: Check gofumpt
-        run: |
-          output="$(GOFMT_FLAGS=-l make -s fmt)"
-          if [ ! -z "$output" ]; then
-            echo "The following files are not properly formatted; run 'make fmt' to format them."
-            echo "$output"
-            exit 1
-          else
-            echo 'Succeeded.'
-          fi
-  modtidy:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        goversion:
+          - "1.21.x"
+        args:
+          - _test.genstd
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
 
       - name: Install Go
         uses: actions/setup-go@v4
         with:
-          go-version: 1.21.x
+          go-version: ${{ matrix.goversion }}
 
-      - name: Check go.mods
+      - name: Test
+        working-directory: misc
         run: |
-          set -xe
-          # Find all go.mod files
-          gomods=$(find . -type f -name go.mod)
-          
-          # Calculate sums for all go.mod files
-          sums=$(sha256sum $gomods)
-          
-          # Iterate over each go.mod file
-          for modfile in $gomods; do
-            dir=$(dirname "$modfile")
-          
-            # Run go mod tidy in the directory
-            (cd "$dir" && go mod tidy -v) || exit 1
-          done
-          
-          # Verify the sums
-          echo "$sums" | sha256sum -c
+          export GOPATH=$HOME/go
+          export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic"
+          make ${{ matrix.args }}
+
+      - if: runner.os == 'Linux'
+        uses: codecov/codecov-action@v3
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          name: misc
+          flags: misc,misc-${{matrix.args}},go-${{ matrix.goversion }}
+          files: ./misc/coverage.out
+          fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}
diff --git a/Makefile b/Makefile
@@ -25,6 +25,7 @@ test.components:
 	$(MAKE) --no-print-directory -C gnovm    test
 	$(MAKE) --no-print-directory -C gno.land test
 	$(MAKE) --no-print-directory -C examples test
+	$(MAKE) --no-print-directory -C misc     test
 
 .PHONY: test.docker
 test.docker: