-
Notifications
You must be signed in to change notification settings - Fork 379
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(gnovm): test execution refactor + speedup (#3119)
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right). ![filetests](https://github.com/user-attachments/assets/049680f2-baeb-4f24-8f0f-60ae5fa4bce5) - Fixes #1084 - Fixes #2826 (by addressing root cause of slowness) - Fixes #588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [![Open Source Saturday](https://img.shields.io/badge/%E2%9D%A4%EF%B8%8F-open%20source%20saturday-F64060.svg)](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <[email protected]>
- Loading branch information
Showing
320 changed files
with
1,928 additions
and
4,452 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -99,3 +99,4 @@ func main() { | |
// | ||
// ## Page 7 of 6 | ||
// [1](?page=1) | … | [5](?page=5) | [6](?page=6) | _7_ | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -44,6 +44,7 @@ func main() { | |
} | ||
|
||
// Output: | ||
// | ||
// -- INITIAL | ||
// | ||
// # Gnome 😃 | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,3 +22,4 @@ func main() { | |
// List of all Groups: | ||
// | ||
// * [test_group](/r/demo/groups:test_group) | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -76,3 +76,5 @@ func main() { | |
// Group Last MemberID: 0000000001 | ||
// | ||
// Group Members: | ||
// | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,3 +21,5 @@ func main() { | |
// Output: | ||
// 1 | ||
// List of all Groups: | ||
// | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -49,3 +49,4 @@ func main() { | |
// | ||
// * various improvements | ||
// * new shiny logo | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -33,3 +33,5 @@ func main() { | |
// | ||
// | ||
// Per request limit: 350000000ugnot | ||
// | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -33,3 +33,5 @@ func main() { | |
// | ||
// | ||
// Per request limit: 350000000ugnot | ||
// | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -17,3 +17,5 @@ func main() { | |
// | ||
// ## Personal ToDo List | ||
// - [ ] fill this todo list... | ||
// | ||
// |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -33,3 +33,5 @@ func main() { | |
// - [ ] bbb | ||
// - [ ] ddd | ||
// - [ ] eee | ||
// | ||
// |
Oops, something went wrong.