feat: gnovm benchmarking tool

[piux2]

Contributors' checklist...

• [ x] Added new tests, or not needed, or not feasible
• [ x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
• [ x] Updated the official documentation or not needed
• [ x] No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
• [ x] Added references to related issues and PRs
• [ x] Provided any useful hints for running manual tests
• Added new benchmarks to generated graphs, if any. More info here.

We build this tool mainly for the following issues

#1826
#1828
#1281
#1973

We could also use it in the following cases.

#1973
#2222

gnobench benchmarks the time consumed for each VM CPU OpCode and persistent access to the store, including marshalling and unmarshalling of realm objects.

Design consideration

Minimum Overhead and Footprint

• Constant build flags enable benchmarking.
• Encode operations and measurements in binary.
• Dump to a local file in binary.
• No logging, printout, or network access involved.

Accuracy

• Pause the timer for storage access while performing VM opcode benchmarking.
• Measure each OpCode execution in nanoseconds.
• Store access includes the duration for Amino marshalling and unmarshalling.

It is built on top of @deelawn's design and framework with @jaekwon's input.
#2073

Usage

Simple mode

The benchmark only involves the GnoVM and the persistent store. It benchmarks the bare minimum components, and the results are isolated from other components. We use standardize gno contract to perform the benchmarking.

This mode is the best for benchmarking each major release and/or changes in GnoVM.

make opcode
make storage

Production mode

It benchmarks the node in the production environment with minimum overhead.
We can only benchmark with standardize the contract but also capture the live usage in production environment.
It gives us a complete picture of the node perform.

1. Build the production node with benchmarking flags:

go build -tags "benchmarkingstorage benchmarkingops" gno.land/cmd/gnoland

2. Run the node in the production environment. It will dump benchmark data to a benchmark.bin file.

3. call the realm contracts at gno.land/r/x/benchmark/opcodes and gno.land/r/x/benchmark/storage

4. Stop the server after the benchmarking session is complete.

5. Run the following command to convert the binary dump:

gnobench -bin path_to_benchmark_bin

it converts the binary dump to results.csv and results_stats.csv.

Results ( Examples )

The benchmarking results are stored in two files:

1. The raw results are saved in results.csv.

Operation	Elapsed Time	Disk IO Bytes
OpEval	40333	0
OpPopBlock	208	0
OpHalt	167	0
OpEval	500	0
OpInterfaceType	458	0
OpPopBlock	166	0
OpHalt	125	0
OpInterfaceType	21125	0
OpEval	541	0
OpEval	209	0
OpInterfaceType	334	0

2. The averages and standard deviations are summarized in results_stats.csv.

Operation	Avg Time	Avg Size	Time Std Dev	Count
OpAdd	101	0	45	300
OpAddAssign	309	0	1620	100
OpArrayLit	242	0	170	700
OpArrayType	144	0	100	714
OpAssign	136	0	95	2900
OpBand	92	0	30	100
OpBandAssign	127	0	62	100
OpBandn	97	0	54	100
OpBandnAssign	125	0	113	100
OpBinary1	128	0	767	502
OpBody	127	0	145	13700

[codecov]

Codecov Report

Attention: Patch coverage is 16.53061% with 409 lines in your changes missing coverage. Please review.

Project coverage is 60.46%. Comparing base (a73cb22) to head (6c0f085).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
gnovm/pkg/benchops/cmd/stats.go	0.00%	105 Missing ⚠️
gnovm/pkg/gnolang/store.go	15.04%	76 Missing and 20 partials ⚠️
gnovm/pkg/benchops/bench.go	0.00%	53 Missing ⚠️
gnovm/pkg/benchops/exporter.go	0.00%	50 Missing ⚠️
gnovm/pkg/gnolang/machine.go	12.90%	20 Missing and 7 partials ⚠️
gnovm/pkg/benchops/cmd/run.go	63.23%	24 Missing and 1 partial ⚠️
gnovm/pkg/benchops/cmd/main.go	0.00%	23 Missing ⚠️
gnovm/pkg/benchops/cmd/store.go	65.38%	6 Missing and 3 partials ⚠️
gnovm/pkg/benchops/ops.go	0.00%	8 Missing ⚠️
gnovm/pkg/gnolang/realm.go	0.00%	6 Missing and 2 partials ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2241      +/-   ##
==========================================
- Coverage   61.10%   60.46%   -0.64%     
==========================================
  Files         564      572       +8     
  Lines       75355    79491    +4136     
==========================================
+ Hits        46045    48066    +2021     
- Misses      25945    27822    +1877     
- Partials     3365     3603     +238     

Flag	Coverage Δ
contribs/gnodev	61.46% <ø> (ø)
contribs/gnofaucet	15.31% <ø> (ø)
gno.land	67.92% <ø> (ø)
gnovm	65.31% <16.59%> (-0.86%)	⬇️
misc/genstd	80.54% <ø> (ø)
misc/logos	19.88% <ø> (-0.36%)	⬇️
tm2	62.10% <ø> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[deelawn]

Looks good. Perhaps we could move the transient size variable so that is defined inside the benchmarking package as an exported variable. Then it can be set and read strictly within benchmarking constant conditional blocks and avoid any unnecessary allocations when benchmarking is not enabled.

What do you think?

[thehowl]

Could this be reorganised to:

• gnovm/cmd/gnobench + gnovm/pkg/gasbench OR
• contribs/gnobecnh?

I don't want to add another top-level directory.

[jaekwon]

gnovm/pkg/opsbench

[jaekwon]

Waiting for updates as per discussions with Ray