Add reporting to goose shoot and improve error handling (#42)

* Add reporting to goose shoot and improve error handling

* Add `num_blocks` field to `last_x_blocks_metrics`

* Use return instead of ? to clarify intent for set_failure

* Abstract out making individual reports

* Update README and rename Mean to Average

* Refactors for goose reporting

* Fix goose wait_for_tx not found case

* Rename report location and remove SysInfo Default impl

* Create GOOSE_TIME_UNIT constant

* cargo fmt

* Remove outdated comment

* cargo fmt

README.md

@@ -85,11 +85,12 @@ The configuration is defined by the following spec
 
   - `num_erc20_transfers`: Number of ERC20 `transfer` transactions
   - `num_erc721_mints`: Number of ERC721 `mint` transactions
+  - `concurrency`: How many transactions to do simultaneously
 
 - `report`
 
   - `num_blocks`: Number of last blocks to take into account in the report
-  - `reports_dir`: Path to the directory where to save the reports
+  - `output_location`: Path to the file where to save the reports
 
 - `deployer`
 
@@ -103,6 +104,39 @@ The configuration is defined by the following spec
 gatling shoot -c config/default.yaml
 ```
 
+### Output
+
+The main output of gomu gomu is the report output location specified in specified in the configuration file.
+
+At the end of all benchmarks gomu gomu will collect the results into a single json file with the following structure:
+
+- `users`: The amount of goose users used to do the benchmarks, changed by `concurrency`
+
+- `all_bench_report`: A report over all benchmarks done, has a portion of metrics that `benches` has
+
+- `benches`: A array of reports for all benchmarks
+
+  - `name`: The name of the benchmark
+  - `amount`: How many times this benchmark was ran
+  - `metrics`: Metrics over the whole benchmark
+
+    - `name`: The name of the metric
+    - `unit`: The unit of the metric, empty when there is no unit
+    - `value`: The metrics value, a number
+
+      - For floats, `Infinite` and `NaN` are not JSON numbers and thus will be turned into `null`
+      - Values gotten from submission time are calculated from the latency to add a new transaction to the node
+      - Values gotten from verification time are calculated from the latency to get the transaction receipt after the transactions have been processed
+
+  - `last_x_blocks_metrics`: Metrics over the last blocks of the benchmark
+
+    - `num_blocks`: The amount of last transactions that were measured
+    - `metrics`: An array of metrics
+
+- `extra`: Extra information for this run
+
+Gomu gomu will also display into the console information about each step in the benchmark.
+
 ## Resources
 
 - Gomu Gomu is originally inspired from [Flood](https://github.com/paradigmxyz/flood)

config/default.yaml

@@ -22,7 +22,7 @@ run:
 
 report:
   num_blocks: 4
-  reports_dir: "reports"
+  output_location: "report"
 
 deployer:
   salt: "1"

config/katana.yaml

@@ -22,7 +22,7 @@ run:
 
 report:
   num_blocks: 4
-  reports_dir: "reports"
+  output_location: "report"
 
 deployer:
   salt: "1"

config/sharingan.yaml

@@ -22,7 +22,7 @@ run:
 
 report:
   num_blocks: 4
-  reports_dir: "reports"
+  output_location: "report"
 
 # need to add an adress and signing key on sharingan network
 deployer:

config/v2.1.0.yaml

@@ -28,7 +28,7 @@ run:
 
 report:
   num_blocks: 3
-  reports_dir: "reports"
+  output_location: "report"
 
 deployer:
   salt: "1"