Skip to content

Commit

Permalink
go/oasis-test-runner: Refactor code to refer to scenario(s) consistently
Browse files Browse the repository at this point in the history
go/oasis-test-runner/cmd: Rename --test flag to --scenario (-s for short
version) for consistency.

go/oasis-test-runner/env:
- Rename TestInstanceInfo type to ScenarioInstanceInfo and its Test field
  to Scenario for consistency.
- Rename Env's TestInfo() and WriteTestInfo() methods to ScenarioInfo()
  and WriteScenarioInfo() for consistency.

go/oasis-node/cmd/common/metrics: Use "scenario" metrics label instead of
"test" for consistency.
  • Loading branch information
tjanez committed Jul 14, 2020
1 parent 63fc250 commit e81f7d7
Show file tree
Hide file tree
Showing 15 changed files with 481 additions and 271 deletions.
3 changes: 3 additions & 0 deletions .changelog/3108.breaking.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
go/oasis-test-runner/cmd: Rename `--test` flag to `--scenario` flag

Rename the short version from `-t` to `-s`.
7 changes: 7 additions & 0 deletions .changelog/3108.breaking.2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
go/oasis-test-runner/env: Rename types, fields, functions to refer to scenario

Rename `TestInstanceInfo` type to `ScenarioInstanceInfo` and its `Test` field
to `Scenario`.

Rename `Env`'s `TestInfo()` and `WriteTestInfo()` methods to `ScenarioInfo()`
and `WriteScenarioInfo()` for consistency.
1 change: 1 addition & 0 deletions .changelog/3108.breaking.3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
go/oasis-node/cmd/common/metrics: Rename "scenario" metrics label to "test"
1 change: 1 addition & 0 deletions .changelog/3108.internal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
go/oasis-test-runner: Refactor code to refer to scenario(s) consistently
8 changes: 4 additions & 4 deletions go/oasis-node/cmd/common/metrics/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ const (
MetricsLabelInstance = "instance"
MetricsLabelRun = "run"
MetricsLabelSoftwareVersion = "software_version"
MetricsLabelTest = "test"
MetricsLabelScenario = "scenario"

MetricsModeNone = "none"
MetricsModePull = "pull"
Expand All @@ -51,7 +51,7 @@ var (
UpGauge = prometheus.NewGauge(
prometheus.GaugeOpts{
Name: MetricUp,
Help: "Is oasis-test-runner active for specific test.",
Help: "Is oasis-test-runner active for specific scenario.",
},
)
)
Expand Down Expand Up @@ -271,11 +271,11 @@ func EscapeLabelCharacters(l string) string {
}

// GetDefaultPushLabels generates standard Prometheus push labels based on test current test instance info.
func GetDefaultPushLabels(ti *env.TestInstanceInfo) map[string]string {
func GetDefaultPushLabels(ti *env.ScenarioInstanceInfo) map[string]string {
labels := map[string]string{
MetricsLabelInstance: ti.Instance,
MetricsLabelRun: strconv.Itoa(ti.Run),
MetricsLabelTest: ti.Test,
MetricsLabelScenario: ti.Scenario,
MetricsLabelSoftwareVersion: version.SoftwareVersion,
}
if version.GitBranch != "" {
Expand Down
87 changes: 52 additions & 35 deletions go/oasis-test-runner/README.md
Original file line number Diff line number Diff line change
@@ -1,79 +1,96 @@
# Oasis Test Runner

`oasis-test-runner` initializes and executes end-to-end and remote signer test
suites.
The Oasis Test Runner initializes and executes different types of tests (e.g.
end-to-end, remote signer) also referred to as scenarios.

To list all supported tests and corresponding parameters, type:
To list all available scenarios and their corresponding parameters, use:

```bash
oasis-test-runner list
```

If no flags provided, all tests are executed. If you want to run specific test,
pass `--test` parameter:
If no flags are provided, all scenarios are executed.

To run a specific scenario, e.g `e2e/runtime/runtime-dynamic`, pass the
`--scenario` flag (or its short version, `-s`) followed by scenario's name:

```bash
oasis-test-runner --test e2e/runtime/runtime-dynamic
oasis-test-runner --scenario e2e/runtime/runtime-dynamic
```

## Benchmarking

To benchmark tests, set the `--metrics.address` parameter to the address of the
Prometheus push gateway along with `--metrics.interval`. Additionally, you
can set test-specific parameters and the number of runs each test should be run
with `--num_runs` parameter.
To benchmark scenarios, set the `--metrics.address` flag to the address of the
Prometheus push gateway along with the `--metrics.interval` flag.
Additionally, you can set scenario-specific parameters and the number of runs of
each scenario with the `--num_runs` flag.

## Benchmark analysis with `oasis-test-runner cmp` command

`cmp` command connects to Prometheus server instance containing benchmark
results generated by `oasis-test-runner` and corresponding `oasis-node`
workers. It compares the benchmark results of the last (`source`) and
pre-last (`target`) test batch (also called *instance*). You need to pass the
address of Prometheus query server using `--metrics.address` flag:
The `cmp` sub-command connects to the Prometheus server instance containing
benchmark results generated by Oasis Test Runner and the corresponding Oasis
Node workers.
It compares the benchmark results of the last (`source`) and pre-last (`target`)
scenario batch (also called *instance*).
You need to pass the address of the Prometheus query server using
`--metrics.address` flag, e.g.:

```bash
oasis-test-runner cmp \
--metrics.address http://prometheus.myorg.com:9090
```

By default `cmp` command will fetch the results of all tests and metrics
supported by `oasis-test-runner`. If you want to compare specific metric(s) and
test(s), provide `--metrics` and `--test` flags respectively:
By default, the `oasis-test-runner cmp` command will fetch the results of all
scenarios and metrics supported by the Oasis Test Runner.
If you want to compare specific metric(s) and scenario(s), set the `--metrics`
and `--scenario` flags appropriately, e.g.:

```bash
oasis-test-runner cmp \
--metrics time \
--test e2e/runtime/multiple-runtimes
--scenario e2e/runtime/multiple-runtimes
```

`cmp` takes *thresholds* for each metric into account. Currently, `avg_ratio`
and `max_ratio` are supported which correspond to the average and maximum ratio
of metric values of all test runs in the benchmark batch. For each ratio, you
can set the `max_threshold` and `min_threshold`. The former requires that the
ratio should not be exceeded and the latter that the provided threshold must be
reached. If not, `oasis-test-runner cmp` will exit with error code. This is
useful for integration into CI pipelines. Thresholds can be using the flag
`--{min|max}_threshold.<metric name>.{avg|max}_ratio`.
The `cmp` sub-command takes *thresholds* for each metric into account.
Currently, `avg_ratio` and `max_ratio` are supported which correspond to the
average and maximum ratio of metric values of all scenario runs in the benchmark
batch.

For each ratio, you can set the `max_threshold` and `min_threshold`.
The former specifies which ratio should not be exceeded and the latter which
ratio must be reached.
If a value is not within these thresholds, the `oasis-test-runner cmp` command
will exit with an error code.

This is useful for integration into CI pipelines.
Thresholds can be set using the following flag specification:

```text
--{min|max}_threshold.<metric name>.{avg|max}_ratio
```

For example:

```bash
oasis-test-runner cmp \
--metrics time \
--test e2e/runtime/multiple-runtimes \
--scenario e2e/runtime/multiple-runtimes \
--max_threshold.time.avg_ratio 1.1
```

will require that the average duration of the test in the last benchmark batch
should be at most 10\% slower than from the pre-last benchmark batch.
will require that the average duration of the scenario in the last benchmark
batch should be at most 10\% slower than from the pre-last benchmark batch.

If you are developing or improving a feature in a separate git branch, you will
want to perform benchmarks and compare the results of your branch to the ones
from the master branch. `oasis-test-runner` automatically sends information of
the git branch it was compiled on to Prometheus, so all you need to do is
perform benchmarks on both master and feature branches using the same Prometheus
instance. Then, pass the name of source and target branch names to `cmp` and it
will compare the last benchmark batch from each git branch respectively:
from the `master` branch.
The Oasis Test Runner automatically sends information of the git branch it was
compiled on to Prometheus, so all you need to do is perform benchmarks on both,
the `master` and a feature branch, using the same Prometheus instance.

Afterwards, pass the name of source and target branch names to the `cmp`
sub-command and it will compare the last benchmark batch from each git branch
(respectively):

```bash
oasis-test-runner cmp \
Expand Down
Loading

0 comments on commit e81f7d7

Please sign in to comment.