forked from cockroachdb/cockroach
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
cmd: add roachprod-microbench command
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
- Loading branch information
1 parent
ac65603
commit 9571e98
Showing
17 changed files
with
1,891 additions
and
35 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| load("//build/bazelutil/unused_checker:unused.bzl", "get_x_data") | ||
| load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test") | ||
|
|
||
| go_library( | ||
| name = "roachprod-microbench_lib", | ||
| srcs = [ | ||
| "benchmark.go", | ||
| "compare.go", | ||
| "main.go", | ||
| "metadata.go", | ||
| "report.go", | ||
| "storage.go", | ||
| ], | ||
| importpath = "github.com/cockroachdb/cockroach/pkg/cmd/roachprod-microbench", | ||
| visibility = ["//visibility:private"], | ||
| deps = [ | ||
| "//pkg/cmd/roachprod-microbench/cluster", | ||
| "//pkg/cmd/roachprod-microbench/google", | ||
| "//pkg/roachprod", | ||
| "//pkg/roachprod/logger", | ||
| "//pkg/roachprod/ssh", | ||
| "//pkg/roachprod/vm", | ||
| "//pkg/util/syncutil", | ||
| "//pkg/util/timeutil", | ||
| "@com_github_cockroachdb_errors//:errors", | ||
| "@com_github_klauspost_compress//gzip", | ||
| "@com_google_cloud_go_storage//:storage", | ||
| "@org_golang_x_perf//benchstat", | ||
| "@org_golang_x_perf//storage/benchfmt", | ||
| ], | ||
| ) | ||
|
|
||
| go_binary( | ||
| name = "roachprod-microbench", | ||
| embed = [":roachprod-microbench_lib"], | ||
| visibility = ["//visibility:public"], | ||
| ) | ||
|
|
||
| go_test( | ||
| name = "roachprod-microbench_test", | ||
| srcs = ["benchmark_test.go"], | ||
| args = ["-test.timeout=295s"], | ||
| data = glob(["testdata/**"]), | ||
| embed = [":roachprod-microbench_lib"], | ||
| deps = [ | ||
| "//pkg/testutils/datapathutils", | ||
| "@com_github_cockroachdb_datadriven//:datadriven", | ||
| ], | ||
| ) | ||
|
|
||
| get_x_data(name = "get_x_data") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| ## Name | ||
| **roachprod-microbench** - Execute microbenchmarks across a `roachprod` cluster | ||
|
|
||
| ## Synopsis | ||
| **roachprod-microbench** `<working dir> [<flags>] -- [<test args>]` | ||
|
|
||
| ## Examples | ||
| > **roachprod-microbench** ./artifacts/log -binaries=bin.tar.gz -cluster=user-bench -- -test.benchtime=1ns | ||
| Execute microbenchmarks present in the `bin.tar.gz` archive across the `user-bench` roachprod cluster. | ||
|
|
||
| Pass the additional test argument `-test.benchtime=1ns` to all executions. | ||
|
|
||
| Microbenchmark results are written to the `./artifacts/log` directory. This directory will also contain logs for any failures that occurred. | ||
|
|
||
| > **roachprod-microbench** ./artifacts/log -binaries=bin.tar.gz -cluster=user-bench -publishdir=gs://microbench/2042 | ||
| Publish the logs captured from the microbenchmarks in the `<working dir>` `./artifacts/log` to a GCS bucket `gs://microbench/2042`. | ||
|
|
||
| > **roachprod-microbench** ./artifacts/log -binaries=bin.tar.gz -cluster=user-bench -comparedir=gs://microbench/1899 | ||
| Compare the logs captured from the microbenchmarks in the `<working dir>` `./artifacts/log` to a GCS bucket `gs://microbench/1899` of a previously published **roachprod-microbench** run. | ||
|
|
||
| ## Description | ||
|
|
||
| The **roachprod-microbench** command operates on a portable test binaries archive that has been built with the `dev test-binaries` tooling. | ||
| Compressed and uncompressed `tar` archives are supported, since the `dev` tooling can produce both. | ||
| It should be noted that the archive has to adhere to a specific layout and contains `run.sh` scripts for each package, which facilitates execution. | ||
| This tool only supports binaries archives produced in this format by the `dev test-binaries` tooling. | ||
|
|
||
| As part of the orchestration, test binaries are copied to the target cluster and extracted. | ||
| Once the cluster is ready, the tool will first probe for all microbenchmarks available and then execute the microbenchmarks according to any flags that have been set. | ||
|
|
||
| The `<working dir>` must always be specified and all results from the microbenchmarks will be captured to this directory, including any errors. | ||
| If the `publishdir` or `comparedir` flags are specified these will also use the `<working dir>` to publish from or compare against. | ||
|
|
||
| The publish functionality is used to store the results of a run to a specified location. | ||
| On a subsequent run the compare functionality can then be pointed to that same location to do a comparison which will publish Google sheets for further inspection. | ||
|
|
||
| For additional information on flags, run the **roachprod-microbench** command without any arguments to display a helpful message. |
Oops, something went wrong.