cmd: add roachprod-microbench command #91184
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
herkolategan
requested review from
a team,
srosenberg and
renatolabs
and removed request for
a team
|
This change is |
herkolategan
force-pushed
the
hbl/cmd-roachprod-bench
branch
2 times, most recently
from
e9b9e39 to
4b3137e
Compare
herkolategan
force-pushed
the
hbl/cmd-roachprod-bench
branch
from
4b3137e to
b2992bb
Compare
herkolategan
force-pushed
the
hbl/cmd-roachprod-bench
branch
16 times, most recently
from
b09da92 to
d9d1158
Compare
herkolategan
force-pushed
the
hbl/cmd-roachprod-bench
branch
from
48878d8 to
93281c8
Compare
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
herkolategan
force-pushed
the
hbl/cmd-roachprod-bench
branch
from
93281c8 to
989a48c
Compare
|
bors r=renatolabs |
|
Build succeeded: |
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this pull request
Apr 19, 2023
Add a new weekly teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. The compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: cockroachdb#93893 See also: cockroachdb#90837, cockroachdb#91184 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this pull request
May 9, 2023
Add a new weekly teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. The compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: cockroachdb#93893 See also: cockroachdb#90837, cockroachdb#91184 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this pull request
May 9, 2023
Add a new weekly teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. The compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: cockroachdb#93893 See also: cockroachdb#90837, cockroachdb#91184 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this pull request
May 9, 2023
Add a new weekly teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. The compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: cockroachdb#93893 See also: cockroachdb#90837, cockroachdb#91184 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this pull request
May 9, 2023
Add a new weekly teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. The compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: cockroachdb#93893 See also: cockroachdb#90837, cockroachdb#91184 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this pull request
May 9, 2023
Add a new weekly teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. The compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: cockroachdb#93893 See also: cockroachdb#90837, cockroachdb#91184 Epic: CRDB-20903 Release note: None
craig bot
pushed a commit
that referenced
this pull request
May 11, 2023
96723: build: add microbenchmark script for roachprod-microbench r=srosenberg a=herkolategan Add a new nightlies teamcity build script that utilises the `dev test-binaries` and the `roachprod-microbench` command. The script can be used to store a microbenchmarks checkpoint for a given `git` revision to a bucket in gcs, by specifying a publish path. A `roachprod` cluster is created by the script, as specified by environment variables. The clean-up is managed by TeamCity. Alternatively two binaries can be compared by supplying the related comparison binaries environment variables. This compare directory should be omitted in this case. The script requires a gcs path to download the comparison binaries from. The running time will be considerably longer, but there will be less variance in the results. Subsequent runs can then be made with the same script to compare against a previous checkpoint, by supplying a compare path. The script takes numerous environment variables to configure the microbenchmarks cluster and arguments. These can be tweaked to configure different run variations and packages to target. Resolves: #93893 See also: #90837, #91184 Epic: CRDB-20903 Release note: None Co-authored-by: Herko Lategan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cmd: add roachprod-microbench command
Add command roachprod-microbench to the cmd package. This command depends on the
portable test binaries outputted from the dev test-binaries command.
In order to run multiple microbenchmark iterations with an acceptable overall
duration the roachprod-microbench command distributes microbenchmark executions
at a per microbenchmark function level to a roachprod cluster.
The command also handles the orchestration of copying and extracting the
portable binaries to the cluster. Failures and results are logged for each
microbenchmark. On completion a report is generated for further analysis.
Results from microbenchmarks are also sanitized to remove any noise in the log
output.
Additionally, comparisons to previous runs can be made. Google cloud storage can
be used to store the results of previous runs. The internal Go tooling is used
to do the comparison and output a Google Sheet to Google Drive for further
inspection.
Resolves: #90958
See also: #90837
Epic: CRDB-20903
Release note: None