cmd: add roachprod-microbench #90958
Assignees
Labels
A-testing
Testing tools and infrastructure
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
C-performance
Perf of queries or internals. Solution not expected to change functional behavior.
T-testeng
TestEng Team
Comments
herkolategan
added
C-enhancement
|
cc @cockroachdb/test-eng |
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Nov 7, 2022
Add command roachprod-bench to dev. The command will build all the test binaries for a given set of packages, using Bazel. Any runfiles required by the binaries will also be staged. Binaries and runfiles are added to a tar in artifacts and can be optionally compressed. The output is portable without being dependent on the repository. Each package is supplied with a run script that will setup environment variables for Bazel runfiles to be used by the test binary. Additionally, this builds the binary for a sub command roachprod-bench that will orchestrate the execution of the binaries. The sub command is also executed by the roachprod-bench command with specified arguments passed to it. Resolves: cockroachdb#90837 See also: cockroachdb#90958 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Nov 13, 2022
Add command roachprod-bench to dev. The command will build all the test binaries for a given set of packages, using Bazel. Any runfiles required by the binaries will also be staged. Binaries and runfiles are added to a tar in artifacts and can be optionally compressed. The output is portable without being dependent on the repository. Each package is supplied with a run script that will setup environment variables for Bazel runfiles to be used by the test binary. Additionally, this builds the binary for a sub command roachprod-bench that will orchestrate the execution of the binaries. The sub command is also executed by the roachprod-bench command with specified arguments passed to it. Resolves: cockroachdb#90837 See also: cockroachdb#90958 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Nov 13, 2022
Add command roachprod-bench to the cmd package. This command depends on the portable test binaries output from the dev roachprod-bench command. In order to run multiple benchmark iterations with an acceptable overall duration the roachprod-bench command distributes benchmark executions at a per benchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each benchmark. On completion a report is generated for further analysis. Results from benchmarks are also sanitized to remove any noise in the log output. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Nov 16, 2022
Add command roachprod-bench-wrapper to dev. The command will build all the test binaries for a given set of packages, using Bazel. Any runfiles required by the binaries will also be staged. Binaries and runfiles are added to a tar in artifacts and can be optionally compressed. The output is portable without being dependent on the repository. Each package is supplied with a run script that will setup environment variables for Bazel runfiles to be used by the test binary. Additionally, this builds the binary for a sub command roachprod-bench that will orchestrate the execution of the binaries. The sub command is also executed by the roachprod-bench command with specified arguments passed to it. Resolves: cockroachdb#90837 See also: cockroachdb#90958 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Nov 17, 2022
Add command roachprod-bench-wrapper to dev. The command will build all the test binaries for a given set of packages, using Bazel. Any runfiles required by the binaries will also be staged. Binaries and runfiles are added to a tar in artifacts and can be optionally compressed. The output is portable without being dependent on the repository. Each package is supplied with a run script that will setup environment variables for Bazel runfiles to be used by the test binary. Additionally, this builds the binary for a sub command roachprod-bench that will orchestrate the execution of the binaries. The sub command is also executed by the roachprod-bench command with specified arguments passed to it. Resolves: cockroachdb#90837 See also: cockroachdb#90958 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Nov 17, 2022
Add command roachprod-bench-wrapper to dev. The command will build all the test binaries for a given set of packages, using Bazel. Any runfiles required by the binaries will also be staged. Binaries and runfiles are added to a tar in artifacts and can be optionally compressed. The output is portable without being dependent on the repository. Each package is supplied with a run script that will setup environment variables for Bazel runfiles to be used by the test binary. Additionally, this builds the binary for a sub command roachprod-bench that will orchestrate the execution of the binaries. The sub command is also executed by the roachprod-bench command with specified arguments passed to it. Resolves: cockroachdb#90837 See also: cockroachdb#90958 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Jan 17, 2023
This intends to decouple generating test binaries and running the roachprod microbenchmarks command. Rather than having a strong coupling this command will now just build and archive the portable test binaries that can then be used by any other commands in the future. The command also no longer builds the libraries or the roachprod microbenchmarks binary (formerly roachprod-bench). After this change `dev test-binaries` will only generate an archive of portable test binaries. This archive can then be used with the roachprod microbenchmarks command, possibly amongst other things, to run microbenchmarks. See also: cockroachdb#90958 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Jan 17, 2023
This intends to decouple generating test binaries and running the roachprod microbenchmarks command. Rather than having a strong coupling this command will now just build and archive the portable test binaries that can then be used by any other commands in the future. The command also no longer builds the libraries or the roachprod microbenchmarks binary (formerly roachprod-bench). After this change `dev test-binaries` will only generate an archive of portable test binaries. This archive can then be used with the roachprod microbenchmarks command, possibly amongst other things, to run microbenchmarks. See also: cockroachdb#90958 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 2, 2023
This intends to decouple generating test binaries and running the roachprod microbenchmarks command. Rather than having a strong coupling this command will now just build and archive the portable test binaries that can then be used by any other commands in the future. The command also no longer builds the libraries or the roachprod microbenchmarks binary (formerly roachprod-bench). After this change `dev test-binaries` will only generate an archive of portable test binaries. This archive can then be used with the roachprod microbenchmarks command, possibly amongst other things, to run microbenchmarks. See also: cockroachdb#90958 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 2, 2023
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 6, 2023
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 7, 2023
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 7, 2023
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 7, 2023
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
herkolategan
added a commit
to herkolategan/cockroach
that referenced
this issue
Feb 7, 2023
Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: cockroachdb#90958 See also: cockroachdb#90837 Epic: CRDB-20903 Release note: None
craig bot
pushed a commit
that referenced
this issue
Feb 10, 2023
91184: cmd: add roachprod-microbench command r=renatolabs a=herkolategan cmd: add roachprod-microbench command Add command roachprod-microbench to the cmd package. This command depends on the portable test binaries outputted from the dev test-binaries command. In order to run multiple microbenchmark iterations with an acceptable overall duration the roachprod-microbench command distributes microbenchmark executions at a per microbenchmark function level to a roachprod cluster. The command also handles the orchestration of copying and extracting the portable binaries to the cluster. Failures and results are logged for each microbenchmark. On completion a report is generated for further analysis. Results from microbenchmarks are also sanitized to remove any noise in the log output. Additionally, comparisons to previous runs can be made. Google cloud storage can be used to store the results of previous runs. The internal Go tooling is used to do the comparison and output a Google Sheet to Google Drive for further inspection. Resolves: #90958 See also: #90837 Epic: CRDB-20903 Release note: None 96884: docs: Add full URL issue refs to docs product change issue bodies r=nickvigilante a=nickvigilante To help ease some of the triaging burden of docs product change issues, this PR introduces epic and issue references directly in the docs issue body. Fixes DOC-6645 Release note: None 96937: roachtest: decommission disable roachprod init on restarts r=msbutler a=aliher1911 Roachtest Cluster.Start() will always try to init cluster and do one time ops if not explicitly disabled. This is not a good fit for tests that introduce failures as post start operations could fail and thus break test. This commit adds default options that prevent init and changes decommission test to use them when restarting nodes. Release note: None Fixes #96791 96946: kvserver: fix locking before addToReplicasByKey* r=kvoli a=pavelkalinnikov This commit fixes test failures under `--race` which see `r.mu` not locked in methods that require it. The mutex lock wasn't necessary during `struct` creation, but now is done to work around the test failures. Release note: none Epic: none Co-authored-by: Herko Lategan <[email protected]> Co-authored-by: Nick Vigilante <[email protected]> Co-authored-by: Oleg Afanasyev <[email protected]> Co-authored-by: Pavel Kalinnikov <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Mar 7, 2023
95353: bazel: refactor dev roachprod-bench-wrapper command to dev test-binaries r=srosenberg a=herkolategan This intends to decouple generating test binaries and running the roachprod microbenchmarks command. Rather than having a strong coupling this command will now just build and archive the portable test binaries that can then be used by any other commands in the future. The command also no longer builds the libraries or the roachprod microbenchmarks binary (formerly roachprod-bench). After this change `dev test-binaries` will only generate an archive of portable test binaries. This archive can then be used with the roachprod microbenchmarks command, possibly amongst other things, to run microbenchmarks. See also: #90958 Release note: None 98169: copy: add test with 1PC optimization disabled r=otan a=rafiss This makes sure everything in the COPY is still committed even if the insert_fast_path is disabled. Epic: None Release note: None Co-authored-by: Herko Lategan <[email protected]> Co-authored-by: Rafi Shamim <[email protected]>
herkolategan
added a commit
that referenced
this issue
Apr 4, 2023
Add command roachprod-bench-wrapper to dev. The command will build all the test binaries for a given set of packages, using Bazel. Any runfiles required by the binaries will also be staged. Binaries and runfiles are added to a tar in artifacts and can be optionally compressed. The output is portable without being dependent on the repository. Each package is supplied with a run script that will set up environment variables for Bazel runfiles to be used by the test binary. Additionally, this builds the binary for a sub command roachprod-bench that will orchestrate the execution of the binaries. The sub command is also executed by the roachprod-bench command with specified arguments passed to it. Resolves: #90837 See also: #90958 Epic: CRDB-20903 Release justification: test-only change. Release note: None
herkolategan
added a commit
that referenced
this issue
Apr 4, 2023
This intends to decouple generating test binaries and running the roachprod microbenchmarks command. Rather than having a strong coupling this command will now just build and archive the portable test binaries that can then be used by any other commands in the future. The command also no longer builds the libraries or the roachprod microbenchmarks binary (formerly roachprod-bench). After this change `dev test-binaries` will only generate an archive of portable test binaries. This archive can then be used with the roachprod microbenchmarks command, possibly amongst other things, to run microbenchmarks. See also: #90958 Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A new command
roachprod-microbenchshould be added to thecmdpackage, to orchestrate execution of the benchmarks present in the portable test binaries, which are produced by the dev command [1], across aroachprodcluster.The
roachprod-benchtool will handle orchestration involving copying the binaries and dependencies to aroachprodcluster. After which it will distribute a benchmark execution plan across the cluster. Finally results are collected and aggregated. Additional processing like output sanitization, validation and analysis can be applied.[1] #90837
Jira issue: CRDB-21042
Epic CRDB-20903
The text was updated successfully, but these errors were encountered: