-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
release: 22.1 microbenchmark regression investigation and sign-off #78592
Comments
Missing Benchmark ResultsResults of many (as in hundreds) benches both in
|
Thanks for running these benchmarks! The logging can be suppressed by adding this to the start of the benchmarks:
This still outputs a couple of lines, but they should be easy enough to grep out:
|
The regressions in Append/bytes and Copy/bytes were known consequences of #76202 which we traded for faster unordered distinct and top-k operations over bytes. (In absolute numbers the speeds of append and copy are still in GB/sec.) |
The optimizer alloc/op regressions like
|
Thanks @erikgrinaker, this works! The extra stdout is actually appended after the benchmark results (
Note, My other dilemma is what to do with all the missing benchmarks because of the unsuppressed logging? Should we instrument them with [1] https://github.com/cockroachdb/cockroach/blob/master/pkg/util/log/test_log_scope.go#L161 |
I see. Seems reasonable to change it to pass
Yeah, I think in general all integration tests and benchmarks should use |
It used to be the case that you could run benchmarks with |
I fixed up the crash on |
@petermattis That no longer works. See the discussion in #57979. |
78704: sql: propagate limit for top K sort correctly in tests r=yuzefovich a=yuzefovich In 22.1 time frame we started propagating the value of K for top K sort in the spec of the processor, and not in the post-processing spec, but we forgot to update some of the tests accordingly. Informs: #78592. Release note: None 78949: kvserver: gossip l0sublevels instead of read amp r=kvoli a=kvoli Previously read amplification was gossipped among stores to enable future allocation decisions that would avoid candidates with high read amplification. L0 Sublevels represents the number of levels with L0 and is a portion of read amplification. This patch change read amplification to l0 sublevels, as it is a better indicator of store health. Release justification: low risk, replace deprecated gossip signal. Release note: None 78984: sql: deflake unique logic test r=mgartner a=mgartner PR #78685 changes the query plan of one query in a logic test in a way that makes the test flakey. This commit guarantees that the test cannot be flakey, regardless of the query plan. Release note: None Co-authored-by: Yahor Yuzefovich <[email protected]> Co-authored-by: Austen McClernon <[email protected]> Co-authored-by: Marcus Gartner <[email protected]>
78703: colexecop: reset the internal batch in a test operator r=yuzefovich a=yuzefovich This commit makes a test operator that "owns" a batch to properly reset the batch on each `Next` call. This is the expectation that we forgot to follow. This was discovered when running `BenchmarkLikeOps` which uses a Bytes vector. Due to the way `Bytes.Copy` is implemented, the Bytes vector's buffer could grow arbitrarily large because the test operator never reset the vector. Informs: #78592. Release note: None Co-authored-by: Yahor Yuzefovich <[email protected]>
That stinks. I only skimmed #57979, but couldn't catch the reason that a flag to globally disable logging is infeasible. Such a flag seems preferable to sprinkling |
Have you tried this should disable file creation |
to disable entirely, maybe something like |
to remove the stderr logging, |
Can there be a short-hand added for that |
i suppose there can but what is the desired behavior? |
Disabling all logging to stderr. When running microbenchmarks such logging isn't needed. Or perhaps disable all logging below FATAL. |
okay but the convo above mentioned/started with file/directory creation. should this be disabledd too? |
Yes. Good to avoid any disk IO or other activity which can skew the benchmarks. |
I rerun |
It doesn't work because
Thus, by including the above
If we really want to enable logging (original case),
If the above sounds reasonable, I'll send a PR to add the missing [1] https://github.com/cockroachdb/cockroach/blob/master/pkg/util/log/flags.go#L56-L64 |
I reran |
This is a tracking bug for issues we want to investigate related to performance for 21.2 similar to what's been done for prior releases (#71820).
SQL
pkg/sql/...
spread sheets
pkg/bench
spread sheets
NOTE:
BenchmarkSQL//InsertFK
has been skipped since it occasionally gets stuck on the old sha.pkg/col...
spread sheets
KV
./pkg/gossip/...
./pkg/roachpb
./pkg/rpc/...
./pkg/server/...
spread sheets
Bulk I/O
spread sheets
NOTE: results are empty because the benches don't suppress
stdout
causing results to be on subsequent linesStorage
pkg/ccl/storageccl/...
spread sheets
Misc
./pkg/base ./pkg/ccl/utilccl/... ./pkg/ccl/workloadccl/... ./pkg/cli/... ./pkg/clusterversion ./pkg/cmd/... ./pkg/internal/... ./pkg/security ./pkg/settings ./pkg/testutils/... ./pkg/ts/... ./pkg/util/... ./pkg/workload/...
spread sheets
NOTE: pkg/util had to be skipped because it failed to build in the old version,
Issues:
Jira issue: CRDB-14191
The text was updated successfully, but these errors were encountered: