-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
116298: roachtest: improve sqlsmith for an elusive setup error r=mgartner a=yuzefovich **roachtest: log erroneous stmt in sqlsmith.log** Previously, if we hit an error when executing a setup query, we would only include it into the error message (so it would only be seen in the test.log file). This commit makes it so that the erroneous setup query is included into sqlsmith.log as well. Informs: #116160. Informs: #116307. **roachtest/sqlsmith: print a setup query as integers** This commit adjusts the sqlsmith logging so that if it encounters an error during the setup that contains "does not exist" substring, then we will also log the failed stmt as space-separated integers. This change is made in hopes of being able to reproduce the elusive 'pq: column "crdb_internal_idx_expr" does not exist' error we've seen a few times. My hypothesis is that there is some non-visible character that gets lost when stringified. Epic: None Release note: None 116980: pkg/bench: benchmark latency impact of gc when hashing r=dt a=dt Release note: none. Epic: none. 116987: tenantcostclient: add client consumption metrics r=JeffSwenson a=jaylim-crl Previously, consumption metrics were only available in KV servers. These metrics are updated whenever SQL servers send a TokenBucketRequest with the consumption since the last request to KV servers. However, those metrics were insufficient for CockroachDB Cloud's use case. Specifically, for metrics export to work, we need to fetch these data separately from the KV servers, which introduces its own issues. To address that, this commit introduces consumption metrics within the SQL servers. This change allows us to rely solely on metrics exported by the SQL servers, eliminating the need to build something externally to fetch data from the KV servers. It is worth nothing that the aggregated values for these metrics will always be greater than or equal to the values in the KV servers since metrics are updated locally before requests to KV servers. Additionally, this commit also fixes a bug where the existing tenantcostclient metrics, introduced in #113512, were not added to the main registry. As a result, they were unavailable via the status server. Epic: [CC-26682](https://cockroachlabs.atlassian.net/browse/CC-26682) Release note: None 117118: sql/rowcontainer: remove propagation of synthetic timestamp bit r=nvanbenschoten a=nvanbenschoten Informs #101938. This PR removes the propagation of the synthetic flag through `kvStreamerResultDiskBuffer`. It then cleans up the use of synthetic timestamps in a few SQL tests. This flag has been deprecated since v22.2 and is no longer consulted in uncertainty interval checks or by transaction commit-wait. It does not need to be propagated. Release note: None Co-authored-by: Yahor Yuzefovich <[email protected]> Co-authored-by: David Taylor <[email protected]> Co-authored-by: Jay <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]>
- Loading branch information
Showing
19 changed files
with
703 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| load("@io_bazel_rules_go//go:def.bzl", "go_test") | ||
|
|
||
| go_test( | ||
| name = "hashbench_test", | ||
| srcs = ["hash_gc_test.go"], | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| // Copyright 2023 The Cockroach Authors. | ||
| // | ||
| // Use of this software is governed by the Business Source License | ||
| // included in the file licenses/BSL.txt. | ||
| // | ||
| // As of the Change Date specified in that file, in accordance with | ||
| // the Business Source License, use of this software will be governed | ||
| // by the Apache License, Version 2.0, included in the file | ||
| // licenses/APL.txt. | ||
|
|
||
| package hashbench | ||
|
|
||
| import ( | ||
| "crypto/md5" | ||
| "crypto/sha256" | ||
| "fmt" | ||
| "hash" | ||
| "runtime" | ||
| "testing" | ||
| "time" | ||
| ) | ||
|
|
||
| // chunkingHasher wraps a hash but chunks up writes passed to it. | ||
| type chunkingHash struct { | ||
| hash.Hash | ||
| chunkSize int64 | ||
| } | ||
|
|
||
| var _ hash.Hash = chunkingHash{} | ||
|
|
||
| func (c chunkingHash) Write(b []byte) (n int, _ error) { | ||
| for int64(len(b)) > c.chunkSize { | ||
| w, _ := c.Hash.Write(b[:c.chunkSize]) | ||
| b = b[w:] | ||
| n += w | ||
| } | ||
| w, _ := c.Hash.Write(b) | ||
| return n + w, nil | ||
| } | ||
|
|
||
| // BenchmarkLatencyWhileHashing demonstrates the latency impact (in this case on | ||
| // a loop that just sleeps) that hashing various sizes of blocks of bytes has in | ||
| // the presence of GC, due to https://github.com/golang/go/issues/64417. | ||
| func BenchmarkLatencyWhileHashing(b *testing.B) { | ||
| for _, hashImpl := range []struct { | ||
| name string | ||
| hash.Hash | ||
| }{{name: "MD5", Hash: md5.New()}, {name: "SHA256", Hash: sha256.New()}} { | ||
| for _, blockSizeKB := range []int64{0, 64, 128, 512, 1024, 2048, 4096} { | ||
| for _, fileSizeMB := range []int64{4, 8, 16, 32} { | ||
| blockSizeStr := fmt.Sprintf("%dKB", blockSizeKB) | ||
| if blockSizeKB == 0 { | ||
| blockSizeStr = "whole" | ||
| } | ||
| b.Run(fmt.Sprintf("hash=%s/block=%s/file=%dMB", hashImpl.name, blockSizeStr, fileSizeMB), func(b *testing.B) { | ||
| h, fileSize := hashImpl.Hash, fileSizeMB<<20 | ||
| if blockSizeKB > 0 { | ||
| h = chunkingHash{Hash: h, chunkSize: blockSizeKB << 10} | ||
| } | ||
|
|
||
| stop, stoppedGC, stoppedHashing := make(chan struct{}), make(chan struct{}), make(chan struct{}) | ||
|
|
||
| // Hammer GC, in a loop, for the duration of the test. These GCs may | ||
| // stop the world, which would contribute to the latency measured in | ||
| // the benchmark loop below. | ||
| go func() { | ||
| defer close(stoppedGC) | ||
| for { | ||
| select { | ||
| case <-stop: | ||
| return | ||
| default: | ||
| runtime.GC() | ||
| } | ||
| } | ||
| }() | ||
|
|
||
| // Do some hashing work for the duration of the test. These hashing | ||
| // calls will need to be stopped along with the rest of the world by | ||
| // the GC's being being run in the loop above; it they are slow to | ||
| // stop, they'd delay the GC stops and thus increase the observed | ||
| // latency in the benchmark loop. | ||
| go func() { | ||
| defer close(stoppedHashing) | ||
| for { | ||
| select { | ||
| case <-stop: | ||
| return | ||
| default: | ||
| if h != nil { | ||
| buf := make([]byte, fileSize) | ||
| for i := 0; i < 10; i++ { | ||
| h.Write(buf) | ||
| h.Sum(nil) | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }() | ||
|
|
||
| b.ResetTimer() | ||
|
|
||
| // Measure worst-case latency one loop iteration to the next, with | ||
| // each iteration sleeping a fixed interval, while the GC and hashing | ||
| // loops above are also running. | ||
| var worst time.Duration | ||
| before := time.Now() | ||
| for i := 0; i < b.N; i++ { | ||
| time.Sleep(time.Microsecond * 100) | ||
| if d := time.Since(before); d > worst { | ||
| worst = d | ||
| } | ||
| before = time.Now() | ||
| } | ||
|
|
||
| b.StopTimer() | ||
| b.ReportMetric(float64(worst.Microseconds()), "max-latency") | ||
|
|
||
| // Stop the the two background loops and wait for them to exit before | ||
| // moving on to the next test-case. | ||
| close(stop) | ||
| <-stoppedGC | ||
| <-stoppedHashing | ||
| }) | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.