Merge #85392 #86063 #86558 #86596 #86684

85392: streamingccl: support sst event in random stream client r=gh-casper a=gh-casper

Previously random stream client lacks test coverage
for AddSSTable operation, this PR enables random stream
client to generate and validate random SSTableEvents.

This PR also cleans up the InterceptableStreamClient
to avoid unnecessary abstraction, making code cleaner.

Release justification: low risk, high benefit changes to 
existing functionality

Release note : None

86063: base,kvserver,server: configuration of provisioned bandwidth for a store r=irfansharif a=sumeerbhola

A previous PR #85722 added support for disk bandwidth as a bottlneck
resource, in the admission control package. To utilize this, admission
control needs to be provided the provisioned bandwidth and the observed
read and write bytes. This PR adds configuration support for this via
the StoreSpec (that uses the --store flag). The StoreSpec now has an
optional ProvisionedRateSpec that contains the name of the disk
corresponding to the store, and an optional provisioned bandwidth,
that are specified as
provisioned-rate=name=<disk-name>[:bandwidth=<bandwidth-bytes>].

The disk-name is used to map the DiskStats, retrieved via the existing
code in status.GetDiskCounters to the correct Store. These DiskStats
contain the read and write bytes. The optional bandwidth is used to
override the provisioned bandwidth set via the new cluster setting
kv.store.admission.provisioned_bandwidth.

Fixes #82898

Release note (ops change): Disk bandwidth constraint can now be
used to control admission of elastic writes. This requires configuration
for each store, via the --store flag, that now contains an optional
provisioned-rate field. The provisioned-rate field, if specified,
needs to provide a disk-name for the store and optionally a disk
bandwidth. If the disk bandwidth is not provided the cluster setting
kv.store.admission.provisioned_bandwidth will be used. The cluster
setting defaults to 0 (which means that the disk bandwidth constraint
is disabled). If the effective disk bandwidth, i.e., using the possibly
overridden cluster setting is 0, there is no disk bandwidth constraint.
Additionally, the admission control cluster setting
admission.disk_bandwidth_tokens.elastic.enabled (defaults to true)
can be used to turn off enforcement even when all the other
configuration has been setup. Turning off enforcement will still
output all the relevant information about disk bandwidth usage, so
can be used to observe part of the mechanism in action.
To summarize, to enable this for a cluster with homogenous disk,
provide a disk-name in the provisioned-rate field in the store-spec,
and set the kv.store.admission.provisioned_bandwidth cluster setting
to the bandwidth limit. To only get information about disk bandwidth
usage by elastic traffic (currently via logs, not metrics), do the
above but also set admission.disk_bandwidth_tokens.elastic.enabled
to false.

Release justification: Low risk, high benefit change that allows
an operator to enable new functionality (disabled by default).


86558: spanconfigtestutils: copy gc job testing knobs to tenant r=ajwerner a=stevendanna

Some of the sqltranslator tests require that we can disable the GC job
related to an index creation so that we can observe the span
configuration on the temporary indexes.  We pause the GC job via a
testing knob.

Fixes #85507

Release justification: Test-only change

Release note: None

86596: ui/cluster-ui: create statements insights api r=xinhaoz a=xinhaoz

This commit adds a function to the `cluster-ui` pkg that queries
`crdb_internal.cluster_execution_insights` to surface slow running
queries. The information retrieved  is intended to be used in the
insights statement overview and details pages. A new field
`statementInsights` is added to the `cachedData` object in the
db-console redux store, and the corresponding function to issue the
data fetch is also added in `apiReducers`. This commit does not add
any reducers  or sagas to CC to fetch and store this data.

Release justification: non-production code change
Release note: None

86684: release: update cockroachdb version to v22.1.6 r=rhu713 a=rail

Release justification: not a part of the build
Release note: None

Co-authored-by: Casper <[email protected]>
Co-authored-by: sumeerbhola <[email protected]>
Co-authored-by: Steven Danna <[email protected]>
Co-authored-by: Xin Hao Zhang <[email protected]>
Co-authored-by: Rail Aliiev <[email protected]>

cloud/kubernetes/bring-your-own-certs/client.yaml

@@ -20,7 +20,7 @@ spec:
   serviceAccountName: cockroachdb
   containers:
   - name: cockroachdb-client
-    image: cockroachdb/cockroach:v22.1.5
+    image: cockroachdb/cockroach:v22.1.6
     # Keep a pod open indefinitely so kubectl exec can be used to get a shell to it
     # and run cockroach client commands, such as cockroach sql, cockroach node status, etc.
     command:

cloud/kubernetes/bring-your-own-certs/cockroachdb-statefulset.yaml

@@ -153,7 +153,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: Change these to appropriate values for the hardware that you're running. You can see
         # the resources that can be allocated on each of your Kubernetes nodes by running:

cloud/kubernetes/client-secure.yaml

@@ -32,7 +32,7 @@ spec:
       mountPath: /cockroach-certs
   containers:
   - name: cockroachdb-client
-    image: cockroachdb/cockroach:v22.1.5
+    image: cockroachdb/cockroach:v22.1.6
     imagePullPolicy: IfNotPresent
     volumeMounts:
     - name: client-certs

cloud/kubernetes/cluster-init-secure.yaml

@@ -34,7 +34,7 @@ spec:
           mountPath: /cockroach-certs
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         volumeMounts:
         - name: client-certs

cloud/kubernetes/cluster-init.yaml

@@ -10,7 +10,7 @@ spec:
     spec:
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         command:
           - "/cockroach/cockroach"

cloud/kubernetes/cockroachdb-statefulset-secure.yaml

@@ -195,7 +195,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: Change these to appropriate values for the hardware that you're running. You can see
         # the resources that can be allocated on each of your Kubernetes nodes by running:

cloud/kubernetes/cockroachdb-statefulset.yaml

@@ -98,7 +98,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: Change these to appropriate values for the hardware that you're running. You can see
         # the resources that can be allocated on each of your Kubernetes nodes by running:

cloud/kubernetes/multiregion/client-secure.yaml

@@ -9,7 +9,7 @@ spec:
   serviceAccountName: cockroachdb
   containers:
   - name: cockroachdb-client
-    image: cockroachdb/cockroach:v22.1.5
+    image: cockroachdb/cockroach:v22.1.6
     imagePullPolicy: IfNotPresent
     volumeMounts:
     - name: client-certs

cloud/kubernetes/multiregion/cluster-init-secure.yaml

@@ -11,7 +11,7 @@ spec:
       serviceAccountName: cockroachdb
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         volumeMounts:
         - name: client-certs

cloud/kubernetes/multiregion/cockroachdb-statefulset-secure.yaml

@@ -167,7 +167,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 26257

cloud/kubernetes/multiregion/eks/cockroachdb-statefulset-secure-eks.yaml

@@ -185,7 +185,7 @@ spec:
           name: cockroach-env
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: Change these to appropriate values for the hardware that you're running. You can see
         # the resources that can be allocated on each of your Kubernetes nodes by running:

cloud/kubernetes/performance/cockroachdb-daemonset-insecure.yaml

@@ -82,7 +82,7 @@ spec:
       hostNetwork: true
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: If you configured taints to give CockroachDB exclusive access to nodes, feel free
         # to remove the requests and limits sections. If you didn't, you'll need to change these to

cloud/kubernetes/performance/cockroachdb-daemonset-secure.yaml

@@ -198,7 +198,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: If you configured taints to give CockroachDB exclusive access to nodes, feel free
         # to remove the requests and limits sections. If you didn't, you'll need to change these to

cloud/kubernetes/performance/cockroachdb-statefulset-insecure.yaml

@@ -141,7 +141,7 @@ spec:
       - name: cockroachdb
         # NOTE: Always use the most recent version of CockroachDB for the best
         # performance and reliability.
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: Change these to appropriate values for the hardware that you're running. You can see
         # the resources that can be allocated on each of your Kubernetes nodes by running:

cloud/kubernetes/performance/cockroachdb-statefulset-secure.yaml

@@ -232,7 +232,7 @@ spec:
       - name: cockroachdb
         # NOTE: Always use the most recent version of CockroachDB for the best
         # performance and reliability.
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         # TODO: Change these to appropriate values for the hardware that you're running. You can see
         # the resources that can be allocated on each of your Kubernetes nodes by running:

cloud/kubernetes/v1.6/client-secure.yaml

@@ -32,7 +32,7 @@ spec:
       mountPath: /cockroach-certs
   containers:
   - name: cockroachdb-client
-    image: cockroachdb/cockroach:v22.1.5
+    image: cockroachdb/cockroach:v22.1.6
     imagePullPolicy: IfNotPresent
     volumeMounts:
     - name: client-certs

cloud/kubernetes/v1.6/cluster-init-secure.yaml

@@ -34,7 +34,7 @@ spec:
           mountPath: /cockroach-certs
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         volumeMounts:
         - name: client-certs

cloud/kubernetes/v1.6/cluster-init.yaml

@@ -10,7 +10,7 @@ spec:
     spec:
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         command:
           - "/cockroach/cockroach"

cloud/kubernetes/v1.6/cockroachdb-statefulset-secure.yaml

@@ -178,7 +178,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 26257

cloud/kubernetes/v1.6/cockroachdb-statefulset.yaml

@@ -81,7 +81,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 26257

cloud/kubernetes/v1.7/client-secure.yaml

@@ -32,7 +32,7 @@ spec:
       mountPath: /cockroach-certs
   containers:
   - name: cockroachdb-client
-    image: cockroachdb/cockroach:v22.1.5
+    image: cockroachdb/cockroach:v22.1.6
     imagePullPolicy: IfNotPresent
     volumeMounts:
     - name: client-certs

cloud/kubernetes/v1.7/cluster-init-secure.yaml

@@ -34,7 +34,7 @@ spec:
           mountPath: /cockroach-certs
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         volumeMounts:
         - name: client-certs

cloud/kubernetes/v1.7/cluster-init.yaml

@@ -10,7 +10,7 @@ spec:
     spec:
       containers:
       - name: cluster-init
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         command:
           - "/cockroach/cockroach"

cloud/kubernetes/v1.7/cockroachdb-statefulset-secure.yaml

@@ -190,7 +190,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 26257

cloud/kubernetes/v1.7/cockroachdb-statefulset.yaml

@@ -93,7 +93,7 @@ spec:
               topologyKey: kubernetes.io/hostname
       containers:
       - name: cockroachdb
-        image: cockroachdb/cockroach:v22.1.5
+        image: cockroachdb/cockroach:v22.1.6
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 26257

docs/generated/settings/settings.html

@@ -56,6 +56,7 @@
 <tr><td><code>kv.snapshot_delegation.enabled</code></td><td>boolean</td><td><code>false</code></td><td>set to true to allow snapshots from follower replicas</td></tr>
 <tr><td><code>kv.snapshot_rebalance.max_rate</code></td><td>byte size</td><td><code>32 MiB</code></td><td>the rate limit (bytes/sec) to use for rebalance and upreplication snapshots</td></tr>
 <tr><td><code>kv.snapshot_recovery.max_rate</code></td><td>byte size</td><td><code>32 MiB</code></td><td>the rate limit (bytes/sec) to use for recovery snapshots</td></tr>
+<tr><td><code>kv.store.admission.provisioned_bandwidth</code></td><td>byte size</td><td><code>0 B</code></td><td>if set to a non-zero value, this is used as the provisioned bandwidth (in bytes/s), for each store. It can be over-ridden on a per-store basis using the --store flag</td></tr>
 <tr><td><code>kv.transaction.max_intents_bytes</code></td><td>integer</td><td><code>4194304</code></td><td>maximum number of bytes used to track locks in transactions</td></tr>
 <tr><td><code>kv.transaction.max_refresh_spans_bytes</code></td><td>integer</td><td><code>4194304</code></td><td>maximum number of bytes used to track refresh spans in serializable transactions</td></tr>
 <tr><td><code>kv.transaction.reject_over_max_intents_budget.enabled</code></td><td>boolean</td><td><code>false</code></td><td>if set, transactions that exceed their lock tracking budget (kv.transaction.max_intents_bytes) are rejected instead of having their lock spans imprecisely compressed</td></tr>

pkg/base/store_spec.go

@@ -161,6 +161,86 @@ func (ss *SizeSpec) Set(value string) error {
 	return nil
 }
 
+// ProvisionedRateSpec is an optional part of the StoreSpec.
+//
+// TODO(sumeer): We should map the file path specified in the store spec to
+// the disk name. df can be used to map paths to names like /dev/nvme1n1 and
+// /dev/sdb (these examples are from AWS EBS and GCP PD respectively) and the
+// corresponding names produced by disk_counters.go are nvme1n1 and sdb
+// respectively. We need to find or write a platform independent library --
+// see the discussion on
+// https://github.com/cockroachdb/cockroach/pull/86063#pullrequestreview-1074487018.
+// With that change, the ProvisionedRateSpec would only be needed to override
+// the cluster setting when there are heterogenous bandwidth limits in a
+// cluster (there would be no more DiskName field).
+type ProvisionedRateSpec struct {
+	// DiskName is the name of the disk observed by the code in disk_counters.go
+	// when retrieving stats for this store.
+	DiskName string
+	// ProvisionedBandwidth is the bandwidth provisioned for this store in
+	// bytes/s.
+	ProvisionedBandwidth int64
+}
+
+func newStoreProvisionedRateSpec(
+	field redact.SafeString, value string,
+) (ProvisionedRateSpec, error) {
+	var spec ProvisionedRateSpec
+	used := make(map[string]struct{})
+	for _, split := range strings.Split(value, ":") {
+		if len(split) == 0 {
+			continue
+		}
+		subSplits := strings.Split(split, "=")
+		if len(subSplits) != 2 {
+			return ProvisionedRateSpec{}, errors.Errorf("%s field has invalid value %s", field, value)
+		}
+		subField := subSplits[0]
+		subValue := subSplits[1]
+		if _, ok := used[subField]; ok {
+			return ProvisionedRateSpec{}, errors.Errorf("%s field has duplicate sub-field %s",
+				field, subField)
+		}
+		used[subField] = struct{}{}
+		if len(subField) == 0 {
+			continue
+		}
+		if len(subValue) == 0 {
+			return ProvisionedRateSpec{},
+				errors.Errorf("%s field has no value specified for sub-field %s", field, subField)
+		}
+		switch subField {
+		case "disk-name":
+			spec.DiskName = subValue
+		case "bandwidth":
+			if len(subValue) <= 2 || subValue[len(subValue)-2:] != "/s" {
+				return ProvisionedRateSpec{},
+					errors.Errorf("%s field does not have bandwidth sub-field %s ending in /s",
+						field, subValue)
+			}
+			subValue = subValue[:len(subValue)-2]
+			var err error
+			spec.ProvisionedBandwidth, err = humanizeutil.ParseBytes(subValue)
+			if err != nil {
+				return ProvisionedRateSpec{},
+					errors.Wrapf(err, "could not parse bandwidth in field %s", field)
+			}
+			if spec.ProvisionedBandwidth == 0 {
+				return ProvisionedRateSpec{},
+					errors.Errorf("%s field is trying to set bandwidth to 0", field)
+			}
+		default:
+			return ProvisionedRateSpec{}, errors.Errorf("%s field has unknown sub-field %s",
+				field, subField)
+		}
+	}
+	if len(spec.DiskName) == 0 {
+		return ProvisionedRateSpec{},
+			errors.Errorf("%s field did not specify disk-name", field)
+	}
+	return spec, nil
+}
+
 // StoreSpec contains the details that can be specified in the cli pertaining
 // to the --store flag.
 type StoreSpec struct {
@@ -189,6 +269,8 @@ type StoreSpec struct {
 	// through to C CCL code to set up encryption-at-rest.  Must be set if and
 	// only if encryption is enabled, otherwise left empty.
 	EncryptionOptions []byte
+	// ProvisionedRateSpec is optional.
+	ProvisionedRateSpec ProvisionedRateSpec
 }
 
 // String returns a fully parsable version of the store spec.
@@ -231,6 +313,16 @@ func (ss StoreSpec) String() string {
 		fmt.Fprint(&buffer, optsStr)
 		fmt.Fprint(&buffer, ",")
 	}
+	if len(ss.ProvisionedRateSpec.DiskName) > 0 {
+		fmt.Fprintf(&buffer, "provisioned-rate=disk-name=%s",
+			ss.ProvisionedRateSpec.DiskName)
+		if ss.ProvisionedRateSpec.ProvisionedBandwidth > 0 {
+			fmt.Fprintf(&buffer, ":bandwidth=%s/s,",
+				humanizeutil.IBytes(ss.ProvisionedRateSpec.ProvisionedBandwidth))
+		} else {
+			fmt.Fprintf(&buffer, ",")
+		}
+	}
 	// Trim the extra comma from the end if it exists.
 	if l := buffer.Len(); l > 0 {
 		buffer.Truncate(l - 1)
@@ -259,7 +351,7 @@ var fractionRegex = regexp.MustCompile(`^([-]?([0-9]+\.[0-9]*|[0-9]*\.[0-9]+|[0-
 
 // NewStoreSpec parses the string passed into a --store flag and returns a
 // StoreSpec if it is correctly parsed.
-// There are four possible fields that can be passed in, comma separated:
+// There are five possible fields that can be passed in, comma separated:
 // - path=xxx The directory in which to the rocks db instance should be
 //   located, required unless using a in memory storage.
 // - type=mem This specifies that the store is an in memory storage instead of
@@ -273,6 +365,10 @@ var fractionRegex = regexp.MustCompile(`^([-]?([0-9]+\.[0-9]*|[0-9]*\.[0-9]+|[0-
 //   - 20%             -> 20% of the available space
 //   - 0.2             -> 20% of the available space
 // - attrs=xxx:yyy:zzz A colon separated list of optional attributes.
+// - provisioned-rate=disk-name=<disk-name>[:bandwidth=<bandwidth-bytes/s>] The
+//   provisioned-rate can be used for admission control for operations on the
+//   store. The bandwidth is optional, and if unspecified, a cluster setting
+//   (kv.store.admission.provisioned_bandwidth) will be used.
 // Note that commas are forbidden within any field name or value.
 func NewStoreSpec(value string) (StoreSpec, error) {
 	const pathField = "path"
@@ -399,6 +495,13 @@ func NewStoreSpec(value string) (StoreSpec, error) {
 				return StoreSpec{}, err
 			}
 			ss.PebbleOptions = buf.String()
+		case "provisioned-rate":
+			rateSpec, err := newStoreProvisionedRateSpec("provisioned-rate", value)
+			if err != nil {
+				return StoreSpec{}, err
+			}
+			ss.ProvisionedRateSpec = rateSpec
+
 		default:
 			return StoreSpec{}, fmt.Errorf("%s is not a valid store field", field)
 		}

pkg/base/store_spec_test.go

@@ -151,6 +151,14 @@ target_file_size=2097152`
 		{"path=/mnt/hda1,type=other", "other is not a valid store type", StoreSpec{}},
 		{"path=/mnt/hda1,type=mem,size=20GiB", "path specified for in memory store", StoreSpec{}},
 
+		// provisioned rate
+		{"path=/mnt/hda1,provisioned-rate=disk-name=nvme1n1:bandwidth=200MiB/s", "",
+			StoreSpec{Path: "/mnt/hda1", ProvisionedRateSpec: base.ProvisionedRateSpec{
+				DiskName: "nvme1n1", ProvisionedBandwidth: 200 << 20}}},
+		{"path=/mnt/hda1,provisioned-rate=disk-name=sdb", "", StoreSpec{
+			Path: "/mnt/hda1", ProvisionedRateSpec: base.ProvisionedRateSpec{
+				DiskName: "sdb", ProvisionedBandwidth: 0}}},
+
 		// RocksDB
 		{"path=/,rocksdb=key1=val1;key2=val2", "", StoreSpec{Path: "/", RocksDBOptions: "key1=val1;key2=val2"}},