Merge #66451 #67426 #67540 #67584 #67614

66451: sql: introduce index usage stats subsystem r=mgartner a=Azhng

First commit:

**roachpb: add protobuf definition for index usage statistics**

Release note: None

Second commit:

**sql: introduce index usage stats subsystem**

This commit introduce a new index usage stats subsystem living inside
the sql package. This subsystem is responsible for asynchronously
collecting index usage statistics from the planner and its is created
within the sql.Server.

Release note: None

Related PR:
* 1/3: Current PR
* 2/3: #66639
* 3/3: #66640

67426: builtin: add {de,}compress builtins r=pbardea a=pbardea

```
[email protected]:26257/movr> select compress('hello there', 'gzip');
                                                        compress
------------------------------------------------------------------------------------------------------------------------
  \037\213\010\000\000\000\000\000\000\377\312H\315\311\311W(\311H-J\005\004\000\000\377\377P\351\022m\013\000\000\000
(1 row)

[email protected]:26257/movr> select decompress(compress('hello there', 'gzip'), 'gzip');
  decompress
---------------
  hello there
(1 row)

[email protected]:26257/movr> select length(repeat('hi there', 100)::bytes);
  length
----------
     800
(1 row)

[email protected]:26257/movr> select length(compress(repeat('hi there', 100)::bytes, 'gzip'));
  length
----------
      39
(1 row)

[email protected]:26257/movr> select decompress('not compressed', 'gzip');
ERROR: decompress(): failed to decompress: gzip: invalid header
[email protected]:26257/movr> select decompress('bad codec', 'zip');
ERROR: decompress(): only 'gzip' codec is supported for decompress()
SQLSTATE: 22023
```

This commits adds builtins to compress and decompress bytes. The only
codec currently supported is `gzip`.

Release note (sql change): New builtins, compress(data, codec) and
decompress(data, codec), are added which can compress and decompress
bytes with the specified codec.  Gzip is the only currently supported
codec.


67540: dev: infer and pass in an appropriate `dev` config to bazel invocations r=rail a=rickystewart

Passing `--config=dev` for `dev` builds is a good choice, and
`--config=devdarwinx86_64` is required for most dev builds on macOS
machines. Automatically pass that flag unless the user requests not to.

Release note: None

67584: parser: move scan.go to new scanner package  r=knz,otan a=rafiss

Fixes #64710

See individual commits.

This extracts an interface for sqlSymType so that scan.go can keep using it.

This allows scan.go to live in its own package so that other components can
use its functionality without needing to pull in a large dependency
on sql/sem/tree.

Release note: None

67614: backfill: fix backfills for indexes on non-null virtual columns r=mgartner a=mgartner

Refactoring in #65514 broke backfills for indexes that include non-null
virtual columns. This specific case was not tested so it was not caught
earlier. This commit fixes the issue and adds a test.

Fixes #67528

There is no release note because this bug was not included in any prior
releases.

Release note: None

Co-authored-by: Azhng <[email protected]>
Co-authored-by: Paul Bardea <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
Co-authored-by: Marcus Gartner <[email protected]>

docs/generated/settings/settings-for-tenants.txt

@@ -106,6 +106,7 @@ sql.distsql.temp_storage.workmem	byte size	64 MiB	maximum amount of memory in by
 sql.log.slow_query.experimental_full_table_scans.enabled	boolean	false	when set to true, statements that perform a full table/index scan will be logged to the slow query log even if they do not meet the latency threshold. Must have the slow query log enabled for this setting to have any effect.
 sql.log.slow_query.internal_queries.enabled	boolean	false	when set to true, internal queries which exceed the slow query log threshold are logged to a separate log. Must have the slow query log enabled for this setting to have any effect.
 sql.log.slow_query.latency_threshold	duration	0s	when set to non-zero, log statements whose service latency exceeds the threshold to a secondary logger on each node
+sql.metrics.index_usage_stats.enabled	boolean	true	collect per index usage statistics
 sql.metrics.max_mem_reported_stmt_fingerprints	integer	100000	the maximum number of reported statement fingerprints stored in memory
 sql.metrics.max_mem_reported_txn_fingerprints	integer	100000	the maximum number of reported transaction fingerprints stored in memory
 sql.metrics.max_mem_stmt_fingerprints	integer	100000	the maximum number of statement fingerprints stored in memory

docs/generated/settings/settings.html

@@ -107,6 +107,7 @@
 <tr><td><code>sql.log.slow_query.experimental_full_table_scans.enabled</code></td><td>boolean</td><td><code>false</code></td><td>when set to true, statements that perform a full table/index scan will be logged to the slow query log even if they do not meet the latency threshold. Must have the slow query log enabled for this setting to have any effect.</td></tr>
 <tr><td><code>sql.log.slow_query.internal_queries.enabled</code></td><td>boolean</td><td><code>false</code></td><td>when set to true, internal queries which exceed the slow query log threshold are logged to a separate log. Must have the slow query log enabled for this setting to have any effect.</td></tr>
 <tr><td><code>sql.log.slow_query.latency_threshold</code></td><td>duration</td><td><code>0s</code></td><td>when set to non-zero, log statements whose service latency exceeds the threshold to a secondary logger on each node</td></tr>
+<tr><td><code>sql.metrics.index_usage_stats.enabled</code></td><td>boolean</td><td><code>true</code></td><td>collect per index usage statistics</td></tr>
 <tr><td><code>sql.metrics.max_mem_reported_stmt_fingerprints</code></td><td>integer</td><td><code>100000</code></td><td>the maximum number of reported statement fingerprints stored in memory</td></tr>
 <tr><td><code>sql.metrics.max_mem_reported_txn_fingerprints</code></td><td>integer</td><td><code>100000</code></td><td>the maximum number of reported transaction fingerprints stored in memory</td></tr>
 <tr><td><code>sql.metrics.max_mem_stmt_fingerprints</code></td><td>integer</td><td><code>100000</code></td><td>the maximum number of statement fingerprints stored in memory</td></tr>

docs/generated/sql/functions.md

@@ -2407,6 +2407,8 @@ The swap_ordinate_string parameter is a 2-character string naming the ordinates
 </span></td></tr>
 <tr><td><a name="chr"></a><code>chr(val: <a href="int.html">int</a>) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Returns the character with the code given in <code>val</code>. Inverse function of <code>ascii()</code>.</p>
 </span></td></tr>
+<tr><td><a name="compress"></a><code>compress(data: <a href="bytes.html">bytes</a>, codec: <a href="string.html">string</a>) &rarr; <a href="bytes.html">bytes</a></code></td><td><span class="funcdesc"><p>Compress <code>data</code> with the specified <code>codec</code> (<code>gzip</code>).</p>
+</span></td></tr>
 <tr><td><a name="concat"></a><code>concat(<a href="string.html">string</a>...) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Concatenates a comma-separated list of strings.</p>
 </span></td></tr>
 <tr><td><a name="concat_ws"></a><code>concat_ws(<a href="string.html">string</a>...) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Uses the first argument as a separator between the concatenation of the subsequent arguments.</p>
@@ -2426,6 +2428,8 @@ The output can be used to recreate a database.’</p>
 </span></td></tr>
 <tr><td><a name="decode"></a><code>decode(text: <a href="string.html">string</a>, format: <a href="string.html">string</a>) &rarr; <a href="bytes.html">bytes</a></code></td><td><span class="funcdesc"><p>Decodes <code>data</code> using <code>format</code> (<code>hex</code> / <code>escape</code> / <code>base64</code>).</p>
 </span></td></tr>
+<tr><td><a name="decompress"></a><code>decompress(data: <a href="bytes.html">bytes</a>, codec: <a href="string.html">string</a>) &rarr; <a href="bytes.html">bytes</a></code></td><td><span class="funcdesc"><p>Decompress <code>data</code> with the specified <code>codec</code> (<code>gzip</code>).</p>
+</span></td></tr>
 <tr><td><a name="difference"></a><code>difference(source: <a href="string.html">string</a>, target: <a href="string.html">string</a>) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Convert two strings to their Soundex codes and then reports the number of matching code positions.</p>
 </span></td></tr>
 <tr><td><a name="encode"></a><code>encode(data: <a href="bytes.html">bytes</a>, format: <a href="string.html">string</a>) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Encodes <code>data</code> using <code>format</code> (<code>hex</code> / <code>escape</code> / <code>base64</code>).</p>

pkg/BUILD.bazel

@@ -219,6 +219,7 @@ ALL_TESTS = [
     "//pkg/sql/gcjob/gcjobnotifier:gcjobnotifier_test",
     "//pkg/sql/gcjob:gcjob_test",
     "//pkg/sql/gcjob_test:gcjob_test_test",
+    "//pkg/sql/idxusage:idxusage_test",
     "//pkg/sql/inverted:inverted_test",
     "//pkg/sql/lex:lex_test",
     "//pkg/sql/lexbase:lexbase_test",

pkg/cmd/dev/build.go

@@ -71,6 +71,7 @@ func (d *dev) build(cmd *cobra.Command, targets []string) (err error) {
 	// Don't let bazel generate any convenience symlinks, we'll create them
 	// ourself.
 	args = append(args, "--experimental_convenience_symlinks=ignore")
+	args = append(args, getConfigFlags()...)
 	args = append(args, mustGetRemoteCacheArgs(remoteCacheAddr)...)
 	if numCPUs != 0 {
 		args = append(args, fmt.Sprintf("--local_cpu_resources=%d", numCPUs))
@@ -131,7 +132,9 @@ func (d *dev) symlinkBinaries(ctx context.Context, targets []string) error {
 }
 
 func (d *dev) getPathToBin(ctx context.Context, target string) (string, error) {
-	out, err := d.exec.CommandContextSilent(ctx, "bazel", "info", "bazel-bin", "--color=no")
+	args := []string{"info", "bazel-bin", "--color=no"}
+	args = append(args, getConfigFlags()...)
+	out, err := d.exec.CommandContextSilent(ctx, "bazel", args...)
 	if err != nil {
 		return "", err
 	}

pkg/cmd/dev/datadriven_test.go

@@ -43,6 +43,10 @@ var (
 	)
 )
 
+func init() {
+	predictableDevProfile = true
+}
+
 // TestDatadriven makes use of datadriven and recorder to capture all operations
 // executed by individual `dev` invocations. The testcases are defined under
 // testdata/*, where each test files corresponds to a recording capture found in

pkg/cmd/dev/dev.go

@@ -31,6 +31,7 @@ var (
 	// Shared flags.
 	remoteCacheAddr string
 	numCPUs         int
+	skipDevConfig   bool
 )
 
 func makeDevCmd() *dev {
@@ -76,7 +77,6 @@ lets engineers do a few things:
 		makeLintCmd(ret.lint),
 		makeTestCmd(ret.test),
 	)
-
 	// Add all the shared flags.
 	var debugVar bool
 	for _, subCmd := range ret.cli.Commands() {
@@ -88,6 +88,7 @@ lets engineers do a few things:
 		// support for the  Remote Asset API, which helps speed things up when
 		// the cache sits across the network boundary.
 		subCmd.Flags().StringVar(&remoteCacheAddr, "remote-cache", "", "remote caching grpc endpoint to use")
+		subCmd.Flags().BoolVar(&skipDevConfig, "skip-dev-config", false, "Don't infer an appropriate dev config to build with")
 	}
 	for _, subCmd := range ret.cli.Commands() {
 		subCmd.PreRun = func(cmd *cobra.Command, args []string) {

pkg/cmd/dev/test.go

@@ -109,7 +109,8 @@ func (d *dev) runUnitTest(cmd *cobra.Command, pkgs []string) error {
 	var args []string
 	args = append(args, "test")
 	args = append(args, "--color=yes")
-	args = append(args, "--experimental_convenience_symlinks=ignore") // don't generate any convenience symlinks
+	args = append(args, "--experimental_convenience_symlinks=ignore")
+	args = append(args, getConfigFlags()...)
 	args = append(args, mustGetRemoteCacheArgs(remoteCacheAddr)...)
 	if numCPUs != 0 {
 		args = append(args, fmt.Sprintf("--local_cpu_resources=%d", numCPUs))

pkg/cmd/dev/testdata/build.txt

@@ -1,33 +1,42 @@
 dev build cockroach-short
 ----
-bazel build --color=yes --experimental_convenience_symlinks=ignore //pkg/cmd/cockroach-short
-bazel info workspace --color=no
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev //pkg/cmd/cockroach-short
+bazel info workspace --color=no --config=dev
 mkdir go/src/github.com/cockroachdb/cockroach/bin
-bazel info bazel-bin --color=no
+bazel info bazel-bin --color=no --config=dev
 rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 
 dev build cockroach-short --cpus=12
 ----
-bazel build --color=yes --experimental_convenience_symlinks=ignore --local_cpu_resources=12 //pkg/cmd/cockroach-short
-bazel info workspace --color=no
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev --local_cpu_resources=12 //pkg/cmd/cockroach-short
+bazel info workspace --color=no --config=dev
 mkdir go/src/github.com/cockroachdb/cockroach/bin
-bazel info bazel-bin --color=no
+bazel info bazel-bin --color=no --config=dev
 rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 
 dev build --debug cockroach-short
 ----
-bazel build --color=yes --experimental_convenience_symlinks=ignore //pkg/cmd/cockroach-short
-bazel info workspace --color=no
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev //pkg/cmd/cockroach-short
+bazel info workspace --color=no --config=dev
 mkdir go/src/github.com/cockroachdb/cockroach/bin
-bazel info bazel-bin --color=no
+bazel info bazel-bin --color=no --config=dev
 rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 
 dev build cockroach-short --remote-cache 127.0.0.1:9090
 ----
-bazel build --color=yes --experimental_convenience_symlinks=ignore --remote_local_fallback --remote_cache=grpc://127.0.0.1:9090 --experimental_remote_downloader=grpc://127.0.0.1:9090 //pkg/cmd/cockroach-short
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev --remote_local_fallback --remote_cache=grpc://127.0.0.1:9090 --experimental_remote_downloader=grpc://127.0.0.1:9090 //pkg/cmd/cockroach-short
+bazel info workspace --color=no --config=dev
+mkdir go/src/github.com/cockroachdb/cockroach/bin
+bazel info bazel-bin --color=no --config=dev
+rm go/src/github.com/cockroachdb/cockroach/cockroach-short
+ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
+
+dev build --skip-dev-config cockroach-short
+----
+bazel build --color=yes --experimental_convenience_symlinks=ignore //pkg/cmd/cockroach-short
 bazel info workspace --color=no
 mkdir go/src/github.com/cockroachdb/cockroach/bin
 bazel info bazel-bin --color=no

pkg/cmd/dev/testdata/generate.txt

@@ -1,4 +1,4 @@
 dev gen bazel
 ----
-bazel info workspace --color=no
+bazel info workspace --color=no --config=dev
 go/src/github.com/cockroachdb/cockroach/build/bazelutil/bazel-generate.sh

pkg/cmd/dev/testdata/recording/build.txt

@@ -1,14 +1,14 @@
-bazel build --color=yes --experimental_convenience_symlinks=ignore //pkg/cmd/cockroach-short
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev //pkg/cmd/cockroach-short
 ----
 
-bazel info workspace --color=no
+bazel info workspace --color=no --config=dev
 ----
 go/src/github.com/cockroachdb/cockroach
 
 mkdir go/src/github.com/cockroachdb/cockroach/bin
 ----
 
-bazel info bazel-bin --color=no
+bazel info bazel-bin --color=no --config=dev
 ----
 /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin
 
@@ -18,17 +18,17 @@ rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 ----
 
-bazel build --color=yes --experimental_convenience_symlinks=ignore --local_cpu_resources=12 //pkg/cmd/cockroach-short
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev --local_cpu_resources=12 //pkg/cmd/cockroach-short
 ----
 
-bazel info workspace --color=no
+bazel info workspace --color=no --config=dev
 ----
 go/src/github.com/cockroachdb/cockroach
 
 mkdir go/src/github.com/cockroachdb/cockroach/bin
 ----
 
-bazel info bazel-bin --color=no
+bazel info bazel-bin --color=no --config=dev
 ----
 /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin
 
@@ -38,17 +38,17 @@ rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 ----
 
-bazel build --color=yes --experimental_convenience_symlinks=ignore //pkg/cmd/cockroach-short
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev //pkg/cmd/cockroach-short
 ----
 
-bazel info workspace --color=no
+bazel info workspace --color=no --config=dev
 ----
 go/src/github.com/cockroachdb/cockroach
 
 mkdir go/src/github.com/cockroachdb/cockroach/bin
 ----
 
-bazel info bazel-bin --color=no
+bazel info bazel-bin --color=no --config=dev
 ----
 /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin
 
@@ -58,7 +58,27 @@ rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 ----
 
-bazel build --color=yes --experimental_convenience_symlinks=ignore --remote_local_fallback --remote_cache=grpc://127.0.0.1:9090 --experimental_remote_downloader=grpc://127.0.0.1:9090 //pkg/cmd/cockroach-short
+bazel build --color=yes --experimental_convenience_symlinks=ignore --config=dev --remote_local_fallback --remote_cache=grpc://127.0.0.1:9090 --experimental_remote_downloader=grpc://127.0.0.1:9090 //pkg/cmd/cockroach-short
+----
+
+bazel info workspace --color=no --config=dev
+----
+go/src/github.com/cockroachdb/cockroach
+
+mkdir go/src/github.com/cockroachdb/cockroach/bin
+----
+
+bazel info bazel-bin --color=no --config=dev
+----
+/private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin
+
+rm go/src/github.com/cockroachdb/cockroach/cockroach-short
+----
+
+ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
+----
+
+bazel build --color=yes --experimental_convenience_symlinks=ignore //pkg/cmd/cockroach-short
 ----
 
 bazel info workspace --color=no
@@ -77,4 +97,3 @@ rm go/src/github.com/cockroachdb/cockroach/cockroach-short
 
 ln -s /private/var/tmp/_bazel/99e666e4e674209ecdb66b46371278df/execroot/cockroach/bazel-out/darwin-fastbuild/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short go/src/github.com/cockroachdb/cockroach/cockroach-short
 ----
-

pkg/cmd/dev/testdata/recording/generate.txt

@@ -1,4 +1,4 @@
-bazel info workspace --color=no
+bazel info workspace --color=no --config=dev
 ----
 go/src/github.com/cockroachdb/cockroach