Merge #86460 #86528 #86572 #86579 #86586

86460: opt: reduce statistics allocations for avg size r=mgartner a=mgartner

Prior to the commit, a column's average size in bytes was included in
column statistics. To fetch this average size, the coster requested an
individual column statistic each scanned column. For scans and joins
involving many columns, this caused many allocations of column
statistics and column sets.

Because we only use a column's average size when costing scans and
lookup joins, there was no need to include it in column statistics.
Average size doesn't propagate up an expression tree like other
statistics do.

This commit removes average size from column statistics and instead
builds a map in `props.Statistics` that maps column IDs to average size.
This significantly reduces allocations in some cases.

The only downside to this change is that we no longer set a columns
average size to zero if it has all NULL values, according to statistics.
I believe this is a pretty rare edge case that is unlikely to
significantly affect query plans, so I think the trade-off is worth it.

Fixes #80186

Release justification: This is a minor change that improves optimizer
performance.

Release note: None


86528: storage: add default-off setting for MVCC range tombstones r=msbutler,nicktrav a=erikgrinaker

This patch adds the default-off cluster setting
`storage.mvcc.range_tombstones.enabled` to control whether or not to
write MVCC range tombstones. The setting is internal and system-only.
The read path is always active, this only determines whether KV clients
should write them.

A helper function `CanUseMVCCRangeTombstones()` has also been added.
Callers have not yet been updated to respect this.

Note that any in-flight jobs may not pick up this change, so these need
to be waited out before being certain that the setting has taken effect.

If disabled after being enabled, this will prevent new range tombstones
from being written, but already written tombstones will remain until
GCed. The above note on jobs above also applies in this case.

Release justification: bug fixes and low-risk updates to new functionality

Release note: None

86572: ui: update styles on sessions details page r=maryliag a=maryliag

The Session Details page was updated to use the
same style of summary cards as the other details
pages (e.g. statement, transaction, job).

Fixes #85257

Before
<img width="1236" alt="Screen Shot 2022-08-22 at 12 26 43 PM" src="https://user-images.githubusercontent.com/1017486/185971455-3dc7b57f-07bc-45df-94e3-f0bd7b3e541a.png">


After
<img width="1250" alt="Screen Shot 2022-08-22 at 12 26 19 PM" src="https://user-images.githubusercontent.com/1017486/185971475-1bd563f6-a596-4321-9f90-d6c68470dbb9.png">


Release justification: low risk change
Release note (ui change): New styles of summary cards
on Session Details page to align with other details pages.

86579: sql/builtins: update `Info` for `pg_get_viewdef` r=ZhouXing19 a=ZhouXing19

The pg_builtin func `pg_get_viewdef` was updated from a no-op to an actual
function long time ago, but the info field is still `notUsableInfo`, which
made the doc to miss it by mistake. This PR is to update the info, and let it
be recorded in `functions.md`.

Release justification: bug fix, update a builtin's visibility in docs.
Release note: none

86586: README: make sure roachprod/roachtest docs use dev, not make r=rail a=rickystewart

Release justification: Non-production code changes
Release note: None

Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: Marylia Gutierrez <[email protected]>
Co-authored-by: Jane Xing <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>

docs/generated/sql/functions.md

@@ -3457,6 +3457,10 @@ A write probe will effectively probe reads as well.</p>
 </span></td><td>Immutable</td></tr>
 <tr><td><a name="pg_get_serial_sequence"></a><code>pg_get_serial_sequence(table_name: <a href="string.html">string</a>, column_name: <a href="string.html">string</a>) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Returns the name of the sequence used by the given column_name in the table table_name.</p>
 </span></td><td>Stable</td></tr>
+<tr><td><a name="pg_get_viewdef"></a><code>pg_get_viewdef(view_oid: oid) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Returns the CREATE statement for an existing view.</p>
+</span></td><td>Stable</td></tr>
+<tr><td><a name="pg_get_viewdef"></a><code>pg_get_viewdef(view_oid: oid, pretty_bool: <a href="bool.html">bool</a>) &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Returns the CREATE statement for an existing view.</p>
+</span></td><td>Stable</td></tr>
 <tr><td><a name="pg_has_role"></a><code>pg_has_role(role: <a href="string.html">string</a>, privilege: <a href="string.html">string</a>) &rarr; <a href="bool.html">bool</a></code></td><td><span class="funcdesc"><p>Returns whether or not the current user has privileges for role.</p>
 </span></td><td>Stable</td></tr>
 <tr><td><a name="pg_has_role"></a><code>pg_has_role(role: oid, privilege: <a href="string.html">string</a>) &rarr; <a href="bool.html">bool</a></code></td><td><span class="funcdesc"><p>Returns whether or not the current user has privileges for role.</p>

pkg/cmd/roachprod/README.md

@@ -10,7 +10,7 @@ Note that an internal tutorial is also maintained [at the Developer Infrastructu
 1. Make sure you have [gcloud installed] and configured (`gcloud auth list` to
 check, `gcloud auth login` to authenticate). You may want to update old
 installations (`gcloud components update`).
-1. Build a local binary of `roachprod`: `make bin/roachprod`
+1. Build a local binary of `roachprod`: `dev build roachprod`
 1. Add `$PWD/bin` to your `PATH` so you can run `roachprod` from the root directory of `cockroach`.
 
 ## Summary
@@ -45,7 +45,7 @@ roachprod stage ${CLUSTER} workload
 roachprod stage ${CLUSTER} release v2.0.5
 
 # ...or using roachprod directly (e.g., for your locally-built binary).
-build/builder.sh mkrelease
+dev build --cross
 roachprod put ${CLUSTER} cockroach-linux-2.6.32-gnu-amd64 cockroach
 
 # Start a cluster.

pkg/cmd/roachtest/README.md

@@ -53,7 +53,7 @@ As a general rule of thumb, if running against "real" VMs, and the version to
 test is checked out, the following will build the right binaries and put them
 where `roachtest` will find them:
 
-`build/builder.sh mkrelease amd64-linux-gnu build bin/workload`
+`dev build --cross cockroach workload`
 
 `roachtest` will look first in `$PATH`, then (except when `--local` is
 specified) in `bin.docker_amd64` in the repo root, followed by `bin`. This
@@ -65,7 +65,7 @@ cluster on the local machine created via `roachprod create local`. In that
 case, the `cockroach` and `workload` binary must target the local architecture,
 which means
 
-`make build bin/workload`
+`dev build cockroach workload`
 
 will do the trick. However, most roachtests don't make a lot of sense in local
 mode or will do outright dangerous things (like trying to install packages on
@@ -87,14 +87,12 @@ mv ~/local/1/cockroach cockroach
 ```
 
 This doesn't work for the `workload` binary but this builds a lot faster anyway
-(via `build/builder.sh mkrelease amd64-linux-gnu bin/workload`; note the
-absence of the word `build` which prevents unnecessarily building CockroachDB
-again).  The above strategy also works for recent SHAs from the master/release
-branches, where the incantation is `roachprod stage local cockroach <SHA>`; the
-SHA is optional and defaults to a recent build from the `master` branch, which
-may not be what you want. As a bonus, `roachprod stage local workload <SHA>
---os linux` does allow getting a `workload` binary as well, meaning you can run
-a roachtest without compiling anything yourself.
+(via `dev build --cross workload`).  The above strategy also works for recent
+SHAs from the master/release branches, where the incantation is
+`roachprod stage local cockroach <SHA>`; the SHA is optional and defaults to a
+recent build from the `master` branch, which may not be what you want. As a bonus,
+`roachprod stage local workload <SHA> --os linux` does allow getting a `workload`
+binary as well, meaning you can run a roachtest without compiling anything yourself.
 
 ## Running a test
 
@@ -169,7 +167,7 @@ case which is helpful during development), don't forget to rebuild the binaries
 you've touched (i.e. most likely only `roachtest`). For example, the command
 line that you might use while iterating on a new test `foo/garble-wozzler` is
 
-`make bin/roachtest && roachtest run --local foo/garble-wozzler`.
+`dev build roachtest && roachtest run --local foo/garble-wozzler`.
 
 ### Reproducing a test failure
 

pkg/kv/batch.go

@@ -657,8 +657,8 @@ func (b *Batch) DelRange(s, e interface{}, returnKeys bool) {
 }
 
 // DelRangeUsingTombstone deletes the rows between begin (inclusive) and end
-// (exclusive) using an MVCC range tombstone. Callers must check the
-// MVCCRangeTombstones version gate before using this.
+// (exclusive) using an MVCC range tombstone. Callers must check
+// storage.CanUseMVCCRangeTombstones before using this.
 func (b *Batch) DelRangeUsingTombstone(s, e interface{}) {
 	start, err := marshalKey(s)
 	if err != nil {

pkg/roachpb/api.proto

@@ -375,8 +375,9 @@ message DeleteRangeRequest {
   // for conflicts and adjust MVCC stats). This option cannot be used in a
   // transaction, and it cannot be combined with Inline or ReturnKeys.
   //
-  // The caller must check the MVCCRangeTombstones version gate before using
-  // this parameter, as it is new in 22.2.
+  // The caller must check storage.CanUseMVCCRangeTombstones before using this
+  // parameter: it is new in 22.2, and controlled by the default-off cluster
+  // setting storage.mvcc.range_tombstones.enabled.
   bool use_range_tombstone = 5;
   // If enabled together with UseRangeTombstone, the MVCC range tombstone will
   // only be written if there exists point key/tombstones in the span that

pkg/sql/opt/exec/execbuilder/testdata/distsql_agg

@@ -1023,7 +1023,7 @@ group-by (streaming)
  ├── columns: b:2 count:5
  ├── grouping columns: b:2
  ├── internal-ordering: +2 opt(1)
- ├── stats: [rows=9.561793, distinct(2)=9.56179, null(2)=0, avgsize(2)=4]
+ ├── stats: [rows=9.561793, distinct(2)=9.56179, null(2)=0]
  ├── cost: 25.145618
  ├── key: (2)
  ├── fd: (2)-->(5)
@@ -1032,7 +1032,7 @@ group-by (streaming)
  ├── scan data2
  │    ├── columns: a:1 b:2
  │    ├── constraint: /1/2: [/1 - /1]
- │    ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4, distinct(2)=9.56179, null(2)=0, avgsize(2)=4]
+ │    ├── stats: [rows=10, distinct(1)=1, null(1)=0, distinct(2)=9.56179, null(2)=0]
  │    ├── cost: 24.8200001
  │    ├── key: (2)
  │    ├── fd: ()-->(1)

pkg/sql/opt/exec/execbuilder/testdata/enums

@@ -121,13 +121,13 @@ distinct-on
  ├── grouping columns: x:1
  ├── internal-ordering: +1
  ├── cardinality: [0 - 4]
- ├── stats: [rows=4, distinct(1)=4, null(1)=0, avgsize(1)=4]
+ ├── stats: [rows=4, distinct(1)=4, null(1)=0]
  ├── cost: 1114.88
  ├── key: (1)
  ├── distribution: test
  └── scan checks@checks_x_y_idx
       ├── columns: x:1
-      ├── stats: [rows=1000, distinct(1)=4, null(1)=0, avgsize(1)=4]
+      ├── stats: [rows=1000, distinct(1)=4, null(1)=0]
       ├── cost: 1104.82
       ├── ordering: +1
       ├── distribution: test
@@ -172,28 +172,28 @@ union
  ├── right columns: nulls.x:6
  ├── internal-ordering: +11
  ├── cardinality: [0 - 5]
- ├── stats: [rows=5, distinct(11)=5, null(11)=1, avgsize(11)=4]
+ ├── stats: [rows=5, distinct(11)=5, null(11)=1]
  ├── cost: 2276.86667
  ├── key: (11)
  ├── distribution: test
  ├── interesting orderings: (+11)
  ├── project
  │    ├── columns: nulls.x:1
- │    ├── stats: [rows=333.3333, distinct(1)=5, null(1)=3.33333, avgsize(1)=4]
+ │    ├── stats: [rows=333.3333, distinct(1)=5, null(1)=3.33333]
  │    ├── cost: 1138.40333
  │    ├── ordering: +1
  │    ├── distribution: test
  │    ├── interesting orderings: (+1)
  │    └── select
  │         ├── columns: nulls.x:1 y:2
- │         ├── stats: [rows=333.3333, distinct(1)=5, null(1)=3.33333, avgsize(1)=4, distinct(2)=33.3333, null(2)=0, avgsize(2)=4]
+ │         ├── stats: [rows=333.3333, distinct(1)=5, null(1)=3.33333, distinct(2)=33.3333, null(2)=0]
  │         ├── cost: 1135.05
  │         ├── ordering: +1
  │         ├── distribution: test
  │         ├── interesting orderings: (+1,+2)
  │         ├── scan nulls@nulls_x_y_idx
  │         │    ├── columns: nulls.x:1 y:2
- │         │    ├── stats: [rows=1000, distinct(1)=5, null(1)=10, avgsize(1)=4, distinct(2)=100, null(2)=10, avgsize(2)=4]
+ │         │    ├── stats: [rows=1000, distinct(1)=5, null(1)=10, distinct(2)=100, null(2)=10]
  │         │    ├── cost: 1125.02
  │         │    ├── ordering: +1
  │         │    ├── distribution: test
@@ -203,21 +203,21 @@ union
  │              └── y:2 < 0 [outer=(2), constraints=(/2: (/NULL - /-1]; tight)]
  └── project
       ├── columns: nulls.x:6
-      ├── stats: [rows=333.3333, distinct(6)=5, null(6)=3.33333, avgsize(6)=4]
+      ├── stats: [rows=333.3333, distinct(6)=5, null(6)=3.33333]
       ├── cost: 1138.40333
       ├── ordering: +6
       ├── distribution: test
       ├── interesting orderings: (+6)
       └── select
            ├── columns: nulls.x:6 y:7
-           ├── stats: [rows=333.3333, distinct(6)=5, null(6)=3.33333, avgsize(6)=4, distinct(7)=33.3333, null(7)=0, avgsize(7)=4]
+           ├── stats: [rows=333.3333, distinct(6)=5, null(6)=3.33333, distinct(7)=33.3333, null(7)=0]
            ├── cost: 1135.05
            ├── ordering: +6
            ├── distribution: test
            ├── interesting orderings: (+6,+7)
            ├── scan nulls@nulls_x_y_idx
            │    ├── columns: nulls.x:6 y:7
-           │    ├── stats: [rows=1000, distinct(6)=5, null(6)=10, avgsize(6)=4, distinct(7)=100, null(7)=10, avgsize(7)=4]
+           │    ├── stats: [rows=1000, distinct(6)=5, null(6)=10, distinct(7)=100, null(7)=10]
            │    ├── cost: 1125.02
            │    ├── ordering: +6
            │    ├── distribution: test

pkg/sql/opt/exec/execbuilder/testdata/explain

@@ -1300,21 +1300,21 @@ EXPLAIN (OPT,VERBOSE) SELECT * FROM tc WHERE a = 10 ORDER BY b
 ----
 sort
  ├── columns: a:1 b:2
- ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4]
+ ├── stats: [rows=10, distinct(1)=1, null(1)=0]
  ├── cost: 86.8243864
  ├── fd: ()-->(1)
  ├── ordering: +2 opt(1) [actual: +2]
  ├── prune: (2)
  └── index-join tc
       ├── columns: a:1 b:2
-      ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4]
+      ├── stats: [rows=10, distinct(1)=1, null(1)=0]
       ├── cost: 85.7400007
       ├── fd: ()-->(1)
       ├── prune: (2)
       └── scan tc@c
            ├── columns: a:1 rowid:3
            ├── constraint: /1/3: [/10 - /10]
-           ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4]
+           ├── stats: [rows=10, distinct(1)=1, null(1)=0]
            ├── cost: 24.8200001
            ├── key: (3)
            └── fd: ()-->(1)
@@ -1324,21 +1324,21 @@ EXPLAIN (OPT,TYPES) SELECT * FROM tc WHERE a = 10 ORDER BY b
 ----
 sort
  ├── columns: a:1(int!null) b:2(int)
- ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4]
+ ├── stats: [rows=10, distinct(1)=1, null(1)=0]
  ├── cost: 86.8243864
  ├── fd: ()-->(1)
  ├── ordering: +2 opt(1) [actual: +2]
  ├── prune: (2)
  └── index-join tc
       ├── columns: a:1(int!null) b:2(int)
-      ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4]
+      ├── stats: [rows=10, distinct(1)=1, null(1)=0]
       ├── cost: 85.7400007
       ├── fd: ()-->(1)
       ├── prune: (2)
       └── scan tc@c
            ├── columns: a:1(int!null) rowid:3(int!null)
            ├── constraint: /1/3: [/10 - /10]
-           ├── stats: [rows=10, distinct(1)=1, null(1)=0, avgsize(1)=4]
+           ├── stats: [rows=10, distinct(1)=1, null(1)=0]
            ├── cost: 24.8200001
            ├── key: (3)
            └── fd: ()-->(1)
@@ -1386,22 +1386,22 @@ TABLE t
 inner-join (hash)
  ├── columns: a:1 b:2 k:6 v:7
  ├── multiplicity: left-rows(zero-or-one), right-rows(zero-or-more)
- ├── stats: [rows=990, distinct(1)=99, null(1)=0, avgsize(1)=4, distinct(6)=99, null(6)=0, avgsize(6)=4]
+ ├── stats: [rows=990, distinct(1)=99, null(1)=0, distinct(6)=99, null(6)=0]
  ├── cost: 2269.90625
  ├── fd: (6)-->(7), (1)==(6), (6)==(1)
  ├── distribution: test
  ├── prune: (2,7)
  ├── scan tc
  │    ├── columns: a:1 b:2
- │    ├── stats: [rows=1000, distinct(1)=100, null(1)=10, avgsize(1)=4]
+ │    ├── stats: [rows=1000, distinct(1)=100, null(1)=10]
  │    ├── cost: 1125.02
  │    ├── distribution: test
  │    ├── prune: (1,2)
  │    ├── interesting orderings: (+1)
  │    └── unfiltered-cols: (1-5)
  ├── scan t
  │    ├── columns: k:6 v:7
- │    ├── stats: [rows=1000, distinct(6)=1000, null(6)=0, avgsize(6)=4]
+ │    ├── stats: [rows=1000, distinct(6)=1000, null(6)=0]
  │    ├── cost: 1104.82
  │    ├── key: (6)
  │    ├── fd: (6)-->(7)
@@ -1942,22 +1942,22 @@ memo (optimized, ~11KB, required=[presentation: info:10] [distribution: test])
 inner-join (hash)
  ├── columns: a:1 b:2 k:6 v:7
  ├── multiplicity: left-rows(zero-or-one), right-rows(zero-or-more)
- ├── stats: [rows=990, distinct(1)=99, null(1)=0, avgsize(1)=4, distinct(6)=99, null(6)=0, avgsize(6)=4]
+ ├── stats: [rows=990, distinct(1)=99, null(1)=0, distinct(6)=99, null(6)=0]
  ├── cost: 2269.90625
  ├── fd: (6)-->(7), (1)==(6), (6)==(1)
  ├── distribution: test
  ├── prune: (2,7)
  ├── scan tc
  │    ├── columns: a:1 b:2
- │    ├── stats: [rows=1000, distinct(1)=100, null(1)=10, avgsize(1)=4]
+ │    ├── stats: [rows=1000, distinct(1)=100, null(1)=10]
  │    ├── cost: 1125.02
  │    ├── distribution: test
  │    ├── prune: (1,2)
  │    ├── interesting orderings: (+1)
  │    └── unfiltered-cols: (1-5)
  ├── scan t
  │    ├── columns: k:6 v:7
- │    ├── stats: [rows=1000, distinct(6)=1000, null(6)=0, avgsize(6)=4]
+ │    ├── stats: [rows=1000, distinct(6)=1000, null(6)=0]
  │    ├── cost: 1104.82
  │    ├── key: (6)
  │    ├── fd: (6)-->(7)

pkg/sql/opt/exec/execbuilder/testdata/forecast

@@ -193,7 +193,7 @@ scan g
  ├── columns: a:1
  ├── constraint: /1: [/0 - /99]
  ├── cardinality: [0 - 100]
- ├── stats: [rows=12, distinct(1)=10, null(1)=0, avgsize(1)=1]
+ ├── stats: [rows=12, distinct(1)=10, null(1)=0]
  │   histogram(1)=  0 1.3333 0 0.66667 0 0.66667 0 1.3333 0 1.3333 0 0.66667 0 0.66667 0 1.3333 0 1.3333 0 0.66667 0 0.66667 0 1.3333
  │                <---- 0 ------- 1 ------- 2 ------ 3 ------ 4 ------- 5 ------- 6 ------ 7 ------ 8 ------- 9 ------ 10 ------ 11 -
  ├── cost: 21.13
@@ -413,7 +413,7 @@ scan s
  ├── columns: b:1
  ├── constraint: /1: [/0 - /99]
  ├── cardinality: [0 - 100]
- ├── stats: [rows=1, distinct(1)=1, null(1)=0, avgsize(1)=1]
+ ├── stats: [rows=1, distinct(1)=1, null(1)=0]
  │   histogram(1)=
  ├── cost: 10.02
  ├── key: (1)
@@ -591,7 +591,7 @@ EXPLAIN (OPT, VERBOSE) SELECT * FROM c WHERE h >= '1988-08-07'
 scan c
  ├── columns: h:1
  ├── constraint: /1: [/'1988-08-07 00:00:00+00:00' - ]
- ├── stats: [rows=24, distinct(1)=24, null(1)=0, avgsize(1)=7]
+ ├── stats: [rows=24, distinct(1)=24, null(1)=0]
  │   histogram(1)=  0               1               5               1               5               1               5               1               5               0
  │                <--- '1988-08-07 00:00:00+00:00' --- '1988-08-07 06:00:00+00:00' --- '1988-08-07 12:00:00+00:00' --- '1988-08-07 18:00:00+00:00' --- '1988-08-08 00:00:00+00:00'
  ├── cost: 39.7