Merge #68102 #68390 #68619 #68621 #68663 #68806

68102: storage/storageccl: Add an option to break export mid key r=pbardea,sumeerbhola a=aliher1911

Previously when exporting ranges with very large number of versions
export could hit the situation where single key export would exceed
maximum export byte limit. To resolve this, safety threshold has to
be raised which could cause nodes going out of memory.

To address this, this patch adds an ability to stop and resume on
the arbitrary key timestamp so that it could be stopped mid key and
resumed later.

Release note: None

Fixes #68231

68390: ui: rebuild the databases pages r=matthewtodd a=matthewtodd

Previously, the databases pages featured an older, outdated design. This
change aligns their UX with that of the other modern pages in the DB
console, statements, transactions, and sessions.

This is a first pass at the job. We will return to layer on more
features, including search, and more data, including node & region
information.

Addresses #65737.

Before | After
--- | ---
<img width="1372" alt="Screen Shot 2021-08-05 at 4 54 31 PM" src="https://user-images.githubusercontent.com/5261/128420405-7a096c2f-32a1-42d9-a8f0-186f699fb612.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 12 PM" src="https://user-images.githubusercontent.com/5261/128070069-28d0f461-69dd-47cb-a04b-bc6cd0126ca4.png">
<img width="1372" alt="Screen Shot 2021-08-05 at 4 54 57 PM" src="https://user-images.githubusercontent.com/5261/128420491-71d03570-1c0d-413b-ab40-09b45cf614ee.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 21 PM" src="https://user-images.githubusercontent.com/5261/128070090-bd62d125-a1b2-4c83-b0a5-33d0723849eb.png">
<img width="1372" alt="Screen Shot 2021-08-05 at 5 03 37 PM" src="https://user-images.githubusercontent.com/5261/128420959-338ea79b-abf0-4773-91bf-cecac64fec5f.png"> | (This overview of database grants no longer exists.)
(This rolled-up view of table grants did not previously exist.) | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 27 PM" src="https://user-images.githubusercontent.com/5261/128070100-1305c94f-c017-40a5-ba2d-f19fd61bdecf.png">
<img width="1372" alt="Screen Shot 2021-08-05 at 4 55 21 PM" src="https://user-images.githubusercontent.com/5261/128420596-2d9f04ad-718c-4dce-b045-b5647f718339.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 35 PM" src="https://user-images.githubusercontent.com/5261/128070103-831fd5c3-38da-4864-9d82-9091f3209606.png">
<img width="1372" alt="Screen Shot 2021-08-05 at 4 55 28 PM" src="https://user-images.githubusercontent.com/5261/128420658-4042c631-b283-4b0c-ac4a-3235bd3bcf39.png"> | <img width="1372" alt="Screen Shot 2021-08-03 at 2 50 41 PM" src="https://user-images.githubusercontent.com/5261/128070113-73528f2c-56bf-4fcf-a8d8-fcc3f100695f.png">

Release note (ui change): The databases pages in the DB console were
updated to bring them into alignment with our modern UX.

68619: storage: add file storing min version needed for backwards compatibility r=jbowens a=andyyang890

A new STORAGE_MIN_VERSION file containing the minimum version that the
storage engine needs to maintain backwards compatibility with will be
added to the directory for each storage engine. This will be used in
the encryption-at-rest registry migration.

Release note: None

68621: dev: implement `dev generate docs` r=rail a=rickystewart

Do some refactoring so that we don't have to repeat code in a bunch of
places. The implementation is pretty straightforward and just involves
building the docs then copying them over to the workspace.

Closes #68563.

Release note: None

68663: dev: hoist generated code out of sandbox into workspace r=rail a=rickystewart

(Only the most recent commit applies for this review.)

One might be expected that this command be implemented as
`dev generate protobuf` or `dev generate execgen` or similar, but that's
not really practical for a couple reasons:

1. Building all the protobuf in-tree isn't really possible on its own:
   see #58018 (comment).
2. For non-proto generated code, one can imagine that we do a bunch of
   `bazel query`s to list all the generated files in tree and to find
   which files are generated. Given that there are hundreds of generated
   files in tree, and a single `bazel query` can take ~1s, this is not
   tractable.

The implementation here is ad-hoc and a bit hacky. Basically we search
`bazel-bin` for files ending in `.go` and apply some heuristics to
figure out whether they should be copied and where those files should
go.

This is gated behind a feature flag for now because actually copying
these files into the workspace right now dirties a ton of files. (The
diffs are benign -- mostly comments.) We can flip this option to be on
by default when it's safe (probably after the `Makefile` is deleted).

Closes #68565.

Release note: None

68806: bazel: set `--include_source_info` when generating protobuf code r=rail a=rickystewart

This flag [instructs Bazel](bazelbuild/rules_proto#56 (comment))
to set a [command-line flag](protocolbuffers/protobuf#7623 (comment))
when invoking `protoc` that causes the generated proto descriptor sets
to contain extra info:

```
  --include_source_info       When using --descriptor_set_out, do not strip
                              SourceCodeInfo from the FileDescriptorProto.
                              This results in vastly larger descriptors that
                              include information about the original
                              location of each decl in the source file as
                              well as surrounding comments.
```

Setting this solves two problems:

1. We need the descriptor sets to have comments for #65814.
2. Without this change, generated `.pb.go` files from the sandbox won't
   contain comments. This makes the files more difficult to read and
   dirties the files in your checkout if you copy those `.pb.go` files
   to your workspace.

Also delete an unnecessary `--symlink_prefix=_bazel/` from the `test`
configuration (it's inherited from the `build` configuration so it's
redundant).

Release note: None

Co-authored-by: Oleg Afanasyev <[email protected]>
Co-authored-by: Matthew Todd <[email protected]>
Co-authored-by: Andy Yang <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>

.bazelrc

@@ -1,6 +1,6 @@
-build --symlink_prefix=_bazel/ --ui_event_filters=-DEBUG --define gotags=bazel,crdb_test_off,gss
+build --symlink_prefix=_bazel/ --ui_event_filters=-DEBUG --define gotags=bazel,crdb_test_off,gss --experimental_proto_descriptor_sets_include_source_info
 test --config=test
-build:test --symlink_prefix=_bazel/ --define gotags=bazel,crdb_test,gss --test_env=TZ=
+build:test --define gotags=bazel,crdb_test,gss --test_env=TZ=
 query --ui_event_filters=-DEBUG
 
 try-import %workspace%/.bazelrc.user

build/teamcity/cockroach/ci/builds/build_impl.sh

@@ -14,7 +14,7 @@ CONFIG="$1"
 DOC_TARGETS=
 if [ "$CONFIG" == "crosslinux" ]
 then
-   DOC_TARGETS="//docs/generated:gen-logging-md //docs/generated:gen-logsinks-md //docs/generated:gen-eventlog-md //docs/generated:gen-logformats-md //docs/generated/settings:settings //docs/generated/settings:settings_for_tenants //docs/generated/sql //docs/generated/sql/bnf"
+   DOC_TARGETS=$(grep '^//' docs/generated/bazel_targets.txt)
 fi
 
 bazel build //pkg/cmd/bazci --config=ci

docs/generated/bazel_targets.txt

@@ -0,0 +1,11 @@
+This file lists all the targets you need to build to build all the generated
+documentation. Lines not beginning with // should be ignored.
+
+//docs/generated:gen-logging-md
+//docs/generated:gen-logsinks-md
+//docs/generated:gen-eventlog-md
+//docs/generated:gen-logformats-md
+//docs/generated/settings
+//docs/generated/settings:settings_for_tenants
+//docs/generated/sql
+//docs/generated/sql/bnf

pkg/BUILD.bazel

@@ -7,6 +7,7 @@ ALL_TESTS = [
     "//pkg/bench/rttanalysis:rttanalysis_test",
     "//pkg/bench:bench_test",
     "//pkg/blobs:blobs_test",
+    "//pkg/build/util:util_test",
     "//pkg/ccl/backupccl/backupresolver:backupresolver_test",
     "//pkg/ccl/backupccl:backupccl_test",
     "//pkg/ccl/baseccl:baseccl_test",

pkg/build/util/BUILD.bazel

@@ -0,0 +1,15 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "util",
+    srcs = ["util.go"],
+    importpath = "github.com/cockroachdb/cockroach/pkg/build/util",
+    visibility = ["//visibility:public"],
+)
+
+go_test(
+    name = "util_test",
+    srcs = ["util_test.go"],
+    embed = [":util"],
+    deps = ["@com_github_stretchr_testify//require"],
+)

pkg/build/util/util.go

@@ -0,0 +1,75 @@
+// Copyright 2015 The Cockroach Authors.
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+//
+// This file contains assorted utilities for working with Bazel internals.
+
+package util
+
+import (
+	"fmt"
+	"path/filepath"
+	"regexp"
+	"strings"
+)
+
+// OutputOfBinaryRule returns the path of the binary produced by the
+// given build target, relative to bazel-bin. That is,
+//    filepath.Join(bazelBin, OutputOfBinaryRule(target)) is the absolute
+// path to the build binary for the target.
+func OutputOfBinaryRule(target string) string {
+	colon := strings.Index(target, ":")
+	var bin string
+	if colon >= 0 {
+		bin = target[colon+1:]
+	} else {
+		bin = target[strings.LastIndex(target, "/")+1:]
+	}
+	var head string
+	if strings.HasPrefix(target, "@") {
+		doubleSlash := strings.Index(target, "//")
+		head = filepath.Join("external", target[1:doubleSlash])
+	} else if colon >= 0 {
+		head = strings.TrimPrefix(target[:colon], "//")
+	} else {
+		head = strings.TrimPrefix(target, "//")
+	}
+	return filepath.Join(head, bin+"_", bin)
+}
+
+// OutputsOfGenrule lists the outputs of a genrule. The first argument
+// is the name of the target (e.g. //docs/generated/sql), and the second
+// should be the output of `bazel query --output=xml $TARGET`. The
+// returned slice is the list of outputs, all of which are relative
+// paths atop `bazel-bin` as in `OutputOfBinaryRule`.
+func OutputsOfGenrule(target, xmlQueryOutput string) ([]string, error) {
+	// XML parsing is a bit heavyweight here, and encoding/xml
+	// refuses to parse the query output since it's XML 1.1 instead
+	// of 1.0. Have fun with regexes instead.
+	colon := strings.LastIndex(target, ":")
+	if colon < 0 {
+		colon = len(target)
+	}
+	regexStr := fmt.Sprintf("^<rule-output name=\"%s:(?P<Filename>.*)\"/>$", regexp.QuoteMeta(target[:colon]))
+	re, err := regexp.Compile(regexStr)
+	if err != nil {
+		return nil, err
+	}
+	var ret []string
+	for _, line := range strings.Split(xmlQueryOutput, "\n") {
+		line = strings.TrimSpace(line)
+		submatch := re.FindStringSubmatch(line)
+		if submatch == nil {
+			continue
+		}
+		relBinPath := filepath.Join(strings.TrimPrefix(target[:colon], "//"), submatch[1])
+		ret = append(ret, relBinPath)
+	}
+	return ret, nil
+}

pkg/build/util/util_test.go

@@ -0,0 +1,62 @@
+// Copyright 2021 The Cockroach Authors.
+//
+// Use of this software is governed by the Business Source License
+// included in the file licenses/BSL.txt.
+//
+// As of the Change Date specified in that file, in accordance with
+// the Business Source License, use of this software will be governed
+// by the Apache License, Version 2.0, included in the file
+// licenses/APL.txt.
+
+package util
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestOutputOfBinaryRule(t *testing.T) {
+	require.Equal(t, OutputOfBinaryRule("//pkg/cmd/cockroach-short"), "pkg/cmd/cockroach-short/cockroach-short_/cockroach-short")
+	require.Equal(t, OutputOfBinaryRule("//pkg/cmd/cockroach-short:cockroach-short"), "pkg/cmd/cockroach-short/cockroach-short_/cockroach-short")
+	require.Equal(t, OutputOfBinaryRule("pkg/cmd/cockroach-short"), "pkg/cmd/cockroach-short/cockroach-short_/cockroach-short")
+
+	require.Equal(t, OutputOfBinaryRule("@com_github_cockroachdb_stress//:stress"), "external/com_github_cockroachdb_stress/stress_/stress")
+}
+
+func TestOutputsOfGenrule(t *testing.T) {
+	xmlQueryOutput := `<?xml version="1.1" encoding="UTF-8" standalone="no"?>
+<query version="2">
+    <rule class="genrule" location="/Users/ricky/go/src/github.com/cockroachdb/cockroach/docs/generated/sql/BUILD.bazel:1:8" name="//docs/generated/sql:sql">
+        <string name="name" value="sql"/>
+        <list name="exec_tools">
+            <label value="//pkg/cmd/docgen:docgen"/>
+        </list>
+        <list name="outs">
+            <output value="//docs/generated/sql:aggregates.md"/>
+            <output value="//docs/generated/sql:functions.md"/>
+            <output value="//docs/generated/sql:operators.md"/>
+            <output value="//docs/generated/sql:window_functions.md"/>
+        </list>
+        <string name="cmd" value="&#10;$(location //pkg/cmd/docgen) functions $(RULEDIR) --quiet&#10;"/>
+        <rule-input name="//pkg/cmd/docgen:docgen"/>
+        <rule-input name="@bazel_tools//tools/genrule:genrule-setup.sh"/>
+        <rule-output name="//docs/generated/sql:aggregates.md"/>
+        <rule-output name="//docs/generated/sql:functions.md"/>
+        <rule-output name="//docs/generated/sql:operators.md"/>
+        <rule-output name="//docs/generated/sql:window_functions.md"/>
+    </rule>
+</query>`
+	expected := []string{
+		"docs/generated/sql/aggregates.md",
+		"docs/generated/sql/functions.md",
+		"docs/generated/sql/operators.md",
+		"docs/generated/sql/window_functions.md",
+	}
+	out, err := OutputsOfGenrule("//docs/generated/sql:sql", xmlQueryOutput)
+	require.NoError(t, err)
+	require.Equal(t, out, expected)
+	out, err = OutputsOfGenrule("//docs/generated/sql", xmlQueryOutput)
+	require.NoError(t, err)
+	require.Equal(t, out, expected)
+}

pkg/ccl/storageccl/export.go

@@ -23,6 +23,7 @@ import (
 	"github.com/cockroachdb/cockroach/pkg/settings"
 	"github.com/cockroachdb/cockroach/pkg/sql/sem/builtins"
 	"github.com/cockroachdb/cockroach/pkg/storage"
+	"github.com/cockroachdb/cockroach/pkg/util/hlc"
 	"github.com/cockroachdb/cockroach/pkg/util/log"
 	"github.com/cockroachdb/cockroach/pkg/util/retry"
 	"github.com/cockroachdb/cockroach/pkg/util/tracing"
@@ -177,11 +178,19 @@ func evalExport(
 
 	// Time-bound iterators only make sense to use if the start time is set.
 	useTBI := args.EnableTimeBoundIteratorOptimization && !args.StartTime.IsEmpty()
+	// Only use resume timestamp if splitting mid key is enabled.
+	resumeKeyTS := hlc.Timestamp{}
+	if args.SplitMidKey {
+		if !args.ReturnSST {
+			return result.Result{}, errors.New("SplitMidKey could only be used with ReturnSST option")
+		}
+		resumeKeyTS = args.ResumeKeyTS
+	}
 	var curSizeOfExportedSSTs int64
 	for start := args.Key; start != nil; {
 		destFile := &storage.MemFile{}
-		summary, resume, err := reader.ExportMVCCToSst(ctx, start, args.EndKey, args.StartTime,
-			h.Timestamp, exportAllRevisions, targetSize, maxSize, useTBI, destFile)
+		summary, resume, resumeTS, err := reader.ExportMVCCToSst(ctx, start, args.EndKey, args.StartTime,
+			h.Timestamp, resumeKeyTS, exportAllRevisions, targetSize, maxSize, args.SplitMidKey, useTBI, destFile)
 		if err != nil {
 			if errors.HasType(err, (*storage.ExceedMaxSizeError)(nil)) {
 				err = errors.WithHintf(err,
@@ -206,6 +215,7 @@ func evalExport(
 		}
 		exported := roachpb.ExportResponse_File{
 			Span:       span,
+			EndKeyTS:   resumeTS,
 			Exported:   summary,
 			LocalityKV: localityKV,
 		}
@@ -249,6 +259,7 @@ func evalExport(
 		}
 		reply.Files = append(reply.Files, exported)
 		start = resume
+		resumeKeyTS = resumeTS
 
 		// If we are not returning the SSTs to the processor, there is no need to
 		// paginate the ExportRequest since the reply size will not grow large