Merge #77741 #77808 #78095 #78158 #78223 #78300

77741: ttl: implement TTL scheduled job validation & fixup r=otan a=otan

See individual commits for review.

Resolves #75428

Release justification: high pri addition to new functionality

77808: clisqlshell: add ndjson as --format option r=rafiss a=otan

Resolves #73022

Release justification: low-risk new update

Release note: None

78095: ui: sort items in the sql activity dropdown menu r=gtr a=gtr

Fixes #78081.

Previously, app names in the dropdown menu for the stmts, txns, and
sessions pages were unsorted. This change sorts these app names.

Statements page:
<img width="584" alt="statements-page" src="https://user-images.githubusercontent.com/35943354/159062772-c3a39f9f-50b2-43b8-a635-77506b715456.png">

Transactions page:
<img width="584" alt="transactions-page" src="https://user-images.githubusercontent.com/35943354/159062801-ae172a16-9eee-4b5a-bc9b-b37b2ff6b1e5.png">

Sessions page:
<img width="444" alt="sessions-page" src="https://user-images.githubusercontent.com/35943354/159062817-83d4eb53-9c91-44ae-9788-a3bacf4e469d.png">

Release note (ui change): app names in the dropdown menu are sorted.

78158: cli: debug decode-key to support base64 r=tbg,erikgrinaker a=aliher1911

Previously decode-key supported hex encoded keys. When using json serialization of descriptors and recovery plans you need to decode keys that are base64 encoded.
This patch adds flag to select key encoding format.
The default mode stays the same so any existing tooling willwork as expected.

Release note: None

78223: descmetadata: fix condition for bumping table version r=ajwerner,RichardJCai a=rafiss

This condition was inverted incorrectly. If the cache is on, it's
important to bump the table version so the cache can be invalidated.

Release justification: high priority bugfix to new functionality.

Release note: None

78300: kvserver: don't GC if protected timestamp information isn't available r=arulajmani a=arulajmani

We only want to run GC on a replica that some PTS information (even if
it's stale). We don't want to run GC on a replica if no PTS information
is available however. This can happen if a Replica is being considered
for GC before the initial scan of the KVSubscriber has completed.

This wasn't an issue before this patch for implicit reasons -- this
patch just makes the check explicit and adds a test. Previously, we
wouldn't run GC if no PTS information was available because our lease
was guaranteed to be newer than the empty timestamp.

Release note: None

Co-authored-by: Oliver Tan <[email protected]>
Co-authored-by: Gerardo Torres <[email protected]>
Co-authored-by: Oleg Afanasyev <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
Co-authored-by: arulajmani <[email protected]>

docs/RFCS/20220120_row_level_ttl.md

@@ -13,7 +13,7 @@ This has been a [feature commonly asked for](#20239).
 
 This RFC proposes a CockroachDB level mechanism to support row-level TTL, where
 rows will be deleted after a certain period of time. As a further extension in a
-later release, rows rows will be automatically hidden after they've expired
+later release, rows will be automatically hidden after they've expired
 their TTL and before they've been physically deleted.
 
 The following declarative syntaxes will initialize a table with row-level TTL:
@@ -378,8 +378,8 @@ a few problems:
   process. This adds further complexity to CDC.
 
 As row-level TTL is a "SQL level" feature, it makes sense that something in the
-SQL layer would be most appropriate to handle it. See [comparison
-doc](comparison doc) for other observations.
+SQL layer would be most appropriate to handle it. See [comparison doc](comparison doc)
+for other observations.
 
 ### Alternative TTL columns
 Another proposal for TTL columns was to have two columns:
@@ -465,7 +465,7 @@ has two benefits, both of which are very significant:
 This is predicated on filtering out expired rows working as otherwise users
 could miss entries when querying the secondary index as opposed to the primary.
 
-### Improve the deletion loopl
+### Improve the deletion loop
 We can speed up the deletion by using indexes if one was created on the TTL
 column for the table.
 

docs/generated/sql/functions.md

@@ -3005,6 +3005,8 @@ SELECT * FROM crdb_internal.check_consistency(true, ‘\x02’, ‘\x04’)</p>
 </span></td></tr>
 <tr><td><a name="crdb_internal.range_stats"></a><code>crdb_internal.range_stats(key: <a href="bytes.html">bytes</a>) &rarr; jsonb</code></td><td><span class="funcdesc"><p>This function is used to retrieve range statistics information as a JSON object.</p>
 </span></td></tr>
+<tr><td><a name="crdb_internal.repair_ttl_table_scheduled_job"></a><code>crdb_internal.repair_ttl_table_scheduled_job(oid: oid) &rarr; void</code></td><td><span class="funcdesc"><p>Repairs the scheduled job for a TTL table if it is missing.</p>
+</span></td></tr>
 <tr><td><a name="crdb_internal.reset_index_usage_stats"></a><code>crdb_internal.reset_index_usage_stats() &rarr; <a href="bool.html">bool</a></code></td><td><span class="funcdesc"><p>This function is used to clear the collected index usage statistics.</p>
 </span></td></tr>
 <tr><td><a name="crdb_internal.reset_sql_stats"></a><code>crdb_internal.reset_sql_stats() &rarr; <a href="bool.html">bool</a></code></td><td><span class="funcdesc"><p>This function is used to clear the collected SQL statistics.</p>
@@ -3034,6 +3036,8 @@ table. Returns an error if validation fails.</p>
 </span></td></tr>
 <tr><td><a name="crdb_internal.validate_session_revival_token"></a><code>crdb_internal.validate_session_revival_token(token: <a href="bytes.html">bytes</a>) &rarr; <a href="bool.html">bool</a></code></td><td><span class="funcdesc"><p>Validate a token that was created by create_session_revival_token. Intended for testing.</p>
 </span></td></tr>
+<tr><td><a name="crdb_internal.validate_ttl_scheduled_jobs"></a><code>crdb_internal.validate_ttl_scheduled_jobs() &rarr; void</code></td><td><span class="funcdesc"><p>Validate all TTL tables have a valid scheduled job attached.</p>
+</span></td></tr>
 <tr><td><a name="crdb_internal.void_func"></a><code>crdb_internal.void_func() &rarr; void</code></td><td><span class="funcdesc"><p>This function is used only by CockroachDB’s developers for testing purposes.</p>
 </span></td></tr>
 <tr><td><a name="current_database"></a><code>current_database() &rarr; <a href="string.html">string</a></code></td><td><span class="funcdesc"><p>Returns the current database.</p>

pkg/BUILD.bazel

@@ -343,6 +343,7 @@ ALL_TESTS = [
     "//pkg/sql/sem/tree/eval_test:eval_test_test",
     "//pkg/sql/sem/tree:tree_test",
     "//pkg/sql/sessiondata:sessiondata_test",
+    "//pkg/sql/sessioninit:sessioninit_test",
     "//pkg/sql/span:span_test",
     "//pkg/sql/sqlinstance/instanceprovider:instanceprovider_test",
     "//pkg/sql/sqlinstance/instancestorage:instancestorage_test",

pkg/ccl/backupccl/schedule_exec.go

@@ -17,6 +17,7 @@ import (
 	"github.com/cockroachdb/cockroach/pkg/kv"
 	"github.com/cockroachdb/cockroach/pkg/scheduledjobs"
 	"github.com/cockroachdb/cockroach/pkg/sql"
+	"github.com/cockroachdb/cockroach/pkg/sql/catalog/descs"
 	"github.com/cockroachdb/cockroach/pkg/sql/parser"
 	"github.com/cockroachdb/cockroach/pkg/sql/sem/tree"
 	"github.com/cockroachdb/cockroach/pkg/sql/sqlutil"
@@ -448,6 +449,7 @@ func (e *scheduledBackupExecutor) OnDrop(
 	env scheduledjobs.JobSchedulerEnv,
 	sj *jobs.ScheduledJob,
 	txn *kv.Txn,
+	descsCol *descs.Collection,
 ) error {
 	args := &ScheduledBackupExecutionArgs{}
 	if err := pbtypes.UnmarshalAny(sj.ExecutionArgs().Args, args); err != nil {

pkg/ccl/backupccl/testdata/backup-restore/row_level_ttl

@@ -28,6 +28,10 @@ exec-sql
 RESTORE FROM LATEST IN 'nodelocal://0/full_cluster_backup/'
 ----
 
+exec-sql
+SELECT crdb_internal.validate_ttl_scheduled_jobs()
+----
+
 query-sql
 SELECT create_statement FROM [SHOW CREATE TABLE d.public.t]
 ----
@@ -57,6 +61,10 @@ exec-sql
 RESTORE DATABASE d FROM LATEST IN 'nodelocal://0/database_backup/'
 ----
 
+exec-sql
+SELECT crdb_internal.validate_ttl_scheduled_jobs()
+----
+
 query-sql
 SELECT create_statement FROM [SHOW CREATE TABLE d.public.t]
 ----
@@ -90,6 +98,10 @@ exec-sql
 RESTORE TABLE d.public.t FROM LATEST IN 'nodelocal://0/database_backup/'
 ----
 
+exec-sql
+SELECT crdb_internal.validate_ttl_scheduled_jobs()
+----
+
 query-sql
 SELECT create_statement FROM [SHOW CREATE TABLE d.public.t]
 ----

pkg/cli/clisqlexec/format_table.go

@@ -12,6 +12,7 @@ package clisqlexec
 
 import (
 	"bytes"
+	"encoding/json"
 	"fmt"
 	"html"
 	"io"
@@ -584,6 +585,45 @@ func (p *recordReporter) doneRows(w io.Writer, seenRows int) error {
 func (p *recordReporter) beforeFirstRow(_ io.Writer, _ RowStrIter) error { return nil }
 func (p *recordReporter) doneNoRows(_ io.Writer) error                   { return nil }
 
+type ndjsonReporter struct {
+	cols []string
+}
+
+func (n *ndjsonReporter) describe(w io.Writer, cols []string) error {
+	n.cols = cols
+	return nil
+}
+
+func (n *ndjsonReporter) beforeFirstRow(w io.Writer, allRows RowStrIter) error {
+	return nil
+}
+
+func (n *ndjsonReporter) iter(w, ew io.Writer, rowIdx int, row []string) error {
+	retMap := make(map[string]string, len(row))
+	for i := range row {
+		retMap[n.cols[i]] = row[i]
+	}
+	out, err := json.Marshal(retMap)
+	if err != nil {
+		return err
+	}
+	if _, err := ew.Write(out); err != nil {
+		return err
+	}
+	if _, err := ew.Write([]byte("\n")); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (n *ndjsonReporter) doneRows(w io.Writer, seenRows int) error {
+	return nil
+}
+
+func (n *ndjsonReporter) doneNoRows(w io.Writer) error {
+	return nil
+}
+
 type sqlReporter struct {
 	noColumns bool
 }
@@ -644,6 +684,9 @@ func (sqlExecCtx *Context) makeReporter(w io.Writer) (rowReporter, func(), error
 		reporter, cleanup := makeCSVReporter(w, sqlExecCtx.TableDisplayFormat)
 		return reporter, cleanup, nil
 
+	case TableDisplayNDJSON:
+		return &ndjsonReporter{}, nil, nil
+
 	case TableDisplayRaw:
 		return &rawReporter{}, nil, nil
 

pkg/cli/clisqlexec/format_table_test.go

@@ -203,6 +203,7 @@ func Example_sql_empty_table() {
 	// -----
 	// (0 rows)
 	// sql --format=records -e select * from t.norows
+	// sql --format=ndjson -e select * from t.norows
 	// sql --format=sql -e select * from t.norows
 	// CREATE TABLE results (
 	//   x STRING
@@ -237,6 +238,10 @@ func Example_sql_empty_table() {
 	// (3 rows)
 	// sql --format=records -e select * from t.nocols
 	// (3 rows)
+	// sql --format=ndjson -e select * from t.nocols
+	// {}
+	// {}
+	// {}
 	// sql --format=sql -e select * from t.nocols
 	// CREATE TABLE results (
 	// );
@@ -278,6 +283,7 @@ func Example_sql_empty_table() {
 	// (0 rows)
 	// sql --format=records -e select * from t.nocolsnorows
 	// (0 rows)
+	// sql --format=ndjson -e select * from t.nocolsnorows
 	// sql --format=sql -e select * from t.nocolsnorows
 	// CREATE TABLE results (
 	// );
@@ -593,6 +599,16 @@ func Example_sql_table() {
 	// s | a	b	c+
 	//   | 12	123123213	12313
 	// d | tabs
+	// sql --format=ndjson -e select * from t.t
+	// {"d":"printable ASCII","s":"foo"}
+	// {"d":"printable ASCII with quotes","s":"\"foo"}
+	// {"d":"printable ASCII with backslash","s":"\\foo"}
+	// {"d":"non-printable ASCII","s":"foo\nbar"}
+	// {"d":"printable UTF8","s":"κόσμε"}
+	// {"d":"printable UTF8 using escapes","s":"ñ"}
+	// {"d":"non-printable UTF8 string","s":"\\x01"}
+	// {"d":"UTF8 string with RTL char","s":"܈85"}
+	// {"d":"tabs","s":"a\tb\tc\n12\t123123213\t12313"}
 	// sql --format=sql -e select * from t.t
 	// CREATE TABLE results (
 	//   s STRING,

pkg/cli/clisqlexec/table_display_format.go

@@ -35,6 +35,9 @@ const (
 	// TableDisplayRecords is a record-oriented format. It is somewhat
 	// compatible with 'psql' "expanded display" mode.
 	TableDisplayRecords
+	// TableDisplayNDJSON reports results in an nd-json format
+	// (https://github.com/ndjson/ndjson-spec).
+	TableDisplayNDJSON
 	// TableDisplaySQL reports results using SQL statements that mimic
 	// the creation of a SQL table containing the result values.
 	TableDisplaySQL
@@ -71,6 +74,8 @@ func (f *TableDisplayFormat) String() string {
 		return "records"
 	case TableDisplaySQL:
 		return "sql"
+	case TableDisplayNDJSON:
+		return "ndjson"
 	case TableDisplayHTML:
 		return "html"
 	case TableDisplayRawHTML:
@@ -94,6 +99,8 @@ func (f *TableDisplayFormat) Set(s string) error {
 		*f = TableDisplayRecords
 	case "sql":
 		*f = TableDisplaySQL
+	case "ndjson":
+		*f = TableDisplayNDJSON
 	case "html":
 		*f = TableDisplayHTML
 	case "rawhtml":

pkg/cli/clisqlshell/sql.go

@@ -359,7 +359,7 @@ var options = map[string]struct {
 		display: func(c *cliState) string { return strconv.Itoa(c.sqlExecCtx.TableBorderMode) },
 	},
 	`display_format`: {
-		description:               "the output format for tabular data (table, csv, tsv, html, sql, records, raw)",
+		description:               "the output format for tabular data (table, csv, tsv, html, ndjson, sql, records, raw)",
 		isBoolean:                 false,
 		validDuringMultilineEntry: true,
 		set: func(c *cliState, val string) error {

pkg/cli/debug.go

@@ -129,6 +129,40 @@ func parsePositiveDuration(arg string) (time.Duration, error) {
 	return duration, nil
 }
 
+type keyFormat int
+
+const (
+	hexKey = iota
+	base64Key
+)
+
+func (f *keyFormat) Set(value string) error {
+	switch value {
+	case "hex":
+		*f = hexKey
+	case "base64":
+		*f = base64Key
+	default:
+		return errors.Errorf("unsupported format %s", value)
+	}
+	return nil
+}
+
+func (f *keyFormat) String() string {
+	switch *f {
+	case hexKey:
+		return "hex"
+	case base64Key:
+		return "base64"
+	default:
+		panic(errors.AssertionFailedf("invalid format value %d", *f))
+	}
+}
+
+func (f *keyFormat) Type() string {
+	return "hex|base64"
+}
+
 // OpenEngineOptions tunes the behavior of OpenEngine.
 type OpenEngineOptions struct {
 	ReadOnly                    bool
@@ -528,19 +562,34 @@ func runDebugRangeDescriptors(cmd *cobra.Command, args []string) error {
 	})
 }
 
+var decodeKeyOptions struct {
+	encoding keyFormat
+}
+
 var debugDecodeKeyCmd = &cobra.Command{
 	Use:   "decode-key",
 	Short: "decode <key>",
 	Long: `
-Decode a hexadecimal-encoded key and pretty-print it. For example:
+Decode encoded keys provided as command arguments and pretty-print them.
+Key encoding type could be changed using encoding flag.
+For example:
 
-	$ decode-key BB89F902ADB43000151C2D1ED07DE6C009
+	$ cockroach debug decode-key BB89F902ADB43000151C2D1ED07DE6C009
 	/Table/51/1/44938288/1521140384.514565824,0
 `,
 	Args: cobra.ArbitraryArgs,
 	RunE: func(cmd *cobra.Command, args []string) error {
 		for _, arg := range args {
-			b, err := gohex.DecodeString(arg)
+			var b []byte
+			var err error
+			switch decodeKeyOptions.encoding {
+			case hexKey:
+				b, err = gohex.DecodeString(arg)
+			case base64Key:
+				b, err = base64.StdEncoding.DecodeString(arg)
+			default:
+				return errors.Errorf("unsupported key format %d", decodeKeyOptions.encoding)
+			}
 			if err != nil {
 				return err
 			}
@@ -1697,6 +1746,9 @@ func init() {
 	f.Var(&debugMergeLogsOpts.useColor, "color",
 		"force use of TTY escape codes to colorize the output")
 
+	f = debugDecodeKeyCmd.Flags()
+	f.Var(&decodeKeyOptions.encoding, "encoding", "key argument encoding")
+
 	f = debugDecodeProtoCmd.Flags()
 	f.StringVar(&debugDecodeProtoName, "schema", "cockroach.sql.sqlbase.Descriptor",
 		"fully qualified name of the proto to decode")

pkg/cli/interactive_tests/test_pretty.tcl

@@ -19,6 +19,13 @@ eexpect "1 row"
 eexpect ":/# "
 end_test
 
+
+start_test "Check that tables are pretty-printed when output is not a terminal but --format=ndjson is specified."
+send "echo 'select 1 as WOO;' | $argv sql --format=ndjson | cat\r"
+eexpect "{\"woo\":\"1\"}"
+eexpect ":/# "
+end_test
+
 start_test "Check that tables are pretty-printed when input is not a terminal and --format=table is not specified, but output is a terminal."
 send "echo begin; echo 'select 1 as WOO;' | $argv sql\r"
 eexpect "woo"

pkg/jobs/BUILD.bazel

@@ -40,6 +40,7 @@ go_library(
         "//pkg/sql/catalog",
         "//pkg/sql/catalog/colinfo",
         "//pkg/sql/catalog/descpb",
+        "//pkg/sql/catalog/descs",
         "//pkg/sql/protoreflect",
         "//pkg/sql/sem/builtins",
         "//pkg/sql/sem/tree",

pkg/jobs/scheduled_job_executor.go

@@ -16,6 +16,7 @@ import (
 	"github.com/cockroachdb/cockroach/pkg/jobs/jobspb"
 	"github.com/cockroachdb/cockroach/pkg/kv"
 	"github.com/cockroachdb/cockroach/pkg/scheduledjobs"
+	"github.com/cockroachdb/cockroach/pkg/sql/catalog/descs"
 	"github.com/cockroachdb/cockroach/pkg/sql/sqlutil"
 	"github.com/cockroachdb/cockroach/pkg/util/metric"
 	"github.com/cockroachdb/cockroach/pkg/util/syncutil"
@@ -63,8 +64,14 @@ type ScheduledJobExecutor interface {
 type ScheduledJobController interface {
 	// OnDrop runs before the passed in `schedule` is dropped as part of a `DROP
 	// SCHEDULE` query.
-	OnDrop(ctx context.Context, scheduleControllerEnv scheduledjobs.ScheduleControllerEnv,
-		env scheduledjobs.JobSchedulerEnv, schedule *ScheduledJob, txn *kv.Txn) error
+	OnDrop(
+		ctx context.Context,
+		scheduleControllerEnv scheduledjobs.ScheduleControllerEnv,
+		env scheduledjobs.JobSchedulerEnv,
+		schedule *ScheduledJob,
+		txn *kv.Txn,
+		descsCol *descs.Collection,
+	) error
 }
 
 // ScheduledJobExecutorFactory is a callback to create a ScheduledJobExecutor.