sql: SELECT * FROM crdb_internal.cluster_inflight_traces WHERE trace_id = 1; panics for shared-process tenants

[ecwall]

Easy to reproduce in ./cockroach demo.

Run as the app tenant:

[email protected]:26257/demoapp/movr> SELECT * FROM crdb_internal.cluster_inflight_traces WHERE trace_id = 1;                                                                                                                             
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | runtime.panicmem
*   | 	GOROOT/src/runtime/panic.go:260
*   | runtime.sigpanic
*   | 	GOROOT/src/runtime/signal_unix.go:839
*   | github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func9
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:918
*   | github.com/cockroachdb/cockroach/pkg/util/tracing/collector.(*TraceCollector).StartIter
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/collector/collector.go:96
*   | github.com/cockroachdb/cockroach/pkg/sql.glob..func44
*   | 	github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:1947
*   | github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1
*   | 	github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:823
*   | github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3
*   | 	github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:127
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1047271be]

goroutine 5785 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc004b647f8?, {0x1072e86c0, 0xc001420a20})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x1052ef000, 0x10a964500})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func9({0x1072e86c0?, 0xc001420a20?})
	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:918 +0x3e
github.com/cockroachdb/cockroach/pkg/util/tracing/collector.(*TraceCollector).StartIter(0xc00488e9f0, {0x1072e86c0, 0xc001420a20}, 0x1)
	github.com/cockroachdb/cockroach/pkg/util/tracing/collector/collector.go:96 +0x79
github.com/cockroachdb/cockroach/pkg/sql.glob..func44({0x1072e86c0, 0xc001420a20}, {0x107357270?, 0xc00141a338?}, 0xc00141a338?, {0x0?, 0x0?}, 0xc002847440)
	github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:1947 +0xa5
github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1({0x1072e86c0, 0xc001420a20}, {0x1072a4f80, 0xc003b05de0})
	github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:823 +0x574
github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3({0x1072e86c0, 0xc001420a20})
	github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:127 +0x114
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b

System tenant output for reference:

[email protected]:26257/system/defaultdb> SELECT * FROM crdb_internal.cluster_inflight_traces WHERE trace_id = 1;                                                                                                                         
  trace_id | node_id | root_op_name | trace_str | jaeger_json
-----------+---------+--------------+-----------+--------------
(0 rows)

Time: 4ms total (execution 3ms / network 0ms)

As of commit 9e84bfe

Jira issue: CRDB-29440
Epic: CRDB-26691