protectedts: v22.1.0-beta: interface conversion: tree.Datum is tree.dNull, not *tree.DBytes

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3173966052/?referrer=webhooks_plugin

Panic message:

panic.go:1038: interface conversion: tree.Datum is tree.dNull, not *tree.DBytes
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| runtime.panicdottypeE
| GOROOT/src/runtime/iface.go:261
| runtime.panicdottypeI
| GOROOT/src/runtime/iface.go:271
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage.rowToRecord
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go:418
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage.(*storage).getRecords
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go:372
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage.(*storage).GetState
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go:324
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).doUpdate.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:245
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:932
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:985
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:931
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:894
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).doUpdate
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:229
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).doSingleFlightUpdate
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:208
| github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
| github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128
| runtime.goexit
| src/runtime/asm_amd64.s:1581
Wraps: (2) interface conversion: tree.Datum is tree.dNull, not *tree.DBytes
Error types: (1) *withstack.withStack (2) *runtime.TypeAssertionError
-- report composition:
*runtime.TypeAssertionError
panic.go:1038: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic
GOROOT/src/runtime/iface.go#L260-L262 in runtime.panicdottypeE
GOROOT/src/runtime/iface.go#L270-L272 in runtime.panicdottypeI
https://github.com/cockroachdb/cockroach/blob/a88367d7bd168a3614b7946cd7b77bb9c6374a30/pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go#L417-L419 in pkg/kv/kvserver/protectedts/ptstorage.rowToRecord
https://github.com/cockroachdb/cockroach/blob/a88367d7bd168a3614b7946cd7b77bb9c6374a30/pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go#L371-L373 in pkg/kv/kvserver/protectedts/ptstorage.(*storage).getRecords
https://github.com/cockroachdb/cockroach/blob/a88367d7bd168a3614b7946cd7b77bb9c6374a30/pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go#L323-L325 in pkg/kv/kvserver/protectedts/ptstorage.(*storage).GetState

cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go

Lines 244 to 246 in a88367d

	}
	if state, err = c.storage.GetState(ctx, txn); err != nil {
	return errors.Wrap(err, "failed to fetch protectedts state")

in pkg/kv/kvserver/protectedts/ptcache.(*Cache).doUpdate.func1

cockroach/pkg/kv/db.go

Lines 931 to 933 in a88367d

	err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error {
	return retryable(ctx, txn)
	})

in pkg/kv.runTxn.func1

cockroach/pkg/kv/txn.go

Lines 984 to 986 in a88367d

	}
	err = fn(ctx, txn)

in pkg/kv.(*Txn).exec

cockroach/pkg/kv/db.go

Lines 930 to 932 in a88367d

	func runTxn(ctx context.Context, txn *Txn, retryable func(context.Context, *Txn) error) error {
	err := txn.exec(ctx, func(ctx context.Context, txn *Txn) error {
	return retryable(ctx, txn)

in pkg/kv.runTxn

cockroach/pkg/kv/db.go

Lines 893 to 895 in a88367d

	txn.SetDebugName("unnamed")
	return runTxn(ctx, txn, retryable)
	}

in pkg/kv.(*DB).Txn

cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go

Lines 228 to 230 in a88367d

	)
	err := c.db.Txn(ctx, func(ctx context.Context, txn *kv.Txn) (err error) {
	// NB: because this is a read-only transaction, the commit will be a no-op;

in pkg/kv/kvserver/protectedts/ptcache.(*Cache).doUpdate

cockroach/pkg/util/stop/stopper.go

Lines 343 to 345 in a88367d

	return f(ctx)
	}

in pkg/util/stop.(*Stopper).RunTaskWithErr

cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go

Lines 207 to 209 in a88367d

	defer cancel()
	return nil, c.stopper.RunTaskWithErr(ctx,
	"refresh-protectedts-cache", c.doUpdate)

in pkg/kv/kvserver/protectedts/ptcache.(*Cache).doSingleFlightUpdate

cockroach/pkg/util/syncutil/singleflight/singleflight.go

Lines 127 to 129 in a88367d

	func (g *Group) doCall(c *call, key string, fn func() (interface{}, error)) {
	c.val, c.err = fn()
	c.wg.Done()

in pkg/util/syncutil/singleflight.(*Group).doCall
src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
GOROOT/src/runtime/iface.go in runtime.panicdottypeE at line 261
GOROOT/src/runtime/iface.go in runtime.panicdottypeI at line 271
pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go in pkg/kv/kvserver/protectedts/ptstorage.rowToRecord at line 418
pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go in pkg/kv/kvserver/protectedts/ptstorage.(*storage).getRecords at line 372
pkg/kv/kvserver/protectedts/ptstorage/pkg/kv/kvserver/protectedts/ptstorage/storage.go in pkg/kv/kvserver/protectedts/ptstorage.(*storage).GetState at line 324
pkg/kv/kvserver/protectedts/ptcache/cache.go in pkg/kv/kvserver/protectedts/ptcache.(*Cache).doUpdate.func1 at line 245
pkg/kv/db.go in pkg/kv.runTxn.func1 at line 932
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 985
pkg/kv/db.go in pkg/kv.runTxn at line 931
pkg/kv/db.go in pkg/kv.(*DB).Txn at line 894
pkg/kv/kvserver/protectedts/ptcache/cache.go in pkg/kv/kvserver/protectedts/ptcache.(*Cache).doUpdate at line 229
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 344
pkg/kv/kvserver/protectedts/ptcache/cache.go in pkg/kv/kvserver/protectedts/ptcache.(*Cache).doSingleFlightUpdate at line 208
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall at line 128
src/runtime/asm_amd64.s in runtime.goexit at line 1581

Tag	Value
Cockroach Release	v22.1.0-beta.1
Cockroach SHA:	a88367d
Platform	darwin amd64
Distribution	CCL
Environment	v22.1.0-beta.1
Command	start-single-node
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-14978

[adityamaru]

Marking this as a GA-blocker, I think it is important to fix and backport.

[blathers-crl]

cc @cockroachdb/bulk-io

[adityamaru]

The problem is well understood, I should have a fix + backport out soon.