kv/kvserver: TestStoreRangeSplitAtTablePrefix failed

[cockroach-teamcity]

(kv/kvserver).TestStoreRangeSplitAtTablePrefix failed on master@7b0ccdda99b81613e70f421c9374483c3feddff3:

Fatal error:

panic: unexpected descriptor type &descpb.Descriptor{Union:descpb.isDescriptor_Union(nil)} [recovered]
	panic: unexpected descriptor type &descpb.Descriptor{Union:descpb.isDescriptor_Union(nil)}

Stack:

goroutine 898445 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0096a3c20, 0x523bea0, 0xc00019e010)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:207 +0x126
panic(0x4140b20, 0xc0035495f0)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/sql/catalog/systemschema.ShouldSplitAtDesc(0xc0011c0798, 0xc004640a78)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/systemschema/system.go:48 +0x267
github.com/cockroachdb/cockroach/pkg/config.(*SystemConfig).shouldSplitOnSystemTenantObject(0xc0057945f0, 0x32, 0x8)
	/go/src/github.com/cockroachdb/cockroach/pkg/config/system.go:700 +0x166
github.com/cockroachdb/cockroach/pkg/config.(*SystemConfig).systemTenantTableBoundarySplitKey.func1(0x3300000032, 0x0, 0x71ca380, 0x6)
	/go/src/github.com/cockroachdb/cockroach/pkg/config/system.go:533 +0x585
github.com/cockroachdb/cockroach/pkg/config.(*SystemConfig).systemTenantTableBoundarySplitKey(0xc0057945f0, 0x523bf20, 0xc0039581e0, 0xc006e7edf8, 0x1, 0x8, 0xc006e7ee00, 0x1, 0x8, 0x53b14e, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/config/system.go:585 +0x233
github.com/cockroachdb/cockroach/pkg/config.(*SystemConfig).ComputeSplitKey(0xc0057945f0, 0x523bf20, 0xc0039581e0, 0xc006e7edf8, 0x1, 0x8, 0xc006e7ee00, 0x1, 0x8, 0xc00019e010, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/config/system.go:491 +0x165
github.com/cockroachdb/cockroach/pkg/config.(*SystemConfig).NeedsSplit(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/config/system.go:676
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).requiresSplit(0xc003afa780, 0x523bf20, 0xc0039581e0, 0xc0057945f0, 0x52af200, 0xc009b43800, 0x523be60)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:674 +0xbb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).maybeAdd(0xc003afa780, 0x523bea0, 0xc00019e010, 0x52af200, 0xc009b43800, 0x165af24888277770, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:634 +0x739
github.com/cockroachdb/cockroach/pkg/kv/kvserver.baseQueueHelper.MaybeAdd(0xc003afa780, 0x523bea0, 0xc00019e010, 0x52af200, 0xc009b43800, 0x165af24888277770, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:547 +0x6e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).MaybeAddAsync.func1(0x523bea0, 0xc00019e010, 0x51f26e0, 0xc003afa780)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:591 +0x6f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).Async.func1(0x523bea0, 0xc00019e010)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:580 +0x4e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func2(0xc0096a3c20, 0x523bea0, 0xc00019e010, 0xc00865aca0, 0x14, 0xc002df09a0, 0x0, 0xc00372b8c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:406 +0x112
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:400 +0x252

Log preceding fatal error

=== RUN   TestStoreRangeSplitAtTablePrefix
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestStoreRangeSplitAtTablePrefix403597557
    test_log_scope.go:74: use -show-logs to present logs inline

More

Parameters:

• TAGS=
• GOFLAGS=-parallel=4

make stressrace TESTS=TestStoreRangeSplitAtTablePrefix PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[RaduBerinde]

https://teamcity.cockroachdb.com/viewLog.html?buildId=2595917&buildTypeId=Cockroach_UnitTests_Testrace&tab=buildResultsDiv

[RaduBerinde]

Note: some more discussion in #59031

[thoszhang]

I unskipped the test locally and reproduced this a few times with make roachprod-stressrace CLUSTER=lucy-stress PKG=./pkg/kv/kvserver TESTS=TestStoreRangeSplitAtTablePrefix (I can usually get a failure within ~100 runs). The ID with a missing descriptor is always 50 (which should be defaultdb?), which is keys.UserTableDataMin, which makes sense given what this test is for:

// TestStoreRangeSplitAtTablePrefix verifies a range can be split at
// UserTableDataMin and still gossip the SystemConfig properly.
func TestStoreRangeSplitAtTablePrefix(t *testing.T) {

I also tried this with the enums logic test (failed in #59031), with make roachprod-stressrace CLUSTER=lucy-stress PKG=./pkg/sql/logictest TESTS=TestLogic/^local\$$/enums, and didn't get any panics, but I kept running into unrelated races (probably #40336) after a dozen runs or so, and it's slow so I gave up.

I also tried running TestStoreRangeSplitInsideRow for the sake of comparison and haven't gotten any failures in ~3000 runs.

[thoszhang]

@ajwerner @nvanbenschoten do you know if anything has changed recently in how we use the system config to determine splits?

[nvanbenschoten]

do you know if anything has changed recently in how we use the system config to determine splits?

Nothing in the last 8 months, at least that I'm aware of.

[irfansharif]

Disappearing in @ajwerner's #76279.