kv/kvclient/kvcoord: TestMuxRangeFeedCanCloseStream failed

[cockroach-teamcity]

kv/kvclient/kvcoord.TestMuxRangeFeedCanCloseStream failed with artifacts on master @ 3797229f8951150426ff721d289ead52e063fb32:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x2436675]

Stack:

goroutine 5981619 [running]:
testing.tRunner.func1.2({0x4a4d040, 0x95cc6c0})
	GOROOT/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
	GOROOT/src/testing/testing.go:1548 +0x397
panic({0x4a4d040?, 0x95cc6c0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
	github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:133 +0x2a9
panic({0x4a4d040?, 0x95cc6c0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc01042e6f0?, {0x66d1f48, 0x9c3d0c0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x4a4d040?, 0x95cc6c0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00249c360, {0x66d1f48, 0x9c3d0c0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:521 +0x179
panic({0x4a4d040?, 0x95cc6c0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/span.(*llrbFrontier).Frontier(0x8000000000000000?)
	github.com/cockroachdb/cockroach/pkg/util/span/llrb_frontier.go:173 +0x15
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestMuxRangeFeedCanCloseStream.func1.1()
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_test.go:897 +0x25
github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75 +0x42
github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration(0xc015c4a0c0?, 0xc00ded3510)
	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213 +0x116
github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError(0xc00ded3620, 0xc000e78700?)
	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81 +0x8c
github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x66afb78, 0xc0142d1860}, 0x49822c0?, 0x9c3d0c0?)
	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x4c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestMuxRangeFeedCanCloseStream.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_test.go:896 +0xad
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestMuxRangeFeedCanCloseStream(0xc0142d1860)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_test.go:958 +0x910
testing.tRunner(0xc0142d1860, 0x5619ae8)
	GOROOT/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	GOROOT/src/testing/testing.go:1648 +0x3ad

Log preceding fatal error

=== RUN   TestMuxRangeFeedCanCloseStream
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/40ec041d13327ceafaefe721fe51f834/logTestMuxRangeFeedCanCloseStream396674838
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)

pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_test.go:885: (TestMuxRangeFeedCanCloseStream)
	NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    stopper.go:230: -- test log scope end --

ERROR: a panic has occurred!
Details cannot be printed yet because we are still unwinding.
Hopefully the test harness prints the panic below, otherwise check the test logs.

test logs left over in: /artifacts/tmp/_tmp/40ec041d13327ceafaefe721fe51f834/logTestMuxRangeFeedCanCloseStream396674838
    stopper.go:230: panic: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestMuxRangeFeedCanCloseStream (7.19s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-34081