ccl/serverccl: TestServerControllerMultiNodeTenantStartup failed

[cockroach-teamcity]

ccl/serverccl.TestServerControllerMultiNodeTenantStartup failed on master @ 99c0d277c26344a08903e1372ab62a034336d178:

Fatal error:

panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (100 MiB / 0 B)
    flow-disk-monitor 0 B
 [recovered]
	panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (100 MiB / 0 B)
    flow-disk-monitor 0 B
 [recovered]
	panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (100 MiB / 0 B)
    flow-disk-monitor 0 B

Stack:

goroutine 76668 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x49be65?, {0x7f730b8, 0xc00d8e7f50})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x6077460?, 0xc006a4e650?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x112e48a?, {0x7f730b8, 0xc00d8e7f50})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x6077460?, 0xc006a4e650?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc0058585a0, {0x7f730b8, 0xc0096f0570}, 0x0)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:657 +0x653
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).EmergencyStop(...)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:593
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func6()
	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:862 +0x2c
github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc006309bf0?)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108 +0x12
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc009688090, {0x7f730b8, 0xc00d8e7f50})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550 +0x131
github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1({0x7f730b8, 0xc00d8e7f50})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489 +0x15c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 76667
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe

Log preceding fatal error

*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550
*   | github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) found 1 short-living non-stopped monitors in in-mem temp storage
*   | in-mem temp storage 0 B (100 MiB / 0 B)
*   |     flow-disk-monitor 0 B
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
*
* ERROR: a panic has occurred!
* found 1 short-living non-stopped monitors in in-mem temp storage
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop
*   | 	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:657
*   | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).EmergencyStop
*   | 	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:593
*   | github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func6
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:862
*   | github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550
*   | github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) found 1 short-living non-stopped monitors in in-mem temp storage
*   | in-mem temp storage 0 B (100 MiB / 0 B)
*   |     flow-disk-monitor 0 B
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*

Parameters:

• attempt=1
• run=14
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-39106

[yuzefovich]

dup of #124848