ccl/backupccl: TestDataDriven_restore_tenants failed

[cockroach-teamcity]

ccl/backupccl.TestDataDriven_restore_tenants failed on master @ e5f5597b1c1570fff03ee6dfd4a1e8be65c0e008:

Fatal error:

panic: found 4 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (100 MiB / 0 B)
    ?buffer 1 (running_migration_jobs)?-disk 0 B
    flow-disk-monitor 0 B
        streamer-1-disk-2 0 B
    subquery-disk 0 B
 [recovered]
	panic: found 4 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (100 MiB / 0 B)
    ?buffer 1 (running_migration_jobs)?-disk 0 B
    flow-disk-monitor 0 B
        streamer-1-disk-2 0 B
    subquery-disk 0 B
 [recovered]
	panic: found 4 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (100 MiB / 0 B)
    ?buffer 1 (running_migration_jobs)?-disk 0 B
    flow-disk-monitor 0 B
        streamer-1-disk-2 0 B
    subquery-disk 0 B

Stack:

goroutine 33389474 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x49bee5?, {0x80fa638, 0xc0016a5950})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x6191160?, 0xc021748080?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x113040a?, {0x80fa638, 0xc0016a5950})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x6191160?, 0xc021748080?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc00b0ea780, {0x80fa638, 0xc02db662d0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:657 +0x653
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).EmergencyStop(...)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:593
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func6()
	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:862 +0x2c
github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc007c44e20?)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108 +0x12
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc0154739e0, {0x80fa638, 0xc0016a5950})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550 +0x131
github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1({0x80fa638, 0xc0016a5950})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489 +0x15c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 33389449
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe

Log preceding fatal error

*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) found 4 short-living non-stopped monitors in in-mem temp storage
*   | in-mem temp storage 0 B (100 MiB / 0 B)
*   |     ?buffer 1 (running_migration_jobs)?-disk 0 B
*   |     flow-disk-monitor 0 B
*   |         streamer-1-disk-2 0 B
*   |     subquery-disk 0 B
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
*
* ERROR: a panic has occurred!
* found 4 short-living non-stopped monitors in in-mem temp storage
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop
*   | 	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:657
*   | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).EmergencyStop
*   | 	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:593
*   | github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func6
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:862
*   | github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550
*   | github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1
*   | 	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) found 4 short-living non-stopped monitors in in-mem temp storage
*   | in-mem temp storage 0 B (100 MiB / 0 B)
*   |     ?buffer 1 (running_migration_jobs)?-disk 0 B
*   |     flow-disk-monitor 0 B
*   |         streamer-1-disk-2 0 B
*   |     subquery-disk 0 B
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*

Parameters:

• attempt=1
• deadlock=true
• run=2
• shard=13

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-39109

[yuzefovich]

I'll use #124848 as a tracking issue for this.