Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRestoreAsOfSystemTime/ts4, test/TestRestoreAsOfSystemTime/ts2, test/TestRestoreAsOfSystemTime/ts1, test/TestRestoreAsOfSystemTime/ts3, test/TestRestoreAsOfSystemTime, test/TestRestoreAsOfSystemTime/ts0, testrace/TestRestoreAsOfSystemTime, testrace/TestRestoreAsOfSystemTime/ts0, testrace/TestRestoreAsOfSystemTime/ts1, testrace/TestRestoreAsOfSystemTime/ts2, testrace/TestRestoreAsOfSystemTime/ts3, testrace/TestRestoreAsOfSystemTime/ts4 #21396

Closed
cockroach-teamcity opened this issue Jan 11, 2018 · 1 comment
Closed

teamcity: failed tests on master: test/TestRestoreAsOfSystemTime/ts4, test/TestRestoreAsOfSystemTime/ts2, test/TestRestoreAsOfSystemTime/ts1, test/TestRestoreAsOfSystemTime/ts3, test/TestRestoreAsOfSystemTime, test/TestRestoreAsOfSystemTime/ts0, testrace/TestRestoreAsOfSystemTime, testrace/TestRestoreAsOfSystemTime/ts0, testrace/TestRestoreAsOfSystemTime/ts1, testrace/TestRestoreAsOfSystemTime/ts2, testrace/TestRestoreAsOfSystemTime/ts3, testrace/TestRestoreAsOfSystemTime/ts4 #21396

cockroach-teamcity opened this issue Jan 11, 2018 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#475052:

--- FAIL: test/TestRestoreAsOfSystemTime/ts4 (0.000s)
Test ended in panic.

------- Stdout: -------
I180111 04:28:42.084815 168862 sql/event_log.go:115  [client=127.0.0.1:47004,user=root,n1] Event: "create_database", target: 60, info: {DatabaseName:ts4 Statement:CREATE DATABASE ts4 User:root}
I180111 04:28:42.085781 168285 storage/replica_command.go:819  [split,n1,s1,r84/1:/{Table/59/1/9-Max}] initiating a split of this range at key /Table/60 [r89]
I180111 04:28:42.090486 170002 storage/replica_command.go:819  [n1,s1,r89/1:/{Table/60-Max}] initiating a split of this range at key /Table/61 [r90]
I180111 04:28:42.093811 170002 storage/replica_command.go:819  [n1,s1,r90/1:/{Table/61-Max}] initiating a split of this range at key /Table/61/1/3 [r91]
I180111 04:28:42.094347 170004 storage/replica_command.go:819  [n1,s1,r90/1:/{Table/61-Max}] initiating a split of this range at key /Table/61/1/1 [r92]
W180111 04:28:42.100376 169255 storage/replica.go:2174  [n1,s1,r90/1:/Table/61{-/1/3}] context canceled before command queue: 1 QueryTxn
I180111 04:28:42.100715 170002 storage/replica_command.go:819  [n1,s1,r91/1:/{Table/61/1/3-Max}] initiating a split of this range at key /Table/61/1/6 [r93]
I180111 04:28:42.101395 170003 storage/replica_command.go:819  [n1,s1,r91/1:/{Table/61/1/3-Max}] initiating a split of this range at key /Table/61/1/4 [r94]
panic: test timed out after 4m0s

goroutine 169919 [running]:
testing.startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1145 +0xf9
created by time.goFunc
	/usr/local/go/src/time/sleep.go:170 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200ac690, 0x2346866, 0x19, 0x240d878, 0x8d4f01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4200ac690)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4200ac690, 0xc4206b5db8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc420282c60, 0x3428760, 0x38, 0x38, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420987f18, 0xc42025a050)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.TestMain(0xc4206b5f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/main_test.go:31 +0xda
main.main()
	github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/_test/_testmain.go:172 +0xdb

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3678780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 48 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 168205 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc42171fe00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1266 +0x14b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:302 +0xe51

goroutine 50 [select, locked to thread]:
runtime.gopark(0x2411680, 0x0, 0x2327c90, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc42004af50, 0xc42044a7e0)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 168159 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddToPurgatory.func2(0x3453be0, 0xc42231e390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:705 +0x434
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204001e0, 0xc4211c23f0, 0xc4247b6180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168342 [semacquire]:
sync.runtime_notifyListWait(0xc4236bb458, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4236bb448)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4236bb340)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:625 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:612 +0x4a3

goroutine 168628 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005e6)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa67e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa67e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341600, 0xc4211c23f0, 0xc4203415e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168644 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005df)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420804030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420804030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203418f0, 0xc4211c23f0, 0xc4203418e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168610 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005db)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42325b2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42325b2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341310, 0xc4211c23f0, 0xc420341300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168572 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005b3)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc4245d0030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc4245d0030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340cb0, 0xc4211c23f0, 0xc420340c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168201 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x3453b20, 0xc422eff600, 0xc420892660, 0xc4225bcfb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc4249b6000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:297 +0xdd1

goroutine 168548 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc40000059a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340970, 0xc4211c23f0, 0xc420340960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168688 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3453be0, 0xc4236a8720)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce0b0, 0xc4211c23f0, 0xc4203ce0a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168581 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005b9)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340e80, 0xc4211c23f0, 0xc420340e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168354 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3453be0, 0xc42354e000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204002c0, 0xc4211c23f0, 0xc4247b6120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168593 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005c9)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa6690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa6690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341000, 0xc4211c23f0, 0xc420340ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168360 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4247b61c0, 0x2410a60, 0xc4235f0000, 0x345fb40, 0xc4207da5a0)
	<autogenerated>:1 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4235f0000, 0x3452d60, 0xc4247b61c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:463 +0x196
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x3453be0, 0xc4211660f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:802 +0x43
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204003a0, 0xc4211c23f0, 0xc4247b6320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168553 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc400000596)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa6450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa6450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340a10, 0xc4211c23f0, 0xc420340a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168549 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc400000597)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d366f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d366f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340990, 0xc4211c23f0, 0xc420340980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168522 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc400000580)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340570, 0xc4211c23f0, 0xc420340560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168570 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005b2)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42325b140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42325b140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340c50, 0xc4211c23f0, 0xc420340c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168641 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005ec)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42354e840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42354e840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341890, 0xc4211c23f0, 0xc420341880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168596 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005c8)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc4236a84e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc4236a84e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341080, 0xc4211c23f0, 0xc420341050)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168683 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3453be0, 0xc420804150)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce010, 0xc4211c23f0, 0xc4203ce000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168691 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3453be0, 0xc42325b470)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce110, 0xc4211c23f0, 0xc4203ce100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168653 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3453be0, 0xc420d37020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341a30, 0xc4211c23f0, 0xc421cc9950)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168257 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3453be0, 0xc42231e180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203400a0, 0xc4211c23f0, 0xc4229fe040)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 169991 [IO wait]:
internal/poll.runtime_pollWait(0x7f2f4b9a61f0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4202e2698, 0x72, 0xffffffffffffff00, 0x343d9e0, 0x34308b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4202e2698, 0xc42034e800, 0x800, 0x800)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4202e2680, 0xc42034e800, 0x800, 0x800, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4202e2680, 0xc42034e800, 0x800, 0x800, 0x7fb, 0x800, 0x55)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42093c050, 0xc42034e800, 0x800, 0x800, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
crypto/tls.(*block).readFromUntil(0xc422418870, 0x7f2f4b9029c8, 0xc42093c050, 0x5, 0xc42093c050, 0xd)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc423498a80, 0x2411717, 0xc423498ba0, 0xc420860005)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc423498a80, 0xc420683000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc420389380, 0xc420b3c2e0, 0x5, 0x200, 0xc420eab438, 0x8719e8, 0x240d1c8)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x34327a0, 0xc420389380, 0xc420b3c2e0, 0x5, 0x200, 0x5, 0x857967, 0xc4210067c0, 0x20)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x34327a0, 0xc420389380, 0xc420b3c2e0, 0x5, 0x200, 0x7f2f4b9789a0, 0x0, 0xc420882328)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recvMessage(0xc420b3c2c0, 0xc4210067c0, 0x20287c0, 0x1, 0x36c3c18)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:947 +0x14e
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1Buf(0xc420b3c2c0, 0xc4210067c0, 0xc420b3c2c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:997 +0x39
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1(0xc420b3c2c0, 0xc423498a80, 0xc420b3c2e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1018 +0x85
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).readBindResponse(0xc420b3c2c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1633 +0x2f
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*stmt).exec(0xc424268c80, 0xc4215cf920, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1270 +0x979
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).query(0xc420b3c2c0, 0xc423dd5780, 0x7e, 0xc4215cf920, 0x2, 0x2, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:855 +0x36f
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).QueryContext(0xc420b3c2c0, 0x3453b60, 0xc420016108, 0xc423dd5780, 0x7e, 0xc420285720, 0x2, 0x2, 0xc420eab8d0, 0x8922e6, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:21 +0x1f9
database/sql.ctxDriverQuery(0x3453b60, 0xc420016108, 0x7f2f4b96b5b0, 0xc420b3c2c0, 0xc423dd5780, 0x7e, 0xc420285720, 0x2, 0x2, 0x871329, ...)
	/usr/local/go/src/database/sql/ctxutil.go:48 +0x277
database/sql.(*DB).queryDC.func1()
	/usr/local/go/src/database/sql/sql.go:1320 +0x99
database/sql.withLock(0x343e7a0, 0xc422e3e2a0, 0xc420eabab8)
	/usr/local/go/src/database/sql/sql.go:2855 +0x65
database/sql.(*DB).queryDC(0xc4223200a0, 0x3453b60, 0xc420016108, 0x0, 0x0, 0xc422e3e2a0, 0xc42030c5b0, 0xc423dd5780, 0x7e, 0xc420eabf78, ...)
	/usr/local/go/src/database/sql/sql.go:1319 +0x6c2
database/sql.(*DB).query(0xc4223200a0, 0x3453b60, 0xc420016108, 0xc423dd5780, 0x7e, 0xc420eabf78, 0x2, 0x2, 0xe48fb201, 0xc0c1f0186bcfddc9, ...)
	/usr/local/go/src/database/sql/sql.go:1304 +0x147
database/sql.(*DB).QueryContext(0xc4223200a0, 0x3453b60, 0xc420016108, 0xc423dd5780, 0x7e, 0xc420eabf78, 0x2, 0x2, 0x4a, 0xc421e72fb8, ...)
	/usr/local/go/src/database/sql/sql.go:1281 +0xd2
database/sql.(*DB).Query(0xc4223200a0, 0xc423dd5780, 0x7e, 0xc420eabf78, 0x2, 0x2, 0xc420eabd00, 0xc420eabd30, 0x93e1e9)
	/usr/local/go/src/database/sql/sql.go:1295 +0x82
github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query(0xc420fb25f0, 0x3464980, 0xc4225de4b0, 0xc423dd5780, 0x7e, 0xc420eabf78, 0x2, 0x2, 0xc420eabdb8)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:64 +0x93
github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr(0xc420fb25f0, 0x3464980, 0xc4225de4b0, 0xc423dd5780, 0x7e, 0xc420eabf78, 0x2, 0x2, 0x3442f20, 0xc4215cf900, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:95 +0xaa
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestRestoreAsOfSystemTime.func1(0xc4225de4b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:1761 +0x43a
testing.tRunner(0xc4225de4b0, 0xc424210190)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 168203 [IO wait]:
internal/poll.runtime_pollWait(0x7f2f4b9c78b0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4244bc118, 0x72, 0xffffffffffffff00, 0x343d9e0, 0x34308b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4244bc118, 0xc42463a000, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4244bc100, 0xc42463a000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4244bc100, 0xc42463a000, 0x400, 0x400, 0x96, 0x8, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc420fb2060, 0xc42463a000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
crypto/tls.(*block).readFromUntil(0xc423b9cf60, 0x7f2f4b9029c8, 0xc420fb2060, 0x5, 0xc420fb2060, 0xc4212afa68)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc42138a000, 0x2411717, 0xc42138a120, 0x857967)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc42138a000, 0xc4227ac000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc422cf4360, 0xc420307a78, 0x9, 0x9, 0x0, 0x8719e8, 0xc42046e380)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x34327a0, 0xc422cf4360, 0xc420307a78, 0x9, 0x9, 0x9, 0xc4241d4380, 0xc4212afe28, 0xd5f3fe)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x34327a0, 0xc422cf4360, 0xc420307a78, 0x9, 0x9, 0x2410bd8, 0x200050100, 0xc400000007)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420307a78, 0x9, 0x9, 0x34327a0, 0xc422cf4360, 0x0, 0x0, 0x0, 0xc42093c040)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420307a40, 0xc420d36e70, 0xc420d36e70, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc42171fe00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1118 +0xbe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:266 +0xc80

goroutine 168554 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005a7)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa6480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa6480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340a30, 0xc4211c23f0, 0xc420340a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168551 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005a2)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa63f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa63f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203409d0, 0xc4211c23f0, 0xc4203409c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168550 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc400000592)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa63c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa63c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203409b0, 0xc4211c23f0, 0xc4203409a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168573 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005bd)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42325b170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42325b170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340ce0, 0xc4211c23f0, 0xc420340cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168585 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005b1)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa6600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa6600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340f00, 0xc4211c23f0, 0xc420340ef0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168579 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005ad)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340e40, 0xc4211c23f0, 0xc420340e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 169893 [chan receive]:
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.doRestorePlan(0x3453be0, 0xc423b08120, 0xc424233110, 0x3462f40, 0xc4238fa250, 0xc4210068c0, 0x2, 0x2, 0x1508a70698567056, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/restore.go:1128 +0x512
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.restorePlanHook.func1(0x3453be0, 0xc423b08120, 0xc42440a360, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/restore.go:1073 +0x29f
github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1(0xc421637ea0, 0x3453b20, 0xc422ac1cc0, 0xc4238fa2c8, 0xc4238fa250)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:113 +0x4a
created by github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:112 +0xd6

goroutine 168204 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x3453b20, 0xc4218cc180, 0xc4215e93e0, 0xc4212adfb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc42171fe00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:297 +0xdd1

goroutine 168576 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005c4)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340d60, 0xc4211c23f0, 0xc420340d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168530 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc400000587)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d364b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d364b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340670, 0xc4211c23f0, 0xc420340660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168505 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42262e600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1002 +0x166
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc42262e600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:703 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:694 +0x40d

goroutine 168295 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420d8c160)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x264
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x8fa

goroutine 170004 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc421197e30, 0x343ec20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:119 +0x189
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).AdminSplit(0xc4209d0e00, 0x3453be0, 0xc422fecd50, 0xc42493e190, 0x3, 0x8, 0x0, 0x0, 0x0, 0xc42493e190, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:693 +0x12e
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc4209d0e00, 0x3453be0, 0xc422fecd50, 0x1508a706c30e783b, 0x0, 0x100000001, 0x1, 0x5a, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2449 +0xbec
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc4209d0e00, 0x3453be0, 0xc422fecd50, 0x1508a706c30e783b, 0x0, 0x100000001, 0x1, 0x5a, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1919 +0x60f
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc420169c00, 0x3453be0, 0xc422fecd20, 0x1508a706c30e783b, 0x0, 0x100000001, 0x1, 0x5a, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2688 +0x688
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc4233c2000, 0x3453be0, 0xc422feccc0, 0x0, 0x0, 0x100000001, 0x1, 0x5a, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:214 +0x214
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x3453be0, 0xc422feccc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:865 +0x192
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc4211c23f0, 0x3453be0, 0xc422feccc0, 0x2337386, 0x10, 0xc420bd2fc8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:245 +0xf7
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc422de0000, 0x3453be0, 0xc422feccc0, 0xc420938858, 0xc422feccc0, 0x14ee6be, 0x3453be0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:852 +0x152
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc422de0000, 0x3453be0, 0xc422feccc0, 0xc420938858, 0x1f, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:893 +0x99
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send.func1(0xc42071f3b0, 0xc420938840, 0x3453be0, 0xc422fecc30, 0x0, 0xc420bd34a8, 0x8580c8)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:243 +0x6f8
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send(0xc42071f3b0, 0x3453be0, 0xc422fecc30, 0xc420722010, 0xf, 0xc422469770, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:274 +0x9a
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc42071f3b0, 0x3453be0, 0xc422fecc30, 0xc42363c9c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:208 +0x31b
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc4205fe690, 0x3453be0, 0xc422fecc30, 0xc4205fe6e0, 0x5a, 0xc4230ac4e0, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1246 +0x3a4
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc4205fe690, 0x3453be0, 0xc422fecc30, 0x5a, 0xc4230ac4e0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:382 +0x27b
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc4205fe690, 0x3453be0, 0xc422fecc30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:446 +0x16e
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc4205fe690, 0x3453be0, 0xc422fecc30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1024 +0x347
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc4205fe690, 0x3453be0, 0xc422fecc30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:700 +0x16f8
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc4205fe690, 0x3453be0, 0xc422fecc30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:621 +0x3f5
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc420a64700, 0x3453be0, 0xc422fecc30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:444 +0x1ce
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc425401160, 0x3453b20, 0xc42116e380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:546 +0x2c9
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm(0x3453b20, 0xc42116e380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:480 +0x76
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x3453b20, 0xc42116e380, 0xc420bd54c0, 0xc421f2a000, 0xc42493e190, 0xc422b36230)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:457 +0xf8
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc425401160, 0x3453b20, 0xc42116e380, 0xc421f2a000, 0xc4230ac460, 0xc4230ac460)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:480 +0x94
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminSplit(0xc425401160, 0x3453b20, 0xc42116e380, 0x220b660, 0xc4230ac440, 0x220b660, 0xc4230ac460, 0x2, 0x3)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:386 +0x98
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.splitAndScatter.func2(0x8, 0x24117c8)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/restore.go:676 +0x41b
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc42116e3c0, 0xc423024070)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

goroutine 168555 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005b4)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42354e630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42354e630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340a50, 0xc4211c23f0, 0xc420340a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168692 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3453be0, 0xc420d37230)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce130, 0xc4211c23f0, 0xc4203ce120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168962 [chan receive]:
database/sql.(*DB).connectionOpener(0xc4223200a0)
	/usr/local/go/src/database/sql/sql.go:871 +0x53
created by database/sql.Open
	/usr/local/go/src/database/sql/sql.go:609 +0x1ee

goroutine 168618 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005d0)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc4236a85a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc4236a85a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341410, 0xc4211c23f0, 0xc420341400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168533 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc400000591)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203406d0, 0xc4211c23f0, 0xc4203406c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168703 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func2(0x3453be0, 0xc423b9d020)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:205 +0x13b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42221d510, 0xc4211c23f0, 0xc422d0aff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168537 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc40000059c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340760, 0xc4211c23f0, 0xc420340750)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168658 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3453be0, 0xc420fa6a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438110, 0xc4211c23f0, 0xc422d8c120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168563 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005b7)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420fa6510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420fa6510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340b50, 0xc4211c23f0, 0xc420340b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168648 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc40000057d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42325b3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42325b3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341990, 0xc4211c23f0, 0xc420341980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168615 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005c0)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42325b200)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42325b200)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203413b0, 0xc4211c23f0, 0xc4203413a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168225 [chan receive]:
testing.(*T).Run(0xc4205fe2d0, 0xc420313900, 0x3, 0xc424210190, 0x1)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestRestoreAsOfSystemTime(0xc4205fe2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:1758 +0xfea
testing.tRunner(0xc4205fe2d0, 0x240d878)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 170002 [select]:
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.splitAndScatter.func1(0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/restore.go:649 +0x762
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc42116e3c0, 0xc4238c4540)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

goroutine 168568 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005ba)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc420d36c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc420d36c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420340bf0, 0xc4211c23f0, 0xc420340be0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168613 [semacquire]:
sync.runtime_notifyListWait(0xc4232a4c10, 0xc4000005d2)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4232a4c00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4243c6360, 0x3453be0, 0xc42325b320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3453be0, 0xc42325b320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420341370, 0xc4211c23f0, 0xc420341360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168696 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3453be0, 0xc42145e0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce1b0, 0xc4211c23f0, 0xc4203ce1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168515 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x3453be0, 0xc421166990)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1026 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203401a0, 0xc4211c23f0, 0xc42231e780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168294 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x3453be0, 0xc422f377d0, 0xc422f376e0, 0xc42351bfb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc420d8c160)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8d5

goroutine 170003 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc4221f1e30, 0x343ec20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:119 +0x189
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).AdminSplit(0xc4243d9180, 0x3453be0, 0xc4246aa6f0, 0xc4203d86f0, 0x3, 0x8, 0x0, 0x0, 0x0, 0xc42

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 11, 2018
@benesch
Copy link
Contributor

benesch commented Jan 20, 2018

Timeout caused by slow partitioning tests. Addressed by #21392.

@benesch benesch closed this as completed Jan 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@benesch @cockroach-teamcity