Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRestoreInto, testrace/TestRestoreInto #21409

Closed
cockroach-teamcity opened this issue Jan 11, 2018 · 0 comments
Closed

teamcity: failed tests on master: test/TestRestoreInto, testrace/TestRestoreInto #21409

cockroach-teamcity opened this issue Jan 11, 2018 · 0 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#476276:

--- FAIL: test/TestRestoreInto (0.000s)
Test ended in panic.

------- Stdout: -------
W180111 23:29:09.406773 156445 server/status/runtime.go:109  Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180111 23:29:09.409949 156445 server/config.go:518  [n?] 1 storage engine initialized
I180111 23:29:09.409990 156445 server/config.go:521  [n?] RocksDB cache size: 128 MiB
I180111 23:29:09.410002 156445 server/config.go:521  [n?] store 0: in-memory, size 0 B
I180111 23:29:09.413913 156445 server/node.go:361  [n?] **** cluster 3bcb64eb-e486-4b16-9d1f-33cc7b045b6e has been created
I180111 23:29:09.413960 156445 server/server.go:937  [n?] **** add additional nodes by specifying --join=127.0.0.1:38647
I180111 23:29:09.414902 156445 storage/store.go:1312  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180111 23:29:09.415000 156445 server/node.go:491  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.2 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=3322.00 p25=3322.00 p50=3322.00 p75=3322.00 p90=3322.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180111 23:29:09.415037 156445 server/node.go:339  [n1] node ID 1 initialized
I180111 23:29:09.415106 156445 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:38647" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:9 > 
I180111 23:29:09.415161 156445 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180111 23:29:09.415243 156445 server/node.go:632  [n1] connecting to gossip network to verify cluster ID...
I180111 23:29:09.415280 156445 server/node.go:657  [n1] node connected via gossip and verified as part of cluster "3bcb64eb-e486-4b16-9d1f-33cc7b045b6e"
I180111 23:29:09.415308 156445 server/node.go:433  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180111 23:29:09.415353 156445 sql/distsql_physical_planner.go:122  [n1] creating DistSQLPlanner with address {tcp 127.0.0.1:38647}
I180111 23:29:09.417238 156429 storage/replica_command.go:819  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180111 23:29:09.422017 156445 server/server.go:1164  [n1] starting https server at 127.0.0.1:34027
I180111 23:29:09.422068 156445 server/server.go:1165  [n1] starting grpc/postgres server at 127.0.0.1:38647
I180111 23:29:09.422088 156445 server/server.go:1166  [n1] advertising CockroachDB node at 127.0.0.1:38647
panic: test timed out after 4m0s

goroutine 156677 [running]:
testing.startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1145 +0xf9
created by time.goFunc
	/usr/local/go/src/time/sleep.go:170 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200ae690, 0x23384c6, 0xf, 0x24112d8, 0x367c701)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4200ae690)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4200ae690, 0xc4206a3db8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc420258300, 0x342d4a0, 0x38, 0x38, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc42090bf18, 0x36c9500)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.TestMain(0xc4206a3f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/main_test.go:31 +0xda
main.main()
	github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/_test/_testmain.go:172 +0xdb

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x367e6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 19 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 156445 [semacquire]:
sync.runtime_SemacquireMutex(0xc422cf6794, 0xc4229f8b00)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc422cf6790)
	/usr/local/go/src/sync/mutex.go:134 +0xee
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc422cf6790)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x2d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).propose(0xc422cf6700, 0x34589c0, 0xc42169c510, 0x0, 0x0, 0xc42035a320, 0x100000001, 0x1, 0xc42035a330, 0xc42035a340, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3082 +0xab4
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc422cf6700, 0x34589c0, 0xc42169c510, 0x1508e542bda274de, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc421506460, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2747 +0x387
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc422cf6700, 0x34589c0, 0xc42169c510, 0x1508e542bda274de, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc421506460, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2601 +0xae
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc422cf6700, 0x34589c0, 0xc42169c510, 0x1508e542bda274de, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc421506460, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1913 +0x6a5
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4206fc800, 0x34589c0, 0xc42169c4b0, 0x1508e542bda274de, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc421506460, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2688 +0x688
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc422f76580, 0x34589c0, 0xc42169c3f0, 0x0, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0xc421506460, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:214 +0x214
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x34589c0, 0xc42169c3f0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:865 +0x192
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc420bd54d0, 0x34589c0, 0xc42169c360, 0x233a906, 0x10, 0xc4226f6858, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:245 +0xf7
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc423c21800, 0x34589c0, 0xc42169c360, 0xc420710598, 0xc42169c360, 0x14ee93e, 0x34589c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:852 +0x152
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc423c21800, 0x34589c0, 0xc42169c360, 0xc420710598, 0x1f, 0x91955a, 0xc423b7e000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:893 +0x99
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send.func1(0xc420cbe2a0, 0xc420710580, 0x34589c0, 0xc42169c150, 0x3, 0xc4226f6d38, 0x8580c8)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:243 +0x6f8
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send(0xc420cbe2a0, 0x34589c0, 0xc42169c150, 0xc4202f2010, 0xf, 0xc423ce01e0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:274 +0x9a
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc420cbe2a0, 0x34589c0, 0xc42169c150, 0xc4204b5bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:208 +0x31b
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc4200ae870, 0x34589c0, 0xc42169c150, 0xc4200ae8c0, 0x1, 0xc4229061a0, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1246 +0x3a4
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc4200ae870, 0x34589c0, 0xc42169c150, 0x1, 0xc4229061a0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:382 +0x27b
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc4200ae870, 0x34589c0, 0xc42169c150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421506380, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:446 +0x16e
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc4200ae870, 0x34589c0, 0xc42169c150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421506380, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1024 +0x347
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc4200ae870, 0x34589c0, 0xc42169c150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421506380, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:700 +0x16f8
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc4200ae870, 0x34589c0, 0xc42169c150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4215061c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:621 +0x3f5
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc4228c2380, 0x34589c0, 0xc42169c150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4215061c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:444 +0x1ce
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc420301e60, 0x3458900, 0xc422fc6d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4215061c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:546 +0x2c9
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc421614300, 0x3458900, 0xc422fc6d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:953 +0x308
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).sendEndTxnReq(0xc421614300, 0x3458900, 0xc422fc6d00, 0x2413201, 0x0, 0xc422b90980)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:671 +0x1be
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).commit(0xc421614300, 0x3458900, 0xc422fc6d00, 0xc421614338, 0xc421614338)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:518 +0x51
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Commit(0xc421614300, 0x3458900, 0xc422fc6d00, 0xc4203e0f80, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:550 +0x3f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc421614300, 0x3458900, 0xc422fc6d00, 0xc422fc0101, 0xc424f706b0, 0x0, 0x10048b2)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:764 +0x531
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc420301e60, 0x3458900, 0xc422fc6d00, 0xc420fb0de0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:507 +0x140
github.com/cockroachdb/cockroach/pkg/internal/client.(*LeaseManager).AcquireLease(0xc420fb08a0, 0x34589c0, 0xc4213677a0, 0xc4200140a0, 0x15, 0x15, 0x3fc3333333333333, 0x0, 0x2faf080)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/lease.go:95 +0xfa
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations(0xc422fc66c0, 0x34589c0, 0xc4213677a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:299 +0x611
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc4226e2500, 0x34589c0, 0xc4213677a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1226 +0x32f9
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc421612b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x137
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:203 +0xef
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x3469740, 0xc4224b22d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:173 +0x55
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).doAddServer(0xc421f12230, 0x3469740, 0xc4224b22d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:247 +0x112
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster(0x3469740, 0xc4224b22d0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:155 +0x55a
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetupWithParams(0x3469740, 0xc4224b22d0, 0x1, 0x1, 0x2411300, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:86 +0xfe
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetup(0x3469740, 0xc4224b22d0, 0x1, 0x1, 0x2411300, 0xc42156df30, 0x2212e93b0, 0xc4202ad158, 0xc42156df40, 0x8a4f76, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:151 +0x96
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestRestoreInto(0xc4224b22d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:2033 +0xa5
testing.tRunner(0xc4224b22d0, 0x24112d8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 66 [select, locked to thread]:
runtime.gopark(0x24150b0, 0x0, 0x232b1d5, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc420bbaf50, 0xc420078ae0)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 156433 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x34589c0, 0xc420632cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420383c20, 0xc420bd54d0, 0xc422b9a180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156577 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000007)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad8450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad8450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d68a0, 0xc420bd54d0, 0xc4203d6890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156363 [select]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x34589c0, 0xc420d28120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x6b4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420314080, 0xc420bd54d0, 0xc420314070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156597 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000001d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6ba0, 0xc420bd54d0, 0xc4203d6b80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156661 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc420632d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7610, 0xc420bd54d0, 0xc4203d7600)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156637 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc420632e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7510, 0xc420bd54d0, 0xc4203d7500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156428 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x34589c0, 0xc420632c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203839d0, 0xc420bd54d0, 0xc422b9a0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156668 [semacquire]:
sync.runtime_notifyListWait(0xc42395e100, 0xc400000004)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42395e0f0)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc42395e0c0, 0xc423f2a000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc4206f4050, 0xc423f2a000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc422ec07e0, 0x7f3cea030c38, 0xc4203d6120, 0x5, 0xc4203d6120, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc422656a80, 0x2415117, 0xc422656ba0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc422656a80, 0xc422bd2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc4212e0360, 0xc4227ba118, 0x9, 0x9, 0x28, 0x21f67a0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x3437500, 0xc4212e0360, 0xc4227ba118, 0x9, 0x9, 0x9, 0xc421c6ade8, 0x84730e, 0xc421c6ae97)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x3437500, 0xc4212e0360, 0xc4227ba118, 0x9, 0x9, 0xd0cee4, 0xc4234a0930, 0xc4234a0004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4227ba118, 0x9, 0x9, 0x3437500, 0xc4212e0360, 0x0, 0x0, 0xc4234a0930, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4227ba0e0, 0xc4234a0930, 0xc4234a0930, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc423c22300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1118 +0xbe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:266 +0xc80

goroutine 156624 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x34589c0, 0xc422ad8a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6f70, 0xc420bd54d0, 0xc4215ef860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156579 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000001a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad8660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad8660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6910, 0xc420bd54d0, 0xc4203d6900)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156608 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000039)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422c245a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422c245a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6d50, 0xc420bd54d0, 0xc4203d6d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156601 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000022)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422c24330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422c24330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6c70, 0xc420bd54d0, 0xc4203d6c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156708 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x3458900, 0xc421240180, 0xc421e8c720, 0xc422940fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc423c22300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:297 +0xdd1

goroutine 156605 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000038)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad8900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad8900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6cf0, 0xc420bd54d0, 0xc4203d6ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156617 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000003f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad8990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad8990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6e70, 0xc420bd54d0, 0xc4203d6e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156368 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake(0xc42091a028, 0x3458980, 0xc422ff84e0, 0xc4202f2010, 0xf, 0x3464b80, 0xc420bf2008, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials/credentials.go:149 +0x20c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client(0x3458900, 0xc421034880, 0xc4202f2010, 0xf, 0x0, 0x0, 0x2335b3b, 0xd, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:176 +0x1b14
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.NewClientTransport(0x3458900, 0xc421c62800, 0xc4202f2010, 0xf, 0x0, 0x0, 0x2335b3b, 0xd, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:524 +0xbe
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc424964000, 0xc420d28120, 0xc420bd54f0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:929 +0x2e5
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc424964000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:695 +0x2f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:694 +0x40d

goroutine 156638 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc420632e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7530, 0xc420bd54d0, 0xc4203d7520)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156613 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000003b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422c245d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422c245d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6df0, 0xc420bd54d0, 0xc4203d6de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156587 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000002e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422c243f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422c243f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6a10, 0xc420bd54d0, 0xc4203d6a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156598 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000024)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6c10, 0xc420bd54d0, 0xc4203d6bb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156470 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x34589c0, 0xc422c24240)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1151 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421218d90, 0xc420bd54d0, 0xc421218d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156570 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000000a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad84b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad84b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6750, 0xc420bd54d0, 0xc4203d6730)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156663 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc42041e930)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7650, 0xc420bd54d0, 0xc4203d7640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156644 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc42395e300, 0x34589c0, 0xc420632c60, 0x18bd7850, 0xed1e9ea45, 0x0, 0xc420e9d1c0, 0xc420bd54d0, 0xc422cf6700, 0x8bce19)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:207 +0x226
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc422cf6700, 0xc420e6adb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:269 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc420e6adb0, 0xc42244e140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:322 +0x1ba
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x34589c0, 0xc420632c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:267 +0x1f5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420383c50, 0xc420bd54d0, 0xc422b9a200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156622 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x34589c0, 0xc420632bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6f50, 0xc420bd54d0, 0xc4215ef7c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156563 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000011)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6640, 0xc420bd54d0, 0xc4203d6630)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156691 [select]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x34589c0, 0xc420e6a210)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:125 +0x17b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424f70490, 0xc420bd54d0, 0xc424f70480)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156347 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x34589c0, 0xc422c24090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1469 +0x912
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4212188d0, 0xc420bd54d0, 0xc423b66870)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156574 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000001c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422c242d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422c242d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6840, 0xc420bd54d0, 0xc4203d6830)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156575 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000000d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad8540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad8540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6860, 0xc420bd54d0, 0xc4203d6850)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156562 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000000f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6620, 0xc420bd54d0, 0xc4203d6610)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156345 [chan receive]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x34589c0, 0xc420e6a000)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:326 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4212184b0, 0xc420bd54d0, 0xc420301ac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156568 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000008)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc4206328a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc4206328a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d66f0, 0xc420bd54d0, 0xc4203d66d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156629 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x34589c0, 0xc422c24db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:685 +0x2dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d72e0, 0xc420bd54d0, 0xc42263bc40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156448 [runnable]:
math/big.nat.montgomery(0xc42075c000, 0x10, 0x24, 0xc4226e7e00, 0x10, 0x26, 0xc4228172c0, 0x10, 0x14, 0xc4215289a0, ...)
	/usr/local/go/src/math/big/nat.go:202 +0x379
math/big.nat.expNNMontgomery(0xc4226e7e00, 0x10, 0x26, 0xc420dc3d40, 0x10, 0x25, 0xc4226e77c0, 0x10, 0x25, 0xc4215289a0, ...)
	/usr/local/go/src/math/big/nat.go:1108 +0x755
math/big.nat.expNN(0x0, 0x0, 0x0, 0xc420dc3d40, 0x20, 0x45, 0xc4226e77c0, 0x10, 0x25, 0xc4215289a0, ...)
	/usr/local/go/src/math/big/nat.go:918 +0xa69
math/big.(*Int).Exp(0xc42415f5f8, 0xc42415f618, 0xc420300c40, 0xc420300b80, 0xc422907480)
	/usr/local/go/src/math/big/int.go:437 +0xf6
crypto/rsa.decrypt(0x34377c0, 0xc42007aa20, 0xc4230a42a0, 0xc42415f618, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/rsa/rsa.go:525 +0x1c4
crypto/rsa.decryptAndCheck(0x34377c0, 0xc42007aa20, 0xc4230a42a0, 0xc42415f768, 0xc42415f768, 0x100, 0x0)
	/usr/local/go/src/crypto/rsa/rsa.go:559 +0x56
crypto/rsa.SignPKCS1v15(0x34377c0, 0xc42007aa20, 0xc4230a42a0, 0x5, 0xc42042ee80, 0x20, 0x20, 0xc42415f800, 0x855b2a, 0x10, ...)
	/usr/local/go/src/crypto/rsa/pkcs1v15.go:250 +0x2ca
crypto/rsa.(*PrivateKey).Sign(0xc4230a42a0, 0x34377c0, 0xc42007aa20, 0xc42042ee80, 0x20, 0x20, 0x343f680, 0xc4214d8da0, 0xc4206c9500, 0x8fc, ...)
	/usr/local/go/src/crypto/rsa/rsa.go:105 +0x15a
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc42415fe88, 0xc4203124c0, 0x31)
	/usr/local/go/src/crypto/tls/handshake_client.go:456 +0x109d
crypto/tls.(*Conn).clientHandshake(0xc422f30700, 0x24151d8, 0xc422f30820)
	/usr/local/go/src/crypto/tls/handshake_client.go:228 +0xf9b
crypto/tls.(*Conn).Handshake(0xc422f30700, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1307 +0x18f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake.func1(0xc422ff8540, 0xc422f30700)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials/credentials.go:147 +0x2b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials/credentials.go:146 +0x118

goroutine 156606 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000035)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad88d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad88d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6d10, 0xc420bd54d0, 0xc4203d6d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156593 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000002d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422ad8870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422ad8870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6ae0, 0xc420bd54d0, 0xc4203d6ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156662 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc42041e900)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7630, 0xc420bd54d0, 0xc4203d7620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156614 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000033)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6e10, 0xc420bd54d0, 0xc4203d6e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156567 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000015)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d66c0, 0xc420bd54d0, 0xc4203d66b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156664 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc42041e840)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7670, 0xc420bd54d0, 0xc4203d7660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156431 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x34589c0, 0xc422ad8b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420383c00, 0xc420bd54d0, 0xc422b9a140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156453 [select]:
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).heartbeatLoop(0xc4228c2380, 0x34589c0, 0xc4213660c0, 0x4431388c927c9d9, 0xa83089c90afb8993)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:657 +0x361
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).updateState.func2(0x34589c0, 0xc42169c120)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:964 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc420bd54d0, 0x34589c0, 0xc42169c120, 0xc4205840f0, 0x29, 0x3468020, 0xc4210b4750, 0xc422906000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x133

goroutine 156494 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000043)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d65a0, 0xc420bd54d0, 0xc4203d6590)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156599 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000025)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc422c24360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc422c24360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6c30, 0xc420bd54d0, 0xc4203d6c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156642 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x34589c0, 0xc420632cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420383c30, 0xc420bd54d0, 0xc422b9a1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156566 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000013)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d66a0, 0xc420bd54d0, 0xc4203d6690)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156611 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc40000003c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc420632ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc420632ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6db0, 0xc420bd54d0, 0xc4203d6da0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156706 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x34589c0, 0xc4234a0750, 0xc4234a06f0, 0xc422941fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc42367c2c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8d5

goroutine 156596 [semacquire]:
sync.runtime_notifyListWait(0xc421168f50, 0xc400000019)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421168f40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422484120, 0x34589c0, 0xc4206329f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x34589c0, 0xc4206329f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6b70, 0xc420bd54d0, 0xc4203d6b50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156640 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x34589c0, 0xc4245820f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7570, 0xc420bd54d0, 0xc4203d7560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156623 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x34589c0, 0xc422c24660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6f60, 0xc420bd54d0, 0xc4215ef810)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 156632 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc422cf6700, 0x34589c0, 0xc422cc7e60, 0x1508e542bd6746e1, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2773 +0x62a
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc422cf6700, 0x34589c0, 0xc422cc7e60, 0x1508e542bd6746e1, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2601 +0xae
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc422cf6700, 0x34589c0, 0xc422cc7e60, 0x1508e542bd6746e1, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1913 +0x6a5
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4206fc800, 0x34589c0, 0xc422cc7e00, 0x1508e542bd6746e1, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2688 +0x688
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc422f76580, 0x34589c0, 0xc422cc7d70, 0x0, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:214 +0x214
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x34589c0, 0xc422cc7d70, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:865 +0x192
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc420bd54d0, 0x34589c0, 0xc422cc7d40, 0x233a906, 0x10, 0xc420915598, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:245 +0xf7
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc423c21800, 0x34589c0, 0xc422cc7d40, 0xc422e2f4b8, 0xc422cc7d40, 0x14ee93e, 0x34589c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:852 +0x152
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc423c21800, 0x34589c0, 0xc422cc7d40, 0xc422e2f4b8, 0x1f, 0x91955a, 0xc42395e6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:893 +0x99
github.com/cockroachdb/cockroach

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 11, 2018
@benesch benesch mentioned this issue Jan 16, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity