Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestBackupRestoreIncrementalAddTableMissing, testrace/TestBackupRestoreIncrementalAddTableMissing #21423

Closed
cockroach-teamcity opened this issue Jan 12, 2018 · 0 comments
Closed

teamcity: failed tests on master: test/TestBackupRestoreIncrementalAddTableMissing, testrace/TestBackupRestoreIncrementalAddTableMissing #21423

cockroach-teamcity opened this issue Jan 12, 2018 · 0 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#477021:

--- FAIL: test/TestBackupRestoreIncrementalAddTableMissing (0.000s)
Test ended in panic.

------- Stdout: -------
W180112 19:51:13.440830 174980 server/status/runtime.go:109  Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180112 19:51:13.443253 174980 server/config.go:518  [n?] 1 storage engine initialized
I180112 19:51:13.443279 174980 server/config.go:521  [n?] RocksDB cache size: 128 MiB
I180112 19:51:13.443285 174980 server/config.go:521  [n?] store 0: in-memory, size 0 B
I180112 19:51:13.447618 174980 server/node.go:361  [n?] **** cluster 13e52292-9395-47cd-8b89-39b6e1559a74 has been created
I180112 19:51:13.447657 174980 server/server.go:937  [n?] **** add additional nodes by specifying --join=127.0.0.1:33205
I180112 19:51:13.448876 174980 storage/store.go:1312  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180112 19:51:13.448935 174980 server/node.go:491  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.2 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=3322.00 p25=3322.00 p50=3322.00 p75=3322.00 p90=3322.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180112 19:51:13.448956 174980 server/node.go:339  [n1] node ID 1 initialized
I180112 19:51:13.449005 174980 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:33205" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:9 > 
I180112 19:51:13.449138 174980 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180112 19:51:13.449211 174980 server/node.go:632  [n1] connecting to gossip network to verify cluster ID...
I180112 19:51:13.449237 174980 server/node.go:657  [n1] node connected via gossip and verified as part of cluster "13e52292-9395-47cd-8b89-39b6e1559a74"
I180112 19:51:13.449256 174980 server/node.go:433  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180112 19:51:13.449297 174980 sql/distsql_physical_planner.go:122  [n1] creating DistSQLPlanner with address {tcp 127.0.0.1:33205}
I180112 19:51:13.450857 175015 storage/replica_command.go:819  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180112 19:51:13.452347 174980 server/server.go:1164  [n1] starting https server at 127.0.0.1:46301
I180112 19:51:13.452465 174980 server/server.go:1165  [n1] starting grpc/postgres server at 127.0.0.1:33205
I180112 19:51:13.452479 174980 server/server.go:1166  [n1] advertising CockroachDB node at 127.0.0.1:33205
I180112 19:51:13.464552 175015 storage/replica_command.go:819  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180112 19:51:13.465106 174980 sql/event_log.go:115  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180112 19:51:13.469772 175015 storage/replica_command.go:819  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
I180112 19:51:13.470633 174980 sql/lease.go:348  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-12 19:51:13.469865477 +0000 UTC
I180112 19:51:13.477916 175015 storage/replica_command.go:819  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180112 19:51:13.480019 174980 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
I180112 19:51:13.482534 175015 storage/replica_command.go:819  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180112 19:51:13.489827 175015 storage/replica_command.go:819  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180112 19:51:13.497823 175015 storage/replica_command.go:819  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
W180112 19:51:13.498080 175304 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "sql txn implicit" id=0ade674c key=/Table/SystemConfigSpan/Start rw=true pri=0.01216517 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515786673.485443784,0 orig=1515786673.485443784,0 max=1515786673.485443784,0 wto=false rop=false seq=15
I180112 19:51:13.499618 174980 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
I180112 19:51:13.502515 175015 storage/replica_command.go:819  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180112 19:51:13.505068 174980 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180112 19:51:13.506370 175015 storage/replica_command.go:819  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180112 19:51:13.509203 175015 storage/replica_command.go:819  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180112 19:51:13.513390 175015 storage/replica_command.go:819  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180112 19:51:13.516111 175015 storage/replica_command.go:819  [split,n1,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180112 19:51:13.516693 174980 sql/event_log.go:115  [n1] Event: "alter_table", target: 4, info: {TableName:users Statement:ALTER TABLE system.users ADD COLUMN IF NOT EXISTS "isRole" BOOL NOT NULL DEFAULT false User:node MutationID:1 CascadeDroppedViews:[]}
I180112 19:51:13.520589 174980 sql/lease.go:348  [n1] publish: descID=4 (users) version=2 mtime=2018-01-12 19:51:13.520468009 +0000 UTC
I180112 19:51:13.523756 175015 storage/replica_command.go:819  [split,n1,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180112 19:51:13.526626 174980 sql/lease.go:348  [n1] publish: descID=4 (users) version=3 mtime=2018-01-12 19:51:13.526453029 +0000 UTC
I180112 19:51:13.527212 175015 storage/replica_command.go:819  [split,n1,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180112 19:51:13.530677 174980 sql/backfill.go:133  [n1] Running backfill for "users", v=3, m=1
I180112 19:51:13.530697 175015 storage/replica_command.go:819  [split,n1,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
I180112 19:51:13.534068 175015 storage/replica_command.go:819  [split,n1,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
I180112 19:51:13.536901 175015 storage/replica_command.go:819  [split,n1,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
I180112 19:51:13.539472 174980 sql/lease.go:348  [n1] publish: descID=4 (users) version=4 mtime=2018-01-12 19:51:13.53936217 +0000 UTC
I180112 19:51:13.540169 175015 storage/replica_command.go:819  [split,n1,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
I180112 19:51:13.543171 175015 storage/replica_command.go:819  [split,n1,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
I180112 19:51:13.543587 174980 sql/event_log.go:115  [n1] Event: "finish_schema_change", target: 4, info: {MutationID:1}
I180112 19:51:13.544512 174980 sql/lease.go:274  publish (count leases): descID=4 name=users version=3 count=1
I180112 19:51:13.574610 174980 server/server.go:1235  [n1] done ensuring all necessary migrations have run
I180112 19:51:13.574660 174980 server/server.go:1238  [n1] serving sql connections
I180112 19:51:13.576700 175452 sql/event_log.go:115  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:33205} Attrs: Locality: ServerVersion:1.1-9} ClusterID:13e52292-9395-47cd-8b89-39b6e1559a74 StartedAt:1515786673449242163 LastUp:1515786673449242163}
I180112 19:51:13.588856 175457 sql/event_log.go:115  [client=127.0.0.1:43210,user=root,n1] Event: "create_database", target: 50, info: {DatabaseName:data Statement:CREATE DATABASE data User:root}
I180112 19:51:13.589199 175015 storage/replica_command.go:819  [split,n1,s1,r20/1:/{Table/23-Max}] initiating a split of this range at key /Table/50 [r21]
I180112 19:51:13.592552 175457 sql/event_log.go:115  [client=127.0.0.1:43210,user=root,n1] Event: "create_table", target: 51, info: {TableName:bank Statement:CREATE TABLE bank (id INT PRIMARY KEY, balance INT, payload STRING, FAMILY (id, balance, payload)) User:root}
I180112 19:51:13.593272 175015 storage/replica_command.go:819  [split,n1,s1,r21/1:/{Table/50-Max}] initiating a split of this range at key /Table/51 [r22]
I180112 19:51:13.598043 175457 sql/event_log.go:115  [client=127.0.0.1:43210,user=root,n1] Event: "create_database", target: 52, info: {DatabaseName:data2 Statement:CREATE DATABASE data2 User:root}
I180112 19:51:13.598397 175015 storage/replica_command.go:819  [split,n1,s1,r22/1:/{Table/51-Max}] initiating a split of this range at key /Table/52 [r23]
I180112 19:51:13.602192 175457 sql/event_log.go:115  [client=127.0.0.1:43210,user=root,n1] Event: "create_table", target: 53, info: {TableName:data.t Statement:CREATE TABLE data.t (s STRING PRIMARY KEY) User:root}
I180112 19:51:13.602541 175015 storage/replica_command.go:819  [split,n1,s1,r23/1:/{Table/52-Max}] initiating a split of this range at key /Table/53 [r24]
I180112 19:51:13.614255 175590 ccl/storageccl/export.go:124  [n1,s1,r24/1:/{Table/53-Max}] export [/Table/53/1,/Table/53/2)
I180112 19:51:13.614365 175589 ccl/storageccl/export.go:124  [n1,s1,r22/1:/Table/5{1-2}] export [/Table/51/1,/Table/51/2)
panic: test timed out after 4m0s

goroutine 175572 [running]:
testing.startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1145 +0xf9
created by time.goFunc
	/usr/local/go/src/time/sleep.go:170 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200ae690, 0x23768b2, 0x2b, 0x2414418, 0x8d4f01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4200ae690)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4200ae690, 0xc420731db8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc4204c0180, 0x34314a0, 0x39, 0x39, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420abdf18, 0x2414788)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.TestMain(0xc420731f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/main_test.go:31 +0xda
main.main()
	github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/_test/_testmain.go:174 +0xdb

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3682700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 7 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 66 [select, locked to thread]:
runtime.gopark(0x24182e8, 0x0, 0x232e255, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc420c54f50, 0xc420536540)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 174984 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x345c9e0, 0xc42067a2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1469 +0x912
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422df08d0, 0xc4225c21b0, 0xc42169d860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175221 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420316200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1002 +0x166
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420316200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:703 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:694 +0x40d

goroutine 175115 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000186)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440500, 0xc4225c21b0, 0xc4204404e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175451 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc421e33aa0, 0x2416490, 0xc4225c21b0, 0x345c9e0, 0xc4206db620)
	<autogenerated>:1 +0x65
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc4225f6108, 0x345c9e0, 0xc4206db620, 0xc4225c21b0, 0x345bb60, 0xc421e33aa0, 0xc4206db650, 0xc4206db620, 0xc423bab6d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:119 +0x50
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func17(0x345c9e0, 0xc4206db620)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1248 +0x1f5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42024bf50, 0xc4225c21b0, 0xc420b39c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175195 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc421393590)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d63a0, 0xc4225c21b0, 0xc4203d6390)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 174998 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddToPurgatory.func2(0x345c9e0, 0xc422980ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:705 +0x434
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42039c2b0, 0xc4225c21b0, 0xc420240700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175116 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000187)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440570, 0xc4225c21b0, 0xc420440510)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175150 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000016b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc420e6a630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc420e6a630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440ae0, 0xc4225c21b0, 0xc420440ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175209 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Start.func1(0x345c9e0, 0xc421393620)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:417 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d65e0, 0xc4225c21b0, 0xc42183e1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175235 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc423bc4160)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x264
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x8fa

goroutine 174980 [IO wait]:
internal/poll.runtime_pollWait(0x2b3097800908, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc421377918, 0x72, 0xffffffffffffff00, 0x3446760, 0x3439638)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc421377918, 0xc421472400, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc421377900, 0xc421472400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc421377900, 0xc421472400, 0x400, 0x400, 0x3f3, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4204a48e0, 0xc421472400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
crypto/tls.(*block).readFromUntil(0xc422fce0f0, 0x2b30989c3c30, 0xc4204a48e0, 0x5, 0xc4204a48e0, 0x2b309782cc78)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc421ef8000, 0x2418417, 0xc421ef8120, 0xc421472005)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc421ef8000, 0xc42050b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc421a8a120, 0xc4211322e0, 0x5, 0x200, 0xc423cdb528, 0x8719e8, 0x2413e20)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x343b520, 0xc421a8a120, 0xc4211322e0, 0x5, 0x200, 0x5, 0x857967, 0xc423d6aa80, 0x20)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x343b520, 0xc421a8a120, 0xc4211322e0, 0x5, 0x200, 0x2b309783ed18, 0x0, 0xc4201ea4a8)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recvMessage(0xc4211322c0, 0xc423d6aa80, 0x202e320, 0x1, 0x36cdb98)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:947 +0x14e
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1Buf(0xc4211322c0, 0xc423d6aa80, 0xc4211322e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:997 +0x39
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1(0xc4211322c0, 0xc421ef8000, 0xc4211322e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1018 +0x85
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).readBindResponse(0xc4211322c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1633 +0x2f
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*stmt).exec(0xc42132aa00, 0xc42032fb20, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1270 +0x979
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*stmt).Exec(0xc42132aa00, 0xc42032fb20, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1228 +0xc3
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).Exec(0xc4211322c0, 0x2341a91, 0x13, 0xc42032fb20, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:893 +0x1f5
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).ExecContext(0xc4211322c0, 0x345c960, 0xc420016108, 0x2341a91, 0x13, 0xc420601500, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:43 +0x224
database/sql.ctxDriverExec(0x345c960, 0xc420016108, 0x2b30977b6248, 0xc4211322c0, 0x2341a91, 0x13, 0xc420601500, 0x1, 0x1, 0x871329, ...)
	/usr/local/go/src/database/sql/ctxutil.go:31 +0x277
database/sql.(*DB).execDC.func2()
	/usr/local/go/src/database/sql/sql.go:1253 +0x99
database/sql.withLock(0x3447520, 0xc4226c1e30, 0xc423cdbba8)
	/usr/local/go/src/database/sql/sql.go:2855 +0x65
database/sql.(*DB).execDC(0xc42297e000, 0x345c960, 0xc420016108, 0xc4226c1e30, 0xc423cdbc88, 0x2341a91, 0x13, 0xc420d4df28, 0x1, 0x1, ...)
	/usr/local/go/src/database/sql/sql.go:1252 +0x4e4
database/sql.(*DB).exec(0xc42297e000, 0x345c960, 0xc420016108, 0x2341a91, 0x13, 0xc423cdbf28, 0x1, 0x1, 0x1b14701, 0xc420024070, ...)
	/usr/local/go/src/database/sql/sql.go:1238 +0x13e
database/sql.(*DB).ExecContext(0xc42297e000, 0x345c960, 0xc420016108, 0x2341a91, 0x13, 0xc420d4df28, 0x1, 0x1, 0x309aea8, 0x49, ...)
	/usr/local/go/src/database/sql/sql.go:1216 +0xe1
database/sql.(*DB).Exec(0xc42297e000, 0x2341a91, 0x13, 0xc423cdbf28, 0x1, 0x1, 0x344bce0, 0xc42283b680, 0x0, 0x0)
	/usr/local/go/src/database/sql/sql.go:1230 +0x85
github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Exec(0xc4204a48d8, 0x346d760, 0xc4224fa000, 0x2341a91, 0x13, 0xc423cdbf28, 0x1, 0x1, 0x344bce0, 0xc42283b680)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:38 +0x9e
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestBackupRestoreIncrementalAddTableMissing(0xc4224fa000)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:2466 +0x351
testing.tRunner(0xc4224fa000, 0x2414418)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 175140 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000019c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e47e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e47e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440890, 0xc4225c21b0, 0xc420440880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175194 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc4213935c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6370, 0xc4225c21b0, 0xc4203d6360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175179 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x345c9e0, 0xc4220e49f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1282 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440ed0, 0xc4225c21b0, 0xc423adeec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175022 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc4204a8300, 0x345c9e0, 0xc42201c6c0, 0x1ac57879, 0xed1eb08b1, 0x0, 0xc4207d2100, 0xc4225c21b0, 0xc42230f180, 0xc42230f280)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:207 +0x226
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc42230f180, 0xc421392690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:269 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc421392690, 0xc42380a000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:322 +0x1ba
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x345c9e0, 0xc42201c6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:267 +0x1f5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422df00d0, 0xc4225c21b0, 0xc42037e1c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175177 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc4211c0400, 0x345c9e0, 0xc4220e4a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3850 +0x205
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x345c9e0, 0xc4220e4a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3836 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440e80, 0xc4225c21b0, 0xc420440e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175168 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000166)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e4870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e4870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440d60, 0xc4225c21b0, 0xc420440d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175208 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x345c9e0, 0xc4213935f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1028 +0x96d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d65d0, 0xc4225c21b0, 0xc42183e1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175128 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000191)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204406f0, 0xc4225c21b0, 0xc4204406e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175163 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000171)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc420e6a660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc420e6a660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440cc0, 0xc4225c21b0, 0xc420440cb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175214 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func2(0x345c9e0, 0xc420557950)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:205 +0x13b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42186f8a0, 0xc4225c21b0, 0xc420557830)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175153 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000016d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c4e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c4e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440b60, 0xc4225c21b0, 0xc420440b50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175155 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000168)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e48a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e48a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440ba0, 0xc4225c21b0, 0xc420440b90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175200 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc421393530)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d64e0, 0xc4225c21b0, 0xc4203d64d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175193 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc42201c930)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6340, 0xc4225c21b0, 0xc4203d6330)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175234 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x345c9e0, 0xc42201dcb0, 0xc42201dc50, 0xc4220b9fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc423bc4160)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8d5

goroutine 175132 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000015e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e4810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e4810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440770, 0xc4225c21b0, 0xc420440760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175457 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).Next(0xc421679db0, 0x345c920, 0xc420a39d40, 0xc4208d15d0, 0xc4208d1558, 0xc421679db0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:125 +0x172
github.com/cockroachdb/cockroach/pkg/sql.forEachRow(0x345c920, 0xc420a39d40, 0xc4208d15d0, 0xc4208d1558, 0x3453ae0, 0xc421679db0, 0xc42131c860, 0x0, 0xc4208d2310)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:2075 +0x5d
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execClassic(0xc42381c6c0, 0xc4208d1558, 0x3453ae0, 0xc421679db0, 0x34687e0, 0xc422ff45a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:2056 +0x28a
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execStmt(0xc42381c6c0, 0x345d8e0, 0xc4217e2500, 0xc42284ea20, 0x7, 0x7, 0xc42367e3a0, 0x13, 0x150927f2db7fbe3a, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:2217 +0x709
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execStmtInOpenTxn(0xc42381c6c0, 0xc4208d1300, 0x345d8e0, 0xc4217e2500, 0xc42284ea20, 0x7, 0x7, 0xc42367e3a0, 0x13, 0x150927f2db7fbe3a, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:1850 +0x758
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execSingleStatement(0xc42381c6c0, 0xc4208d1300, 0x345d8e0, 0xc4217e2500, 0xc42284ea20, 0x7, 0x7, 0xc42367e3a0, 0x13, 0x150927f2db7fbe3a, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:1319 +0x6c8
github.com/cockroachdb/cockroach/pkg/sql.runTxnAttempt(0xc42381c6c0, 0xc4208d1300, 0xc421679c70, 0x1, 0x1, 0xc42131d638, 0x1, 0x1, 0x0, 0x34614c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:1195 +0x41f
github.com/cockroachdb/cockroach/pkg/sql.runWithAutoRetry(0xc42381c6c0, 0xc4208d1300, 0xc421679c70, 0x1, 0x1, 0xed1eb0101, 0x0, 0xc42131d638, 0x3ff0000000000000, 0x970800, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:1020 +0x3aa
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execParsed(0xc42381c6c0, 0xc4208d1300, 0xc421679c70, 0x1, 0x1, 0xc42131d638, 0x0, 0xe14946, 0xbe8e624c640fb0bc)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:836 +0x138
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execPrepared(0xc42381c6c0, 0xc4208d1300, 0xc420b6a750, 0xc420daf638, 0x9, 0xc42061f176)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:678 +0x159
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).ExecutePreparedStatement(0xc42381c6c0, 0xc4208d1300, 0xc420b6a750, 0xc420daf638, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:655 +0x136
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).handleExecute(0xc422ff45a0, 0xc422ff45c8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/v3.go:964 +0x2f6
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).serve(0xc422ff45a0, 0x345c9e0, 0xc4223a3920, 0xc420314d60, 0x5400, 0x15000, 0xc42381ca78, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/v3.go:532 +0xb6f
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc42381c900, 0x345c920, 0xc420d7a780, 0x3468180, 0xc42354bc00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:421 +0xbae
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func17.1(0x34689c0, 0xc423822000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1252 +0x149
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc4225c21b0, 0x345c9e0, 0xc4206db620, 0xc4225f6108, 0x34689c0, 0xc423822000, 0xc4206db650)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:140 +0xa9
created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:137 +0x263

goroutine 175130 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000193)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e46f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e46f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440730, 0xc4225c21b0, 0xc420440720)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175127 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000018e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204406d0, 0xc4225c21b0, 0xc4204406c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175160 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000175)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc420e6a6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc420e6a6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440c60, 0xc4225c21b0, 0xc420440c50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175250 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420582000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:935 +0x264
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:230 +0x8fa

goroutine 175185 [semacquire]:
sync.runtime_notifyListWait(0xc4204a8100, 0xc400000003)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4204a80f0)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc4204a80c0, 0xc421e68000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc4225f6010, 0xc421e68000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc422029980, 0x2b30989bf9a8, 0xc420440160, 0x5, 0xc420440160, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc42354a700, 0x2418417, 0xc42354a820, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc42354a700, 0xc4227b4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc4222b6540, 0xc422c34038, 0x9, 0x9, 0x28, 0x21f9820, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x343b520, 0xc4222b6540, 0xc422c34038, 0x9, 0x9, 0x9, 0xc4228e1de8, 0x84730e, 0xc4228e1e97)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x343b520, 0xc4222b6540, 0xc422c34038, 0x9, 0x9, 0xd0cee4, 0xc42074c780, 0xc420740004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc422c34038, 0x9, 0x9, 0x343b520, 0xc4222b6540, 0x0, 0x0, 0xc42074c780, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc422c34000, 0xc42074c780, 0xc42074c780, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc421cf6180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1118 +0xbe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:266 +0xc80

goroutine 175012 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x345c9e0, 0xc420e6a150)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1189 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422df0dd0, 0xc4225c21b0, 0xc422df0dc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175455 [chan receive]:
database/sql.(*DB).connectionOpener(0xc42297e000)
	/usr/local/go/src/database/sql/sql.go:871 +0x53
created by database/sql.Open
	/usr/local/go/src/database/sql/sql.go:609 +0x1ee

goroutine 174981 [select]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc4225c21b0, 0xc423982d80, 0xc4217e80e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:125 +0x10d
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:123 +0xc9

goroutine 174929 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x345c9e0, 0xc4205e9590, 0xc4205e9530, 0xc4230d8fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc420582000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:227 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8d5

goroutine 175236 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x345c920, 0xc421323440, 0xc42074c660, 0xc4228e3fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:750 +0x2e4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc421cf6180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:297 +0xdd1

goroutine 175113 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000182)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204404a0, 0xc4225c21b0, 0xc420440490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175182 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x345c9e0, 0xc4220e4ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440f00, 0xc4225c21b0, 0xc420083a90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 174784 [select]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x345c9e0, 0xc422692240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x6b4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4221f8320, 0xc4225c21b0, 0xc4221f8310)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175139 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000199)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e47b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e47b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440860, 0xc4225c21b0, 0xc420440850)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175170 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000178)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440da0, 0xc4225c21b0, 0xc420440d90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175204 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc42067a180)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6560, 0xc4225c21b0, 0xc4203d6550)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175133 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000162)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc420e6a5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc420e6a5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440790, 0xc4225c21b0, 0xc420440780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175205 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc42201c8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6580, 0xc4225c21b0, 0xc4203d6570)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175199 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x345c9e0, 0xc421393500)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d64c0, 0xc4225c21b0, 0xc4203d64b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175126 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000017f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204406b0, 0xc4225c21b0, 0xc4204406a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175143 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc400000197)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440920, 0xc4225c21b0, 0xc420440910)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175122 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000018f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc4220e4630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc4220e4630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420440630, 0xc4225c21b0, 0xc420440620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175187 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x345c9e0, 0xc42201c7e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:685 +0x2dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6190, 0xc4225c21b0, 0xc42183e0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175014 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x345c9e0, 0xc421393440)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422df0000, 0xc4225c21b0, 0xc42037e080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175145 [semacquire]:
sync.runtime_notifyListWait(0xc422b342d0, 0xc40000019b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422b342c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421b08090, 0x345c9e0, 0xc42201c3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x345c9e0, 0xc42201c3f0)
	/go/src/

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 12, 2018
@benesch benesch mentioned this issue Jan 16, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity