Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/sqlccl: TestBackupRestoreControlJob failed under stress #21846

Closed
cockroach-teamcity opened this issue Jan 27, 2018 · 3 comments · Fixed by #22337
Closed

ccl/sqlccl: TestBackupRestoreControlJob failed under stress #21846

cockroach-teamcity opened this issue Jan 27, 2018 · 3 comments · Fixed by #22337
Assignees
@benesch
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/2c85c61e7bc64c7c62aa7e25e9e762180f4e345a

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495189&tab=buildLog


I180127 10:11:00.463018 20888601 server/config.go:518  [n?] 1 storage engine initialized
I180127 10:11:00.463192 20888601 server/config.go:521  [n?] RocksDB cache size: 128 MiB
I180127 10:11:00.463290 20888601 server/config.go:521  [n?] store 0: in-memory, size 0 B
I180127 10:11:00.471784 20888601 server/node.go:374  [n?] **** cluster 88a8e05f-e9d9-478e-be08-f1c424a1b151 has been created
I180127 10:11:00.471899 20888601 server/server.go:973  [n?] **** add additional nodes by specifying --join=127.0.0.1:41653
I180127 10:11:00.476216 20888601 storage/store.go:1300  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180127 10:11:00.481402 20890332 kv/dist_sender.go:354  [n1] unable to determine this node's attributes for replica selection; node is most likely bootstrapping
I180127 10:11:00.484833 20888601 server/node.go:504  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.3 KiB), ranges=1, leases=1, writes=0.00, bytesPerReplica={p10=3346.00 p25=3346.00 p50=3346.00 p75=3346.00 p90=3346.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180127 10:11:00.491005 20888601 server/node.go:352  [n1] node ID 1 initialized
I180127 10:11:00.491219 20888601 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:41653" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:11 > 
I180127 10:11:00.491462 20888601 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180127 10:11:00.491654 20888601 server/node.go:645  [n1] connecting to gossip network to verify cluster ID...
I180127 10:11:00.491812 20888601 server/node.go:670  [n1] node connected via gossip and verified as part of cluster "88a8e05f-e9d9-478e-be08-f1c424a1b151"
I180127 10:11:00.492008 20888601 server/node.go:446  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180127 10:11:00.492296 20888601 server/server.go:1174  [n1] starting https server at 127.0.0.1:38991
I180127 10:11:00.492604 20888601 server/server.go:1175  [n1] starting grpc/postgres server at 127.0.0.1:41653
I180127 10:11:00.492697 20888601 server/server.go:1176  [n1] advertising CockroachDB node at 127.0.0.1:41653
W180127 10:11:00.492834 20888601 sql/jobs/registry.go:222  [n1] unable to get node liveness: node not in the liveness table
I180127 10:11:00.637002 20889608 storage/replica_command.go:835  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
E180127 10:11:00.661474 20890460 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180127 10:11:00.676631 20888601 sql/event_log.go:114  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180127 10:11:00.687491 20892961 storage/replica_command.go:835  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
W180127 10:11:00.717934 20894117 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "unnamed" id=83dabd8f key=/Table/SystemConfigSpan/Start rw=true pri=0.04597018 iso=SERIALIZABLE stat=PENDING epo=0 ts=1517047860.676848315,0 orig=1517047860.676848315,0 max=1517047860.676848315,0 wto=false rop=false seq=3
I180127 10:11:00.718677 20894312 storage/replica_command.go:835  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
I180127 10:11:00.746887 20895533 storage/replica_command.go:835  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
W180127 10:11:00.766432 20895519 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "unnamed" id=83dabd8f key=/Table/SystemConfigSpan/Start rw=true pri=0.04597018 iso=SERIALIZABLE stat=PENDING epo=0 ts=1517047860.676848315,0 orig=1517047860.676848315,0 max=1517047860.676848315,0 wto=false rop=false seq=3
E180127 10:11:00.768826 20890460 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180127 10:11:00.858850 20897300 storage/replica_command.go:835  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
E180127 10:11:00.872574 20890460 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180127 10:11:00.921363 20888601 sql/lease.go:349  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-27 10:11:00.920741323 +0000 UTC
I180127 10:11:00.968998 20899346 storage/replica_command.go:835  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
E180127 10:11:00.976312 20890460 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180127 10:11:01.035174 20900216 storage/replica_command.go:835  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180127 10:11:01.133052 20903115 storage/replica_command.go:835  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180127 10:11:01.178192 20905073 storage/replica_command.go:835  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180127 10:11:01.245776 20907659 storage/replica_command.go:835  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180127 10:11:01.278923 20909292 storage/replica_command.go:835  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180127 10:11:01.287923 20888601 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
W180127 10:11:01.334227 20911541 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "split" id=6d2fddc9 key=/Local/Range/Table/14/RangeDescriptor rw=true pri=0.01236158 iso=SERIALIZABLE stat=PENDING epo=0 ts=1517047861.279000951,0 orig=1517047861.279000951,0 max=1517047861.279000951,0 wto=false rop=false seq=3
I180127 10:11:01.354691 20912959 storage/replica_command.go:835  [split,n1,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180127 10:11:01.406022 20888601 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
I180127 10:11:01.458327 20915278 storage/replica_command.go:835  [split,n1,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180127 10:11:01.510528 20917345 storage/replica_command.go:835  [split,n1,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180127 10:11:01.565093 20888601 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180127 10:11:01.615410 20921156 storage/replica_command.go:835  [split,n1,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
I180127 10:11:01.748719 20924436 storage/replica_command.go:835  [split,n1,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
I180127 10:11:01.848116 20927266 storage/replica_command.go:835  [split,n1,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
I180127 10:11:01.997706 20931873 storage/replica_command.go:835  [split,n1,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
I180127 10:11:02.033397 20933012 storage/replica_command.go:835  [split,n1,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
I180127 10:11:02.157208 20888601 sql/event_log.go:114  [n1] Event: "alter_table", target: 4, info: {TableName:users Statement:ALTER TABLE system.users ADD COLUMN IF NOT EXISTS "isRole" BOOL NOT NULL DEFAULT false User:node MutationID:0 CascadeDroppedViews:[]}
I180127 10:11:02.291351 20888601 sql/lease.go:349  [n1] publish: descID=4 (users) version=2 mtime=2018-01-27 10:11:02.237713909 +0000 UTC
W180127 10:11:02.325086 20888601 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 10:11:02.384843 20888601 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 10:11:02.502801 20888601 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 10:11:02.692509 20888601 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 10:11:03.129031 20888601 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 10:11:03.898244 20888601 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
F180127 10:11:03.900435 20888601 server/server.go:1242  [n1] failed to run migration "add system.users isRole column and create admin role": column "isRole" does not exist
goroutine 20888601 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x3753e01, 0x150da2fe9b0cf27e, 0x31980b8, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x3753e00, 0xc400000004, 0x31980b8, 0x10, 0x4da, 0xc4242e1e80, 0x73)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x59d
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x352c2e0, 0xc4238bcc90, 0x4, 0x2, 0x0, 0x0, 0xc42a0bba38, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x308
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x352c2e0, 0xc4238bcc90, 0x1, 0xc400000004, 0x0, 0x0, 0xc42a0bba38, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x352c2e0, 0xc4238bcc90, 0xc42a0bba38, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x76
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc425363600, 0x352c2e0, 0xc4238bcc90, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1242 +0x3257
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc4251e2e40, 0x350f4e0, 0xc42b84b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x350b860, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x137
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x350f4e0, 0xc42b84b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x350b860, 0xc4251e2ae0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0xef
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x353cfe0, 0xc42457c000, 0x350f4e0, 0xc42b84b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:172 +0x55
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).doAddServer(0xc4251e2c60, 0x353cfe0, 0xc42457c000, 0x350f4e0, 0xc42b84b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:247 +0x112
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster(0x353cfe0, 0xc42457c000, 0x3, 0x350f4e0, 0xc42b84b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:155 +0x55a
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetupWithParams(0x353cfe0, 0xc42457c000, 0x3, 0x3e8, 0x24aae98, 0x350f4e0, 0xc42b84b4a0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:86 +0xfe
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestBackupRestoreControlJob(0xc42457c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:850 +0x25f
testing.tRunner(0xc42457c000, 0x24aad48)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4205245a0, 0x23e5c3e, 0x1b, 0x24aad48, 0x900c01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4205245a0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4205245a0, 0xc420647db8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc4204a0ec0, 0x3500520, 0x3a, 0x3a, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc42094bf18, 0xc42046a5b0)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.TestMain(0xc420647f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/main_test.go:31 +0xda
main.main()
	github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/_test/_testmain.go:176 +0xdb

goroutine 46 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3753e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1053 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 67 [syscall, 7 minutes]:
os/signal.signal_recv(0xc420039fc0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 145 [select, locked to thread]:
runtime.gopark(0x24aedb8, 0x0, 0x23c2229, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc420483750, 0xc4201f9da0)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 20889856 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000187)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762330, 0xc4245accf0, 0xc432762320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889933 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000155)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5860)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762e70, 0xc4245accf0, 0xc432762e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889945 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000162)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762ff0, 0xc4245accf0, 0xc432762fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888733 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x352c2e0, 0xc42b9bf350)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422631810, 0xc4245accf0, 0xc422631800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889874 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc40000019a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327625b0, 0xc4245accf0, 0xc4327625a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889884 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001a9)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762740, 0xc4245accf0, 0xc432762730)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888812 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x352c2e0, 0xc42b9bfb60)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1189 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421726010, 0xc4245accf0, 0xc421726000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888738 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x352c2e0, 0xc42b9bf440)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4226318b0, 0xc4245accf0, 0xc4226318a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889967 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000166)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327632d0, 0xc4245accf0, 0xc4327632c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889890 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001aa)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762800, 0xc4245accf0, 0xc4327627f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889905 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x1b9)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762a60, 0xc4245accf0, 0xc432762a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889969 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x178)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42bfa21b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42bfa21b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763310, 0xc4245accf0, 0xc432763300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889828 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x352c2e0, 0xc42978acc0, 0xc42978ac60, 0xc42b8e0fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x3c6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc425f52300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:261 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0xee9

goroutine 20889873 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc40000019b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762590, 0xc4245accf0, 0xc432762580)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888740 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x352c2e0, 0xc42b9bf230)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4226318f0, 0xc4245accf0, 0xc4226318e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889865 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000190)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762470, 0xc4245accf0, 0xc432762460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889926 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000156)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c56e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c56e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762d90, 0xc4245accf0, 0xc432762d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889920 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001c7)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5440)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5440)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762cd0, 0xc4245accf0, 0xc432762cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889949 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000164)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763070, 0xc4245accf0, 0xc432763060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889892 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001b0)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762850, 0xc4245accf0, 0xc432762840)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889897 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001ae)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762900, 0xc4245accf0, 0xc4327628f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889977 [semacquire]:
sync.runtime_Semacquire(0xc4247a9dfc)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4247a9df0)
	/usr/local/go/src/sync/waitgroup.go:131 +0x72
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc4247a9d80, 0x352c2e0, 0xc42bfa2420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x352c2e0, 0xc42bfa2420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3819 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763420, 0xc4245accf0, 0xc432763410)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889911 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001b7)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762bb0, 0xc4245accf0, 0xc432762b90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889957 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc40000016e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763170, 0xc4245accf0, 0xc432763160)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20890302 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x352c2e0, 0xc4297617a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424cb67f0, 0xc4245accf0, 0xc42441dcc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889901 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001b4)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762990, 0xc4245accf0, 0xc432762970)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889952 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000167)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327630d0, 0xc4245accf0, 0xc4327630c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20890426 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Start.func1(0x352c2e0, 0xc42aa76db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:418 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763f80, 0xc4245accf0, 0xc429170e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889922 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001c8)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c54a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c54a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762d10, 0xc4245accf0, 0xc432762d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889906 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001bb)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762a80, 0xc4245accf0, 0xc432762a70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889863 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc40000018f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c48a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c48a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762430, 0xc4245accf0, 0xc432762420)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20890409 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x352c2e0, 0xc42438ca50)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:698 +0x2dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763dc0, 0xc4245accf0, 0xc429170c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889891 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001b2)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762830, 0xc4245accf0, 0xc432762820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889898 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001af)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762920, 0xc4245accf0, 0xc432762910)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888811 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x352c2e0, 0xc42b9bfb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1160 +0x46b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422631ff0, 0xc4245accf0, 0xc422631fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889960 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x16f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327631d0, 0xc4245accf0, 0xc4327631c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888784 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func6(0x352c2e0, 0xc42b9bf8f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:810 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422631cc0, 0xc4245accf0, 0xc429558dc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888777 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x352c2e0, 0xc42b9bf5c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1464 +0x1f3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422631b90, 0xc4245accf0, 0xc429ac4990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889955 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc40000016b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763130, 0xc4245accf0, 0xc432763120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889889 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x1a8)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327627e0, 0xc4245accf0, 0xc4327627d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889850 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc40000018e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762270, 0xc4245accf0, 0xc432762260)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888730 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x352c2e0, 0xc42b9bf2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4226317b0, 0xc4245accf0, 0xc4226317a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889947 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000165)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763030, 0xc4245accf0, 0xc432763020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889983 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x352c2e0, 0xc42bfa2bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1404 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327634c0, 0xc4245accf0, 0xc42a3ff310)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889924 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001ca)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5650)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5650)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762d50, 0xc4245accf0, 0xc432762d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889909 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001c1)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c51d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c51d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762b30, 0xc4245accf0, 0xc432762b20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888732 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x352c2e0, 0xc42b9bf320)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4226317f0, 0xc4245accf0, 0xc4226317e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889968 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000179)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42bfa2180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42bfa2180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327632f0, 0xc4245accf0, 0xc4327632e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20886784 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x352c2e0, 0xc42b9bf860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc428224320, 0xc4245accf0, 0xc42e042be0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889877 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x19e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762650, 0xc4245accf0, 0xc432762630)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889854 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x185)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c46f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c46f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327622f0, 0xc4245accf0, 0xc4327622e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889878 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x19f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c4ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c4ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762670, 0xc4245accf0, 0xc432762660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20890721 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc429abcbc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x156
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x197

goroutine 20889824 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42e290c80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1234 +0x25f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc42e290c80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0xeb

goroutine 20889938 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x158)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5980)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5980)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762f10, 0xc4245accf0, 0xc432762f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889832 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x352c2e0, 0xc42a23a840, 0xc42a23a780, 0xc4298b4fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x3c6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc429406300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:261 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0xee9

goroutine 20889966 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000175)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42bfa20c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42bfa20c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4327632b0, 0xc4245accf0, 0xc4327632a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889929 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001cd)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c57a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c57a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762df0, 0xc4245accf0, 0xc432762de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889970 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0x174)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42bfa21e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42bfa21e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432763330, 0xc4245accf0, 0xc432763320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20888781 [IO wait]:
internal/poll.runtime_pollWait(0x7f114d37e430, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4247a9998, 0x72, 0xc4215c8500, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4247a9998, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc4247a9980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc4247a9980, 0x7f114d3db458, 0x0, 0x104d601)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4226f2470, 0xc4215c86f8, 0x10, 0x1541917)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4226f2470, 0x24ad1b0, 0xc429558d80, 0xc424eb2ae0, 0x0)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc429558d80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x95
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x352c2e0, 0xc42b9bf8c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:787 +0x2f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422631c90, 0xc4245accf0, 0xc429aa4ca0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889904 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc4000001b8)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c50e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c50e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc432762a00, 0xc4245accf0, 0xc4327629e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 20889937 [semacquire]:
sync.runtime_notifyListWait(0xc422766750, 0xc400000159)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422766740)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4247a9d80, 0x352c2e0, 0xc42e2c5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x352c2e0, 0xc42e2c5950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 27, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/40a7b4e6677987d8235d5b607832c3dbf20a8f2b

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495848&tab=buildLog

@mberhault mberhault mentioned this issue Feb 2, 2018
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Feb 2, 2018
@nvanbenschoten
sql: don't ignore job status update errors
65622ff 
Fixes cockroachdb#21828.
Fixes cockroachdb#21846.
Fixes cockroachdb#21844.
Fixes cockroachdb#21808.

See discussion in cockroachdb#21802.

We were previously ignoring the errors returned by job operations in
a few places. This meant that we would fail to propagate retryable txn
errors when using the jobs API and we could think a txn succeeded when
it didn't. This change fixes that.

In doing so, it makes schema changes fail when they see errors due to
job status updates. I don't see a compelling reason why the old behavior
was better and I suspect it was allowing subtle inconsistencies.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Feb 2, 2018
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/6a1cc888163fd154f80c1748837f2a6e10e62fe4

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=505494&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c93d304f8b31b5b0678a46d36a7b751cf50fedb1

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=513425&tab=buildLog

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Feb 12, 2018
@nvanbenschoten
sql: don't ignore job status update errors
9edbed9 
Fixes cockroachdb#21828.
Fixes cockroachdb#21846.
Fixes cockroachdb#21844.
Fixes cockroachdb#21808.

See discussion in cockroachdb#21802.

We were previously ignoring the errors returned by job operations in
a few places. This meant that we would fail to propagate retryable txn
errors when using the jobs API and we could think a txn succeeded when
it didn't. This change fixes that.

In doing so, it makes schema changes fail when they see errors due to
job status updates. I don't see a compelling reason why the old behavior
was better and I suspect it was allowing subtle inconsistencies.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benesch benesch

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: don't ignore job status update errors nvanbenschoten/cockroach
2 participants
@benesch @cockroach-teamcity