Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on release-note-script-tweak: testrace/TestBackupRestoreResume, testrace/TestBackupRestoreResume #21828

Closed
cockroach-teamcity opened this issue Jan 26, 2018 · 0 comments · Fixed by #22337
Closed

teamcity: failed tests on release-note-script-tweak: testrace/TestBackupRestoreResume, testrace/TestBackupRestoreResume #21828

cockroach-teamcity opened this issue Jan 26, 2018 · 0 comments · Fixed by #22337
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#494466:

--- FAIL: testrace/TestBackupRestoreResume (0.000s)
Test ended in panic.

------- Stdout: -------
W180126 18:47:31.730267 140886 server/status/runtime.go:109  Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180126 18:47:31.799457 140886 server/config.go:518  [n?] 1 storage engine initialized
I180126 18:47:31.799577 140886 server/config.go:521  [n?] RocksDB cache size: 128 MiB
I180126 18:47:31.799637 140886 server/config.go:521  [n?] store 0: in-memory, size 0 B
I180126 18:47:31.867563 140886 server/node.go:374  [n?] **** cluster 03a31c01-f588-454a-8ae7-fff91e7d2403 has been created
I180126 18:47:31.867758 140886 server/server.go:973  [n?] **** add additional nodes by specifying --join=127.0.0.1:43651
I180126 18:47:31.901632 140886 storage/store.go:1300  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180126 18:47:31.902460 140886 server/node.go:504  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.3 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=3346.00 p25=3346.00 p50=3346.00 p75=3346.00 p90=3346.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180126 18:47:31.902921 140886 server/node.go:352  [n1] node ID 1 initialized
I180126 18:47:31.903320 140886 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:43651" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:11 > 
I180126 18:47:31.928979 140886 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180126 18:47:31.932003 140886 server/node.go:645  [n1] connecting to gossip network to verify cluster ID...
I180126 18:47:31.932361 140886 server/node.go:670  [n1] node connected via gossip and verified as part of cluster "03a31c01-f588-454a-8ae7-fff91e7d2403"
I180126 18:47:31.933077 140886 server/node.go:446  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180126 18:47:31.935560 140886 server/server.go:1174  [n1] starting https server at 127.0.0.1:44351
I180126 18:47:31.935963 140886 server/server.go:1175  [n1] starting grpc/postgres server at 127.0.0.1:43651
I180126 18:47:31.936062 140886 server/server.go:1176  [n1] advertising CockroachDB node at 127.0.0.1:43651
W180126 18:47:31.936423 140886 sql/jobs/registry.go:222  [n1] unable to get node liveness: node not in the liveness table
I180126 18:47:31.984128 140927 storage/replica_command.go:835  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
E180126 18:47:32.075312 141178 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180126 18:47:32.156146 141219 storage/replica_command.go:835  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
E180126 18:47:32.226075 141178 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180126 18:47:32.459902 140886 sql/event_log.go:114  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180126 18:47:32.495246 141221 storage/replica_command.go:835  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
E180126 18:47:32.525890 141178 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180126 18:47:32.587706 140886 sql/lease.go:349  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-26 18:47:32.58499054 +0000 UTC
I180126 18:47:32.617247 140974 storage/replica_command.go:835  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
E180126 18:47:32.662724 141178 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180126 18:47:32.730266 141252 storage/replica_command.go:835  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
E180126 18:47:32.779224 141178 sql/jobs/registry.go:208  error while adopting jobs: relation "system.jobs" does not exist
I180126 18:47:32.922332 141284 storage/replica_command.go:835  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180126 18:47:33.002741 140886 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
I180126 18:47:33.043573 141298 storage/replica_command.go:835  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180126 18:47:33.187009 141301 storage/replica_command.go:835  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
W180126 18:47:33.352397 141288 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "split" id=3c26dcd1 key=/Local/Range/Table/11/RangeDescriptor rw=true pri=0.02297605 iso=SERIALIZABLE stat=PENDING epo=0 ts=1516992453.187309983,0 orig=1516992453.187309983,0 max=1516992453.187309983,0 wto=false rop=false seq=3
I180126 18:47:33.382902 141278 storage/replica_command.go:835  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
W180126 18:47:33.540067 141340 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "split" id=2dbf0168 key=/Local/Range/Table/12/RangeDescriptor rw=true pri=0.06636442 iso=SERIALIZABLE stat=PENDING epo=0 ts=1516992453.383093019,0 orig=1516992453.383093019,0 max=1516992453.383093019,0 wto=false rop=false seq=3
I180126 18:47:33.547424 141261 storage/replica_command.go:835  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180126 18:47:33.548846 140886 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
I180126 18:47:33.700123 141344 storage/replica_command.go:835  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180126 18:47:33.840482 140886 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180126 18:47:33.897428 141369 storage/replica_command.go:835  [split,n1,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180126 18:47:34.003383 141399 storage/replica_command.go:835  [split,n1,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
W180126 18:47:34.086668 141403 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "split" id=a66f3b77 key=/Local/Range/Table/16/RangeDescriptor rw=true pri=0.03153123 iso=SERIALIZABLE stat=PENDING epo=0 ts=1516992454.020737051,0 orig=1516992454.020737051,0 max=1516992454.020737051,0 wto=false rop=false seq=3
I180126 18:47:34.101464 141296 storage/replica_command.go:835  [split,n1,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180126 18:47:34.187227 141433 storage/replica_command.go:835  [split,n1,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
I180126 18:47:34.340633 141391 storage/replica_command.go:835  [split,n1,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
I180126 18:47:34.380611 140886 sql/event_log.go:114  [n1] Event: "alter_table", target: 4, info: {TableName:users Statement:ALTER TABLE system.users ADD COLUMN IF NOT EXISTS "isRole" BOOL NOT NULL DEFAULT false User:node MutationID:0 CascadeDroppedViews:[]}
W180126 18:47:34.427817 141438 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "split" id=179aa0e8 key=/Local/Range/Table/19/RangeDescriptor rw=true pri=0.04685451 iso=SERIALIZABLE stat=PENDING epo=0 ts=1516992454.340857847,0 orig=1516992454.340857847,0 max=1516992454.340857847,0 wto=false rop=false seq=3
I180126 18:47:34.441051 141424 storage/replica_command.go:835  [split,n1,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
I180126 18:47:34.458423 140886 sql/lease.go:349  [n1] publish: descID=4 (users) version=2 mtime=2018-01-26 18:47:34.456696432 +0000 UTC
I180126 18:47:34.581636 141352 storage/replica_command.go:835  [split,n1,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
W180126 18:47:34.645017 140886 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 18:47:34.710085 140886 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
I180126 18:47:34.741100 141467 storage/replica_command.go:835  [split,n1,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
W180126 18:47:34.850308 140886 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 18:47:35.064913 140886 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 18:47:35.514186 140886 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 18:47:36.368659 140886 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
F180126 18:47:36.375338 140886 server/server.go:1242  [n1] failed to run migration "add system.users isRole column and create admin role": column "isRole" does not exist
goroutine 140886 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x4e96e01, 0x0, 0x0, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x44c88e0, 0xc400000004, 0x3f025a4, 0x10, 0x4da, 0xc4204efb00, 0x73)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x6c8
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x42a0580, 0xc4238e8a20, 0x4, 0x2, 0x0, 0x0, 0xc421065b98, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x324
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x42a0580, 0xc4238e8a20, 0x1, 0xc400000004, 0x0, 0x0, 0xc421065b98, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x42a0580, 0xc4238e8a20, 0xc421065b98, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x88
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc4218da000, 0x42a0580, 0xc4238e8a20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1242 +0x5f1f
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc42124d140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x24d
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0x12a
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x42b15c0, 0xc4230eeb40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:172 +0x66
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).doAddServer(0xc42034e0e0, 0x42b15c0, 0xc4230eeb40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:247 +0x154
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster(0x42b15c0, 0xc4230eeb40, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:155 +0x693
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetupWithParams(0x42b15c0, 0xc4230eeb40, 0x3, 0x3e8, 0x2efecf8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:86 +0x11f
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetup(0x42b15c0, 0xc4230eeb40, 0x3, 0x3e8, 0x2efecf8, 0xc422fd8618, 0xc4218046e8, 0x8e5c86, 0x45a, 0x2c193f51, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:151 +0xa7
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestBackupRestoreResume(0xc4230eeb40)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:705 +0x12f
testing.tRunner(0xc4230eeb40, 0x2efec28)
	/usr/local/go/src/testing/testing.go:746 +0x16d
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x569

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200a2690, 0x2e303bb, 0x17, 0x2efec28, 0xc420687c01)
	/usr/local/go/src/testing/testing.go:790 +0x59b
testing.runTests.func1(0xc4200a2690)
	/usr/local/go/src/testing/testing.go:1004 +0xa8
testing.tRunner(0xc4200a2690, 0xc420687d50)
	/usr/local/go/src/testing/testing.go:746 +0x16d
testing.runTests(0xc4202ac1e0, 0x4272720, 0x3a, 0x3a, 0xc42001a4b0)
	/usr/local/go/src/testing/testing.go:1002 +0x522
testing.(*M).Run(0xc4208c5f20, 0x18)
	/usr/local/go/src/testing/testing.go:921 +0x207
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.TestMain(0xc420687f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/main_test.go:31 +0x180
main.main()
	github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/_test/_testmain.go:176 +0x1ce

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x44c88e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1053 +0x8f
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0x13a

goroutine 15 [syscall, 11 minutes]:
os/signal.signal_recv(0x8e46b1)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 35 [select, locked to thread]:
runtime.gopark(0x2f02d60, 0x0, 0x2e14d53, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0xfd
runtime.selectgo(0xc420bc4f50, 0xc420421500)
	/usr/local/go/src/runtime/select.go:395 +0x12c3
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x1a7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 141172 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x42a0580, 0xc4212c1e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1005 +0xdc1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423fe0df0, 0xc420fd5560, 0xc4221480a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141176 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x42a0580, 0xc4205c7f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:373 +0x2bd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423fe0f00, 0xc420fd5560, 0xc4227c9e50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141149 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x179)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4205c6060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4205c6060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4252783c0, 0xc420fd5560, 0xc4252783b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141161 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x42a0580, 0xc42175f230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1270 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278650, 0xc420fd5560, 0xc421b2ec60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141111 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x188)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbcb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbcb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919c70, 0xc420fd5560, 0xc421919c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141116 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x162)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9d080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9d080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919d90, 0xc420fd5560, 0xc421919d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141112 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x193)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919ca0, 0xc420fd5560, 0xc421919c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 140907 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421cf1c50)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918aa0, 0xc420fd5560, 0xc421918a70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141129 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x19a)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc422408900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc422408900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919f40, 0xc420fd5560, 0xc421919f30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141117 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x196)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9cdb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9cdb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919db0, 0xc420fd5560, 0xc421919da0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141209 [select]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x42a0580, 0xc421ebd680)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x911
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278c20, 0xc420fd5560, 0xc425278c10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141109 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x180)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc42175ef60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc42175ef60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919c30, 0xc420fd5560, 0xc421919bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141163 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x42a0580, 0xc4212c0c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1404 +0x4dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278670, 0xc420fd5560, 0xc42204d860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141143 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x173)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4212c0b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4212c0b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278160, 0xc420fd5560, 0xc425278150)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141173 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Start.func1(0x42a0580, 0xc4212c1b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:418 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423fe0e20, 0xc420fd5560, 0xc4221480e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141134 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x166)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9d1d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9d1d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919ff0, 0xc420fd5560, 0xc421919fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141188 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddToPurgatoryLocked.func2(0x42a0580, 0xc42021eed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:788 +0x557
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4233cb7e0, 0xc420fd5560, 0xc423487d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141192 [IO wait]:
internal/poll.runtime_pollWait(0x2ad1bd37a2d0, 0x72, 0x4289c80)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc4257a8b98, 0x72, 0x4289c00, 0x427c6a8, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc4257a8b98, 0xc420921800, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc4257a8b80, 0xc420921800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc4257a8b80, 0xc420921800, 0x400, 0x400, 0xc42285b2a0, 0x47, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420a5e6f8, 0xc420921800, 0x400, 0x400, 0x3a, 0x3f3, 0x2f02c50)
	/usr/local/go/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc420a5e6f8, 0xc420921800, 0x400, 0x400, 0x8e56d2, 0xc4bb2a, 0xc42021e210)
	<autogenerated>:1 +0x6a
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4227d2140, 0xc420921800, 0x400, 0x400, 0xc4202b6f60, 0xc4232696f8, 0xc4bff6)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x25b
crypto/tls.(*block).readFromUntil(0xc42021e210, 0x2ad1be583028, 0xc4227d2140, 0x5, 0xc4227d2140, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x11d
crypto/tls.(*Conn).readRecord(0xc42285b180, 0x2f02e17, 0xc42285b2a0, 0xc42092180d)
	/usr/local/go/src/crypto/tls/conn.go:590 +0x142
crypto/tls.(*Conn).Read(0xc42285b180, 0xc423a14000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x170
bufio.(*Reader).Read(0xc420fca180, 0xc422cfa118, 0x9, 0x9, 0xc422b61f58, 0xffffffff, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x7b8
io.ReadAtLeast(0x427e580, 0xc420fca180, 0xc422cfa118, 0x9, 0x9, 0x9, 0xc423269c70, 0x8bfd1c, 0xc422b61f60)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x427e580, 0xc420fca180, 0xc422cfa118, 0x9, 0x9, 0xc422b61f50, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc422cfa118, 0x9, 0x9, 0x427e580, 0xc420fca180, 0x0, 0x0, 0x3120a1ced0, 0x31)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc422cfa0e0, 0xc420a1ced0, 0xc420a1ced0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x118
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc421d0e180, 0xc422d4e060, 0x2f02010)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:395 +0x87
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc423865600, 0x42abc00, 0xc421d0e180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:634 +0x184
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:591 +0x54
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4203fe020, 0xc423865600, 0x2ad1bb383328, 0xc421d0e180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:599 +0x35
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:598 +0x832

goroutine 141155 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x16e)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbe90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbe90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278520, 0xc420fd5560, 0xc425278500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141004 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x42a04c0, 0xc423326700, 0xc423b22690, 0xc4255e2fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x474
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc4221f1980)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:305 +0xc4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:304 +0x17f6

goroutine 141139 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x16a)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4212c0b10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4212c0b10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4252780a0, 0xc420fd5560, 0xc425278080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141013 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421cf1d70)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918c70, 0xc420fd5560, 0xc421918c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141019 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x42a0580, 0xc421a20030)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:775 +0x84
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918f50, 0xc420fd5560, 0xc4203e30a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141150 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x17a)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4205c6000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4205c6000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4252783f0, 0xc420fd5560, 0xc4252783e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141024 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4203e3040, 0x2f01fc8, 0xc423865600, 0x42ac920, 0xc4227d2140)
	<autogenerated>:1 +0x8a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc423865600, 0x429f700, 0xc4203e3040, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:504 +0x265
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x42a0580, 0xc421ccff50)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:822 +0x8d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918ff0, 0xc420fd5560, 0xc4203e3180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141033 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x183)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc42175f0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc42175f0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919920, 0xc420fd5560, 0xc421919900)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141031 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x184)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc42175f080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc42175f080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919830, 0xc420fd5560, 0xc421919820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141168 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x42a0580, 0xc421ebd3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1e6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278a90, 0xc420fd5560, 0xc421b2ee20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141157 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x17c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4212c0ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4212c0ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278590, 0xc420fd5560, 0xc425278570)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141040 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x185)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9c990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9c990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919a60, 0xc420fd5560, 0xc421919a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141164 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x42a0580, 0xc42175f260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1404 +0x4dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278680, 0xc420fd5560, 0xc42204d8b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141038 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x18e)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc42175f170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc42175f170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919a10, 0xc420fd5560, 0xc4219199f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141148 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x178)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbf20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbf20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278380, 0xc420fd5560, 0xc425278340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141119 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x197)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919df0, 0xc420fd5560, 0xc421919de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141213 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x42a04c0, 0xc42490d8c0, 0xc421964150, 0xc420bc0fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x474
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc424bdec00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:305 +0xc4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:304 +0x17f6

goroutine 140912 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421ccfbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918b70, 0xc420fd5560, 0xc421918b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141029 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x42a0580, 0xc42175efc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:156 +0x63
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4219197e0, 0xc420fd5560, 0xc421b2ec40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141147 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x17b)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbf50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbf50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278330, 0xc420fd5560, 0xc4252782c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141114 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x187)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbdd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbdd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919cf0, 0xc420fd5560, 0xc421919ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141115 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x191)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4211bbe30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4211bbe30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919d30, 0xc420fd5560, 0xc421919d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141030 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x181)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc42175f050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc42175f050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919810, 0xc420fd5560, 0xc421919800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 140910 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421ccfbc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918b10, 0xc420fd5560, 0xc421918b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 140913 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421cf1d10)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918b90, 0xc420fd5560, 0xc421918b80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141037 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x189)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9cb10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9cb10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4219199e0, 0xc420fd5560, 0xc4219199d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141137 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x16d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc4212c0ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc4212c0ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278050, 0xc420fd5560, 0xc425278040)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141127 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x19e)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9cf00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9cf00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421919f00, 0xc420fd5560, 0xc421919ef0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141140 [semacquire]:
sync.runtime_notifyListWait(0xc420712dd0, 0x16c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc420712dc0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42142ab40, 0x42a0580, 0xc420b9d2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x42a0580, 0xc420b9d2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4252780c0, 0xc420fd5560, 0xc4252780b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141170 [select]:
github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x42a0580, 0xc422408d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:122 +0x1a4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423fe0dc0, 0xc420fd5560, 0xc423fe0db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 140991 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x42a0580, 0xc422408ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:698 +0x39e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423fe0c80, 0xc420fd5560, 0xc422148000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141190 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x42a0580, 0xc420a1d020, 0xc420a1cfc0, 0xc42003cfb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x474
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc421d0e180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:261 +0xc4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0x18c6

goroutine 140887 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x42a0580, 0xc421a20420)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420c4d2f0, 0xc420fd5560, 0xc42034ac00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 140903 [chan receive]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x42a0580, 0xc421ccfb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:322 +0x6c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035d290, 0xc420fd5560, 0xc4202f3840)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141015 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421cf1dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918cf0, 0xc420fd5560, 0xc421918cb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141159 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc420bef000, 0x42a0580, 0xc4205c6750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3835 +0x2f2
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x42a0580, 0xc4205c6750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3821 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425278600, 0xc420fd5560, 0xc4252785f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141014 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x42a0580, 0xc421ccfda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421918ca0, 0xc420fd5560, 0xc421918c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 141125 [

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 26, 2018
@mberhault mberhault mentioned this issue Feb 2, 2018
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Feb 2, 2018
@nvanbenschoten
sql: don't ignore job status update errors
65622ff 
Fixes cockroachdb#21828.
Fixes cockroachdb#21846.
Fixes cockroachdb#21844.
Fixes cockroachdb#21808.

See discussion in cockroachdb#21802.

We were previously ignoring the errors returned by job operations in
a few places. This meant that we would fail to propagate retryable txn
errors when using the jobs API and we could think a txn succeeded when
it didn't. This change fixes that.

In doing so, it makes schema changes fail when they see errors due to
job status updates. I don't see a compelling reason why the old behavior
was better and I suspect it was allowing subtle inconsistencies.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Feb 2, 2018
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Feb 12, 2018
@nvanbenschoten
sql: don't ignore job status update errors
9edbed9 
Fixes cockroachdb#21828.
Fixes cockroachdb#21846.
Fixes cockroachdb#21844.
Fixes cockroachdb#21808.

See discussion in cockroachdb#21802.

We were previously ignoring the errors returned by job operations in
a few places. This meant that we would fail to propagate retryable txn
errors when using the jobs API and we could think a txn succeeded when
it didn't. This change fixes that.

In doing so, it makes schema changes fail when they see errors due to
job status updates. I don't see a compelling reason why the old behavior
was better and I suspect it was allowing subtle inconsistencies.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: don't ignore job status update errors nvanbenschoten/cockroach
1 participant
@cockroach-teamcity