sql/jobs: TestRegistryLifecycle failed under stress #21844

cockroach-teamcity · 2018-01-27T07:49:30Z

SHA: https://github.com/cockroachdb/cockroach/commits/e0307f6586cd6109d09319e9000901e1ad3dc997

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=494900&tab=buildLog


I180127 07:49:26.955272 34 server/config.go:518  [n?] 1 storage engine initialized
I180127 07:49:26.955310 34 server/config.go:521  [n?] RocksDB cache size: 128 MiB
I180127 07:49:26.955316 34 server/config.go:521  [n?] store 0: in-memory, size 0 B
I180127 07:49:26.958661 34 server/node.go:374  [n?] **** cluster 20eb6c1e-f8ca-4dd7-8d27-21c6f6e9a0ed has been created
I180127 07:49:26.958691 34 server/server.go:973  [n?] **** add additional nodes by specifying --join=127.0.0.1:40197
I180127 07:49:26.959387 34 storage/store.go:1300  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180127 07:49:26.959446 34 server/node.go:504  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.3 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=3346.00 p25=3346.00 p50=3346.00 p75=3346.00 p90=3346.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180127 07:49:26.959480 34 server/node.go:352  [n1] node ID 1 initialized
I180127 07:49:26.959602 34 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:40197" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:11 > 
I180127 07:49:26.959782 34 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180127 07:49:26.959875 34 server/node.go:645  [n1] connecting to gossip network to verify cluster ID...
I180127 07:49:26.960749 34 server/node.go:670  [n1] node connected via gossip and verified as part of cluster "20eb6c1e-f8ca-4dd7-8d27-21c6f6e9a0ed"
I180127 07:49:26.960780 34 server/node.go:446  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180127 07:49:26.960852 34 server/server.go:1174  [n1] starting https server at 127.0.0.1:37973
I180127 07:49:26.960864 34 server/server.go:1175  [n1] starting grpc/postgres server at 127.0.0.1:40197
I180127 07:49:26.960875 34 server/server.go:1176  [n1] advertising CockroachDB node at 127.0.0.1:40197
W180127 07:49:26.960888 34 sql/jobs/registry.go:222  [n1] unable to get node liveness: node not in the liveness table
I180127 07:49:26.966520 40 storage/replica_command.go:835  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180127 07:49:26.972969 275 storage/replica_command.go:835  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180127 07:49:27.006704 152 storage/replica_command.go:835  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
I180127 07:49:27.008651 34 sql/event_log.go:114  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180127 07:49:27.010801 34 sql/lease.go:349  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-27 07:49:27.010675323 +0000 UTC
I180127 07:49:27.017091 34 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
I180127 07:49:27.026446 158 storage/replica_command.go:835  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180127 07:49:27.036311 34 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
I180127 07:49:27.038632 34 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
W180127 07:49:27.042891 159 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "unnamed" id=eaafe72c key=/Table/SystemConfigSpan/Start rw=true pri=0.03708531 iso=SERIALIZABLE stat=PENDING epo=0 ts=1517039367.038842293,0 orig=1517039367.038842293,0 max=1517039367.038842293,0 wto=false rop=false seq=2
I180127 07:49:27.043012 354 storage/replica_command.go:835  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180127 07:49:27.048707 365 storage/replica_command.go:835  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180127 07:49:27.051926 315 storage/replica_command.go:835  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180127 07:49:27.055126 389 storage/replica_command.go:835  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180127 07:49:27.058484 399 storage/replica_command.go:835  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
W180127 07:49:27.059892 362 storage/intent_resolver.go:320  [n1,s1] failed to push during intent resolution: failed to push "unnamed" id=eaafe72c key=/Table/SystemConfigSpan/Start rw=true pri=0.03708531 iso=SERIALIZABLE stat=PENDING epo=0 ts=1517039367.038842293,0 orig=1517039367.038842293,0 max=1517039367.038842293,0 wto=false rop=false seq=2
I180127 07:49:27.063387 404 storage/replica_command.go:835  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180127 07:49:27.078664 333 storage/replica_command.go:835  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180127 07:49:27.081695 422 storage/replica_command.go:835  [split,n1,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180127 07:49:27.088073 484 storage/replica_command.go:835  [split,n1,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180127 07:49:27.094444 427 storage/replica_command.go:835  [split,n1,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180127 07:49:27.100105 507 storage/replica_command.go:835  [split,n1,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
I180127 07:49:27.104420 414 storage/replica_command.go:835  [split,n1,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
I180127 07:49:27.107803 34 sql/event_log.go:114  [n1] Event: "alter_table", target: 4, info: {TableName:users Statement:ALTER TABLE system.users ADD COLUMN IF NOT EXISTS "isRole" BOOL NOT NULL DEFAULT false User:node MutationID:0 CascadeDroppedViews:[]}
I180127 07:49:27.110186 34 sql/lease.go:349  [n1] publish: descID=4 (users) version=2 mtime=2018-01-27 07:49:27.110113426 +0000 UTC
W180127 07:49:27.112634 34 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
I180127 07:49:27.114388 525 storage/replica_command.go:835  [split,n1,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
I180127 07:49:27.118018 554 storage/replica_command.go:835  [split,n1,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
I180127 07:49:27.127584 429 storage/replica_command.go:835  [split,n1,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
W180127 07:49:27.164875 34 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 07:49:27.274080 34 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 07:49:27.486693 34 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 07:49:27.899827 34 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180127 07:49:28.630929 34 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
F180127 07:49:28.631262 34 server/server.go:1242  [n1] failed to run migration "add system.users isRole column and create admin role": column "isRole" does not exist
goroutine 34 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x3051101, 0x150d9b44a58c9b86, 0x2b0d800, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x30511c0, 0xc400000004, 0x2b0d800, 0x10, 0x4da, 0xc420071e00, 0x73)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x59d
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x2e2c4a0, 0xc42048c030, 0x4, 0x2, 0x0, 0x0, 0xc42141ccc8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x308
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x2e2c4a0, 0xc42048c030, 0x1, 0xc400000004, 0x0, 0x0, 0xc42141ccc8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x2e2c4a0, 0xc42048c030, 0xc42141ccc8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x76
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc4200b2100, 0x2e2c4a0, 0xc42048c030, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1242 +0x3257
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc420010360, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x137
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0xef
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x2e3c6e0, 0xc4201b8870, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:172 +0x55
github.com/cockroachdb/cockroach/pkg/sql/jobs_test.TestRegistryLifecycle(0xc4201b8870)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/jobs_test.go:154 +0x111
testing.tRunner(0xc4201b8870, 0x1fea4f0)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201b8690, 0x1f28d9e, 0x15, 0x1fea4f0, 0x867001)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201b8690)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201b8690, 0xc4206c5db8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc42029d140, 0x2df7ae0, 0x7, 0x7, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420979f18, 0xc4206c5f50)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/sql/jobs_test.TestMain(0xc4206c5f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/main_test.go:33 +0xb8
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/jobs/_test/_testmain.go:56 +0xdb

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x30511c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1053 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 11 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 35 [select, locked to thread]:
runtime.gopark(0x1fed310, 0x0, 0x1f14da1, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc42003ef50, 0xc4204387e0)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 38 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420986000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1234 +0x25f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420986000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0xeb

goroutine 13 [select]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc420144d80, 0xc4200107e0, 0xc4201c0070)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:125 +0x10d
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:123 +0xc9

goroutine 14 [chan receive]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x2e2c4a0, 0xc4205c0000)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:322 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e030, 0xc420144d80, 0xc420227d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 15 [semacquire]:
sync.runtime_notifyListWait(0xc4200aa538, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4200aa528)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4200aa420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:625 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:612 +0x4a3

goroutine 16 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048c3c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044740, 0xc420144d80, 0xc420044510)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 50 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048c5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044bc0, 0xc420144d80, 0xc420044bb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 51 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048c600)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044bf0, 0xc420144d80, 0xc420044be0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 52 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048c660)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044c10, 0xc420144d80, 0xc420044c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 53 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048c720)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044c30, 0xc420144d80, 0xc420044c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 54 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048ca80)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044ce0, 0xc420144d80, 0xc420044cd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 55 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048ccf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044d70, 0xc420144d80, 0xc420044d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 56 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420044f10, 0xc420144d80, 0xc420044d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 57 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048cde0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045520, 0xc420144d80, 0xc4200454d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 58 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048ce40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045650, 0xc420144d80, 0xc420045580)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 59 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048d020)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045680, 0xc420144d80, 0xc420045660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 60 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048cd20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200456b0, 0xc420144d80, 0xc420045690)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 61 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048d230)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200456e0, 0xc420144d80, 0xc4200456c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 62 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048d1d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045850, 0xc420144d80, 0xc4200456f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 63 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048d350)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200458c0, 0xc420144d80, 0xc420045860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 64 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2c4a0, 0xc42048d380)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200458f0, 0xc420144d80, 0xc4200458d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 65 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x2e2c4a0, 0xc42083c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1464 +0x1f3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045900, 0xc420144d80, 0xc4205264e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 66 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x2e2c4a0, 0xc42083c060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045980, 0xc420144d80, 0xc420304ca0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 67 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x2e2c4a0, 0xc42083c090)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:775 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045a00, 0xc420144d80, 0xc420304da0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 68 [IO wait]:
internal/poll.runtime_pollWait(0x2b3f71103eb0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42049a098, 0x72, 0xc42070f500, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42049a098, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc42049a080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc42049a080, 0x2b3f7109f6c8, 0x0, 0x1)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42000e0c0, 0xc42070f6f8, 0x10, 0xfc41c7)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e0c0, 0x1feb908, 0xc4205c4940, 0xc42026a010, 0xc42070f6e0)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc4205c4940, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x95
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x2e2c4a0, 0xc42083c030)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:787 +0x2f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045a10, 0xc420144d80, 0xc420304e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 69 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420304dc0, 0xc420016108, 0x1ce2ae0, 0x2de55d0, 0x1ee2540)
	<autogenerated>:1 +0x65
net/http.(*Server).Serve(0xc42007ad00, 0x2e2b660, 0xc420304dc0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2695 +0x1b2
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x2e2c4a0, 0xc42083c150)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:799 +0x119
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045a20, 0xc420144d80, 0xc420526750)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 70 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420304de0, 0xc42083c120, 0xc420710680, 0x8fefca, 0x1e2d2c0)
	<autogenerated>:1 +0x65
crypto/tls.(*listener).Accept(0xc420304e20, 0xc420016108, 0x1ce2ae0, 0x2de55d0, 0x1ee2540)
	/usr/local/go/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc420581860, 0x2e1d8a0, 0xc420304e20, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2695 +0x1b2
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x2e2c4a0, 0xc42083c0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:806 +0x42
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045a30, 0xc420144d80, 0xc420304e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 71 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func6(0x2e2c4a0, 0xc42083c300)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:810 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045a40, 0xc420144d80, 0xc4205c4980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 72 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420304d40, 0x1fec6f0, 0xc4200aa2c0, 0x2e37d20, 0xc420b2c000)
	<autogenerated>:1 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4200aa2c0, 0x2e2b660, 0xc420304d40, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:504 +0x1ec
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x2e2c4a0, 0xc42083c330)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:822 +0x43
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420045a50, 0xc420144d80, 0xc420304e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 25 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x2e2c4a0, 0xc42083c2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d2c50, 0xc420144d80, 0xc420316040)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 73 [semacquire]:
sync.runtime_notifyListWait(0xc4206fa958, 0xc400000122)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4206fa948)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4206fa840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:625 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:612 +0x4a3

goroutine 74 [chan receive]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2e2c4a0, 0xc42083c360)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:371 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e060, 0xc420144d80, 0xc420305020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 75 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2e2c4a0, 0xc42083c3c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1160 +0x46b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e080, 0xc420144d80, 0xc42026e070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 76 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2e2c4a0, 0xc42083c420)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1189 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e0c0, 0xc420144d80, 0xc42026e090)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 27 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x2e2c4a0, 0xc4208e00c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1058 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3860, 0xc420144d80, 0xc42083c6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 37 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc4208df0c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x156
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x197

goroutine 174 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000014b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083de90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083de90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3e40, 0xc420144d80, 0xc4203d3e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 173 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000014f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083de60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083de60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3e20, 0xc420144d80, 0xc4203d3e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 172 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000149)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083de30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083de30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3e00, 0xc420144d80, 0xc4203d3df0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 171 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000147)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083de00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083de00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3de0, 0xc420144d80, 0xc4203d3dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 170 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000146)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083ddd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083ddd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3dc0, 0xc420144d80, 0xc4203d3db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 168 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000152)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dd70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dd70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3d80, 0xc420144d80, 0xc4203d3d70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 167 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000148)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dd40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dd40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3d60, 0xc420144d80, 0xc4203d3d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 166 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000144)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dd10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dd10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3cf0, 0xc420144d80, 0xc4203d3cd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 165 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000143)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3cc0, 0xc420144d80, 0xc4203d3ca0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 175 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000151)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3e60, 0xc420144d80, 0xc4203d3e50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 163 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000014a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3c60, 0xc420144d80, 0xc4203d3c50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 162 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000140)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dc50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dc50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3c30, 0xc420144d80, 0xc4203d3c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 169 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000014e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3da0, 0xc420144d80, 0xc4203d3d90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 32 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000141)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3be0, 0xc420144d80, 0xc4203d3bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 164 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000145)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dcb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dcb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3c90, 0xc420144d80, 0xc4203d3c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 30 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func15(0x2e2c4a0, 0xc42083db90)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d39a0, 0xc420144d80, 0xc420316380)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 33 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000013f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dc20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dc20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3c00, 0xc420144d80, 0xc4203d3bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 31 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2e2c4a0, 0xc42083dbc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:156 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3bb0, 0xc420144d80, 0xc420316de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 29 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc42083c750)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110 +0x1ac
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:96 +0x3f

goroutine 26 [IO wait]:
internal/poll.runtime_pollWait(0x2b3f71103f70, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42049a018, 0x72, 0xc420712c00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42049a018, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc42049a000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc42049a000, 0xc420b24000, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42000e088, 0xfc42ba, 0x832a70, 0xc420712e80)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e088, 0x1feb910, 0xc4205c4900, 0x2e37ea0, 0xc420b24000)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc4205c4900, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x95
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func10.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1004 +0x2f
sync.(*Once).Do(0xc4202595a0, 0xc4208cc758)
	/usr/local/go/src/sync/once.go:44 +0xbe
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func10(0x2e2c4a0, 0xc4208e00f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1003 +0x5b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3780, 0xc420144d80, 0xc420316160)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 268 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x2e2c3e0, 0xc4208df1c0, 0xc420b22690, 0xc4208e8fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x3c6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc4209b7c80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:305 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:304 +0xdc4

goroutine 263 [semacquire]:
sync.runtime_notifyListWait(0xc4208601c0, 0xc400000003)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4208601b0)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc420860180, 0xc420903000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc42059c058, 0xc420903000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc4208fe750, 0x2b3f7119f418, 0xc4203d3840, 0x5, 0xc4203d3840, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc420972380, 0x1fed417, 0xc4209724a0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc420972380, 0xc420b2e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc420b362a0, 0xc42030ba78, 0x9, 0x9, 0x28, 0x1e07940, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2e0c2a0, 0xc420b362a0, 0xc42030ba78, 0x9, 0x9, 0x9, 0xc4208e5de0, 0x7d971e, 0xc4208e5e8f)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2e0c2a0, 0xc420b362a0, 0xc42030ba78, 0x9, 0x9, 0xc81214, 0xc420b23440, 0xc420b20004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42030ba78, 0x9, 0x9, 0x2e0c2a0, 0xc420b362a0, 0x0, 0x0, 0xc420b23440, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42030ba40, 0xc420b23440, 0xc420b23400, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4209b7c80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1173 +0xe1
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:273 +0xc73

goroutine 176 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000014c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083def0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083def0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3e90, 0xc420144d80, 0xc4203d3e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 177 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000150)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083df20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083df20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3eb0, 0xc420144d80, 0xc4203d3ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 178 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000153)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083df50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083df50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3ee0, 0xc420144d80, 0xc4203d3ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 179 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000156)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083df80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083df80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3f10, 0xc420144d80, 0xc4203d3f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 180 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000157)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc42083dfb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc42083dfb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3f40, 0xc420144d80, 0xc4203d3f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 181 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000154)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3f90, 0xc420144d80, 0xc4203d3f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 182 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000155)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3fb0, 0xc420144d80, 0xc4203d3fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 183 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000015f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d3fd0, 0xc420144d80, 0xc4203d3fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 184 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000158)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ac010, 0xc420144d80, 0xc4203d3fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 185 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc400000159)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ac030, 0xc420144d80, 0xc4202ac020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 186 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000015a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ac050, 0xc420144d80, 0xc4202ac040)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 187 [semacquire]:
sync.runtime_notifyListWait(0xc4202b2a50, 0xc40000015b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4202b2a40)
1 runs completed, 1 failures, over 2s
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420852120, 0x2e2c4a0, 0xc420a1c120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2c4a0, 0xc420a1c120)

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2018-01-28T07:44:25Z

SHA: https://github.com/cockroachdb/cockroach/commits/54550b687bef4d1db5ad991d384ff0369e48729d

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495555&tab=buildLog

cockroach-teamcity · 2018-01-29T07:53:28Z

SHA: https://github.com/cockroachdb/cockroach/commits/6fadcd50dd42272a9579fb950e37aa7e408bc2f2

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=496258&tab=buildLog

cockroach-teamcity · 2018-01-29T09:19:24Z

SHA: https://github.com/cockroachdb/cockroach/commits/6fadcd50dd42272a9579fb950e37aa7e408bc2f2

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=496432&tab=buildLog

maddyblue · 2018-01-29T19:35:56Z

@mberhault These errors look like maybe your thing.

cockroach-teamcity · 2018-01-31T09:04:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/07890702e2faabd34b7885470b141a5612d862cc

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=500026&tab=buildLog

cockroach-teamcity · 2018-01-31T10:15:30Z

SHA: https://github.com/cockroachdb/cockroach/commits/07890702e2faabd34b7885470b141a5612d862cc

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=500202&tab=buildLog

cockroach-teamcity · 2018-02-01T10:36:12Z

SHA: https://github.com/cockroachdb/cockroach/commits/b9416fc1db1f87ef8871c75980efcdb1e249474b

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=501991&tab=buildLog

cockroach-teamcity · 2018-02-02T07:45:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/aa73c6491abcee69c9ad3da87ea138e9da0dfbe2

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=503347&tab=buildLog

Fixes cockroachdb#21828. Fixes cockroachdb#21846. Fixes cockroachdb#21844. Fixes cockroachdb#21808. See discussion in cockroachdb#21802. We were previously ignoring the errors returned by job operations in a few places. This meant that we would fail to propagate retryable txn errors when using the jobs API and we could think a txn succeeded when it didn't. This change fixes that. In doing so, it makes schema changes fail when they see errors due to job status updates. I don't see a compelling reason why the old behavior was better and I suspect it was allowing subtle inconsistencies. Release note: None

cockroach-teamcity · 2018-02-03T09:15:13Z

SHA: https://github.com/cockroachdb/cockroach/commits/cb007b412e551809db05d8e24de129bac056c8d6

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=504908&tab=buildLog

cockroach-teamcity · 2018-02-03T10:34:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/cb007b412e551809db05d8e24de129bac056c8d6

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=505085&tab=buildLog

cockroach-teamcity · 2018-02-04T07:52:12Z

SHA: https://github.com/cockroachdb/cockroach/commits/6a1cc888163fd154f80c1748837f2a6e10e62fe4

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=505551&tab=buildLog

cockroach-teamcity · 2018-02-06T08:02:05Z

SHA: https://github.com/cockroachdb/cockroach/commits/4b2c64a7107e392229939b24b5bda303c1950a1f

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=507914&tab=buildLog

cockroach-teamcity · 2018-02-07T09:28:57Z

SHA: https://github.com/cockroachdb/cockroach/commits/27384c5c82a6f20679aa7f833875eda6f068e4de

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=509911&tab=buildLog

cockroach-teamcity · 2018-02-07T10:40:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/27384c5c82a6f20679aa7f833875eda6f068e4de

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=510092&tab=buildLog

cockroach-teamcity · 2018-02-08T10:32:35Z

SHA: https://github.com/cockroachdb/cockroach/commits/f45bcea79f3d26f9484075ee174e270e9ccdc714

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=511971&tab=buildLog

cockroach-teamcity · 2018-02-09T09:29:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/c93d304f8b31b5b0678a46d36a7b751cf50fedb1

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=513664&tab=buildLog

cockroach-teamcity · 2018-02-09T10:53:07Z

SHA: https://github.com/cockroachdb/cockroach/commits/c93d304f8b31b5b0678a46d36a7b751cf50fedb1

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=513845&tab=buildLog

cockroach-teamcity · 2018-02-10T09:19:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/fb37bf0c50592df50959fdb25903616a682e3c97

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=514860&tab=buildLog

cockroach-teamcity · 2018-02-11T11:09:31Z

SHA: https://github.com/cockroachdb/cockroach/commits/2ced98d8cd4d13abae6331e73effc5b13f369c9a

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=515969&tab=buildLog

cockroach-teamcity · 2018-02-12T08:08:12Z

SHA: https://github.com/cockroachdb/cockroach/commits/135c4999731e5e67f95fc8557f95cee2a0aeadc0

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=516503&tab=buildLog

cockroach-teamcity · 2018-02-12T11:05:01Z

SHA: https://github.com/cockroachdb/cockroach/commits/135c4999731e5e67f95fc8557f95cee2a0aeadc0

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=516865&tab=buildLog

Fixes cockroachdb#21828. Fixes cockroachdb#21846. Fixes cockroachdb#21844. Fixes cockroachdb#21808. See discussion in cockroachdb#21802. We were previously ignoring the errors returned by job operations in a few places. This meant that we would fail to propagate retryable txn errors when using the jobs API and we could think a txn succeeded when it didn't. This change fixes that. In doing so, it makes schema changes fail when they see errors due to job status updates. I don't see a compelling reason why the old behavior was better and I suspect it was allowing subtle inconsistencies. Release note: None

cockroach-teamcity assigned maddyblue Jan 27, 2018

cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 27, 2018

maddyblue assigned mberhault and unassigned maddyblue Jan 29, 2018

mberhault mentioned this issue Feb 2, 2018

jobs: runInTxn cannot AutoRetry, could result in partial transactions #21802

Merged

nvanbenschoten mentioned this issue Feb 2, 2018

sql: don't ignore job status update errors #22337

Merged

nvanbenschoten closed this as completed in #22337 Feb 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql/jobs: TestRegistryLifecycle failed under stress #21844

sql/jobs: TestRegistryLifecycle failed under stress #21844

cockroach-teamcity commented Jan 27, 2018

cockroach-teamcity commented Jan 28, 2018

cockroach-teamcity commented Jan 29, 2018

cockroach-teamcity commented Jan 29, 2018

maddyblue commented Jan 29, 2018

cockroach-teamcity commented Jan 31, 2018

cockroach-teamcity commented Jan 31, 2018

cockroach-teamcity commented Feb 1, 2018

cockroach-teamcity commented Feb 2, 2018

cockroach-teamcity commented Feb 3, 2018

cockroach-teamcity commented Feb 3, 2018

cockroach-teamcity commented Feb 4, 2018

cockroach-teamcity commented Feb 6, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 11, 2018

cockroach-teamcity commented Feb 12, 2018

cockroach-teamcity commented Feb 12, 2018

sql/jobs: TestRegistryLifecycle failed under stress #21844

sql/jobs: TestRegistryLifecycle failed under stress #21844

Comments

cockroach-teamcity commented Jan 27, 2018

cockroach-teamcity commented Jan 28, 2018

cockroach-teamcity commented Jan 29, 2018

cockroach-teamcity commented Jan 29, 2018

maddyblue commented Jan 29, 2018

cockroach-teamcity commented Jan 31, 2018

cockroach-teamcity commented Jan 31, 2018

cockroach-teamcity commented Feb 1, 2018

cockroach-teamcity commented Feb 2, 2018

cockroach-teamcity commented Feb 3, 2018

cockroach-teamcity commented Feb 3, 2018

cockroach-teamcity commented Feb 4, 2018

cockroach-teamcity commented Feb 6, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 11, 2018

cockroach-teamcity commented Feb 12, 2018

cockroach-teamcity commented Feb 12, 2018