sql/jobs: TestRegistryResumeActiveLease failed under stress #21808

cockroach-teamcity · 2018-01-26T07:37:10Z

SHA: https://github.com/cockroachdb/cockroach/commits/af29319fc11eb4c441d9cdec5b33a4bc30b88a51

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=493611&tab=buildLog


I180126 07:37:06.089637 2933 server/config.go:518  [n?] 1 storage engine initialized
I180126 07:37:06.089658 2933 server/config.go:521  [n?] RocksDB cache size: 128 MiB
I180126 07:37:06.089667 2933 server/config.go:521  [n?] store 0: in-memory, size 0 B
I180126 07:37:06.094867 2933 server/node.go:374  [n?] **** cluster 6928edc7-5f66-4b4b-986d-e02a9b59edac has been created
I180126 07:37:06.094905 2933 server/server.go:973  [n?] **** add additional nodes by specifying --join=127.0.0.1:40191
I180126 07:37:06.095680 2933 storage/store.go:1300  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180126 07:37:06.095742 2933 server/node.go:504  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.3 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=3346.00 p25=3346.00 p50=3346.00 p75=3346.00 p90=3346.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180126 07:37:06.095772 2933 server/node.go:352  [n1] node ID 1 initialized
I180126 07:37:06.095829 2933 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:40191" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:11 > 
I180126 07:37:06.095968 2933 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180126 07:37:06.096060 2933 server/node.go:645  [n1] connecting to gossip network to verify cluster ID...
I180126 07:37:06.106881 2933 server/node.go:670  [n1] node connected via gossip and verified as part of cluster "6928edc7-5f66-4b4b-986d-e02a9b59edac"
I180126 07:37:06.106916 2933 server/node.go:446  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180126 07:37:06.107006 2933 server/server.go:1174  [n1] starting https server at 127.0.0.1:36019
I180126 07:37:06.107021 2933 server/server.go:1175  [n1] starting grpc/postgres server at 127.0.0.1:40191
I180126 07:37:06.107033 2933 server/server.go:1176  [n1] advertising CockroachDB node at 127.0.0.1:40191
I180126 07:37:06.123011 2911 storage/replica_command.go:835  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180126 07:37:06.136998 3214 storage/replica_command.go:835  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180126 07:37:06.141762 3234 storage/replica_command.go:835  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
I180126 07:37:06.149782 3251 storage/replica_command.go:835  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180126 07:37:06.157851 2933 sql/event_log.go:114  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180126 07:37:06.160615 2933 sql/lease.go:349  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-26 07:37:06.160439669 +0000 UTC
I180126 07:37:06.175616 3233 storage/replica_command.go:835  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180126 07:37:06.184823 2933 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
I180126 07:37:06.197484 3282 storage/replica_command.go:835  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180126 07:37:06.204503 2933 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
I180126 07:37:06.208405 2933 sql/event_log.go:114  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180126 07:37:06.219559 3286 storage/replica_command.go:835  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180126 07:37:06.227943 3313 storage/replica_command.go:835  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180126 07:37:06.234407 2933 sql/event_log.go:114  [n1] Event: "alter_table", target: 4, info: {TableName:users Statement:ALTER TABLE system.users ADD COLUMN IF NOT EXISTS "isRole" BOOL NOT NULL DEFAULT false User:node MutationID:0 CascadeDroppedViews:[]}
I180126 07:37:06.234689 3191 storage/replica_command.go:835  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180126 07:37:06.236412 2933 sql/lease.go:349  [n1] publish: descID=4 (users) version=2 mtime=2018-01-26 07:37:06.236108019 +0000 UTC
W180126 07:37:06.238874 2933 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
I180126 07:37:06.252745 3293 storage/replica_command.go:835  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180126 07:37:06.262157 3368 storage/replica_command.go:835  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180126 07:37:06.265995 3200 storage/replica_command.go:835  [split,n1,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180126 07:37:06.271279 3246 storage/replica_command.go:835  [split,n1,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180126 07:37:06.276911 3394 storage/replica_command.go:835  [split,n1,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180126 07:37:06.281006 3338 storage/replica_command.go:835  [split,n1,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
I180126 07:37:06.284526 3358 storage/replica_command.go:835  [split,n1,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
I180126 07:37:06.288081 3420 storage/replica_command.go:835  [split,n1,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
W180126 07:37:06.290400 2933 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
I180126 07:37:06.291589 3442 storage/replica_command.go:835  [split,n1,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
I180126 07:37:06.296973 3448 storage/replica_command.go:835  [split,n1,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
W180126 07:37:06.402303 2933 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 07:37:06.629618 2933 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 07:37:07.022792 2933 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
W180126 07:37:07.865576 2933 sqlmigrations/migrations.go:743  [n1] failed to insert admin role into the system.users table: column "isRole" does not exist
F180126 07:37:07.865933 2933 server/server.go:1242  [n1] failed to run migration "add system.users isRole column and create admin role": column "isRole" does not exist
goroutine 2933 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x304f101, 0x150d4c039b2fb80b, 0x2b0bdd0, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x304f1a0, 0xc400000004, 0x2b0bdd0, 0x10, 0x4da, 0xc421038600, 0x73)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x59d
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x2e2a480, 0xc420c89da0, 0x4, 0x2, 0x0, 0x0, 0xc420ca8ea0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x308
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x2e2a480, 0xc420c89da0, 0x1, 0xc400000004, 0x0, 0x0, 0xc420ca8ea0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x2e2a480, 0xc420c89da0, 0xc420ca8ea0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x76
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc42120e580, 0x2e2a480, 0xc420c89da0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1242 +0x3257
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc4209e0120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x137
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0xef
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x2e3a6c0, 0xc420cca000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:172 +0x55
github.com/cockroachdb/cockroach/pkg/sql/jobs_test.TestRegistryResumeActiveLease(0xc420cca000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry_external_test.go:226 +0x1c3
testing.tRunner(0xc420cca000, 0x1fe9140)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201b8690, 0x1f34ec0, 0x1d, 0x1fe9140, 0x867001)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201b8690)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201b8690, 0xc4205efdb8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc420268cc0, 0x2df5ae0, 0x7, 0x7, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420611f18, 0xc4205eff50)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/sql/jobs_test.TestMain(0xc4205eff18)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/main_test.go:33 +0xb8
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/jobs/_test/_testmain.go:56 +0xdb

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x304f1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1053 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 15 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 37 [select, locked to thread]:
runtime.gopark(0x1febf50, 0x0, 0x1f13aa1, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc420740f50, 0xc42042e420)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 3009 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func15(0x2e2a480, 0xc4207a5ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e200, 0xc420b50000, 0xc420268c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3020 [IO wait]:
internal/poll.runtime_pollWait(0x7f3e89682df0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420102518, 0x72, 0xc42070d500, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420102518, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420102500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc420102500, 0x7f3e896de6c8, 0x0, 0x1)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4209141d8, 0xc42070d6f8, 0x10, 0xfc41c7)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4209141d8, 0x1fea548, 0xc421467ec0, 0xc4202ce010, 0xc42070d6e0)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/githu435 runs completed, 1 failures, over 2m53s
b.com/cockroachdb/cmux.(*cMux).Serve(0xc421467ec0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x95
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x2e2a480, 0xc4212c4330)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:787 +0x2f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fab70, 0xc420b50000, 0xc42035d100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3115 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000156)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4207a5ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4207a5ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e5b0, 0xc420b50000, 0xc42026e5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2912 [select]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x2e2a480, 0xc420758630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x6b4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c1d0, 0xc420b50000, 0xc42042c1c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3008 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc4204c2330)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110 +0x1ac
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:96 +0x3f

goroutine 2988 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x2e2a3c0, 0xc4214345c0, 0xc42090a5a0, 0xc421413fb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x3c6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc42071d680)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:305 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:304 +0xdc4

goroutine 3118 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000015b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4207a5f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4207a5f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e630, 0xc420b50000, 0xc42026e620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2934 [select]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc420b50000, 0xc4209e03c0, 0xc420438070)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:125 +0x10d
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:123 +0xc9

goroutine 3016 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2a480, 0xc4204c2000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fa9f0, 0xc420b50000, 0xc4203fa9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2938 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2a480, 0xc4212c4090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fa7c0, 0xc420b50000, 0xc4203fa7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3135 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000016a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc420abc0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc420abc0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e990, 0xc420b50000, 0xc42026e980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3151 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000014f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ebd0, 0xc420b50000, 0xc42026ebb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3126 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000016c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e820, 0xc420b50000, 0xc42026e810)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3206 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x2e2a480, 0xc420759bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:373 +0x2a9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420320550, 0xc420b50000, 0xc420ad2cd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3124 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000016b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e770, 0xc420b50000, 0xc42026e750)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3119 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000015c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4207a5f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4207a5f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e660, 0xc420b50000, 0xc42026e640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3153 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000155)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ec30, 0xc420b50000, 0xc42026ec10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3006 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x2e2a480, 0xc4204c3ef0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1058 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e110, 0xc420b50000, 0xc4204c22a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2977 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x2e2a480, 0xc4212c45d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420221110, 0xc420b50000, 0xc42024b4c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2909 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2e2a480, 0xc4210d8750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c080, 0xc420b50000, 0xc4202fc100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2940 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2a480, 0xc4212c40f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fa800, 0xc420b50000, 0xc4203fa7f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2986 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc42071d800)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:956 +0x264
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:264 +0xf0e

goroutine 3018 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x2e2a480, 0xc4212c4360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203faae0, 0xc420b50000, 0xc42035ce00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3171 [semacquire]:
sync.runtime_Semacquire(0xc42067c084)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc42067c078)
	/usr/local/go/src/sync/waitgroup.go:131 +0x72
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc42067c000, 0x2e2a480, 0xc420abcab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x2e2a480, 0xc420abcab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3819 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ef50, 0xc420b50000, 0xc42026ef40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3202 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x2e2a480, 0xc4207a07e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1005 +0x9ce
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420320470, 0xc420b50000, 0xc42035dfa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3112 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000014e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4207a5e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4207a5e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e540, 0xc420b50000, 0xc42026e530)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2944 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2a480, 0xc4212c41b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fa880, 0xc420b50000, 0xc4203fa870)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3157 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000161)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ed60, 0xc420b50000, 0xc42026ece0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3203 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Start.func1(0x2e2a480, 0xc4207a1350)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:418 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420320480, 0xc420b50000, 0xc42035dfe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3015 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2a480, 0xc4212c42d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fa9c0, 0xc420b50000, 0xc4203fa9b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3027 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2e2a480, 0xc4207a4240)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1160 +0x46b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fadc0, 0xc420b50000, 0xc4203fadb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3102 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x2e2a480, 0xc420abcc00)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:112 +0x2db
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420320340, 0xc420b50000, 0xc42035df40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3167 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000013c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026eed0, 0xc420b50000, 0xc42026eec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3179 [select]:
github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1(0x2e2a480, 0xc4210d8780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:124 +0x1ce
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026f0d0, 0xc420b50000, 0xc4204c3ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3236 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x2e2a480, 0xc42111a1b0, 0xc42111a150, 0xc420c0dfb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x3c6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc420773080)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:261 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0xee9

goroutine 3238 [IO wait]:
internal/poll.runtime_pollWait(0x7f3e89682d30, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420e12418, 0x72, 0xffffffffffffff00, 0x2e14880, 0x2e08498)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420e12418, 0xc4214ac400, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420e12400, 0xc4214ac400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420e12400, 0xc4214ac400, 0x400, 0x400, 0x1, 0xc420f94898, 0x820f7a)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42000e058, 0xc4214ac400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4203ae780, 0xc4214ac400, 0x400, 0x400, 0x1e2a0a0, 0xc4210d8400, 0x7f3e88438028)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x121
crypto/tls.(*block).readFromUntil(0xc4210d84e0, 0x7f3e88438028, 0xc4203ae780, 0x5, 0xc4203ae780, 0xf)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc420d64700, 0x1fec017, 0xc420d64820, 0x30)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc420d64700, 0xc420f06000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc420f3eb40, 0xc420303618, 0x9, 0x9, 0x1, 0xc420c8bfb0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2e0a280, 0xc420f3eb40, 0xc420303618, 0x9, 0x9, 0x9, 0xcd1496, 0xc420c8bec0, 0x0)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2e0a280, 0xc420f3eb40, 0xc420303618, 0x9, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420303618, 0x9, 0x9, 0x2e0a280, 0xc420f3eb40, 0x0, 0x1000100000000, 0xf00000031, 0x3100010001)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203035e0, 0xc42111a060, 0xc42111a060, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420773080, 0xc420f4d590, 0x1feb378)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:395 +0x3a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420b482c0, 0x2e35040, 0xc420773080)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:634 +0xf2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:591 +0x3c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4202a2da0, 0xc420b482c0, 0x7f3e896954c8, 0xc420773080)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:599 +0x27
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:598 +0x656

goroutine 2987 [semacquire]:
sync.runtime_notifyListWait(0xc4203624c0, 0xc400000003)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4203624b0)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc420362480, 0xc420b3a400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc42101e000, 0xc420b3a400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc4204c3f50, 0x7f3e896423b0, 0xc42026e0e0, 0x5, 0xc42026e0e0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc420032e00, 0x1fec017, 0xc420032f20, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc420032e00, 0xc421622000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc420e28900, 0xc420365b58, 0x9, 0x9, 0x28, 0x1e066a0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2e0a280, 0xc420e28900, 0xc420365b58, 0x9, 0x9, 0x9, 0xc420c24d60, 0x7d971e, 0xc420c24e0f)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2e0a280, 0xc420e28900, 0xc420365b58, 0x9, 0x9, 0xc81214, 0xc42090a870, 0xc420900004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420365b58, 0x9, 0x9, 0x2e0a280, 0xc420e28900, 0x0, 0x0, 0xc42090a870, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420365b20, 0xc42090a870, 0xc42090a870, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc42071d800, 0xc42090a810, 0x1feb378)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:395 +0x3a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420b482c0, 0x2e35040, 0xc42071d800)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:634 +0xf2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:591 +0x3c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4202fd900, 0xc420b482c0, 0x7f3e896954c8, 0xc42071d800)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:599 +0x27
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:598 +0x656

goroutine 3205 [select]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x2e2a480, 0xc4207a0750)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:125 +0x17b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203204c0, 0xc420b50000, 0xc4203204b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3170 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000136)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ef30, 0xc420b50000, 0xc42026ef20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3105 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteSummaries.func1(0x2e2a480, 0xc4210d87b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:773 +0x1f6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420320460, 0xc420b50000, 0xc42035df80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2902 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2e2a480, 0xc420758060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c000, 0xc420b50000, 0xc4202fc000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3150 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000014d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c47e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c47e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026eba0, 0xc420b50000, 0xc42026eb80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3144 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000146)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c46c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c46c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026eac0, 0xc420b50000, 0xc42026eab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3021 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42035d0c0, 0xc420016108, 0x1ce19a0, 0x2de35d0, 0x1ee1240)
	<autogenerated>:1 +0x65
net/http.(*Server).Serve(0xc42007c270, 0x2e29640, 0xc42035d0c0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2695 +0x1b2
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x2e2a480, 0xc4212c4450)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:799 +0x119
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fab80, 0xc420b50000, 0xc420c46c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2908 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2e2a480, 0xc4210d86f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c070, 0xc420b50000, 0xc4202fc0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3121 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000164)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc420abc000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc420abc000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e6b0, 0xc420b50000, 0xc42026e6a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2903 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2e2a480, 0xc420758090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c010, 0xc420b50000, 0xc4202fc020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3159 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000015e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026eda0, 0xc420b50000, 0xc42026ed90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2910 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc4203c4820, 0x2e2a480, 0xc420758030, 0xc420b50000, 0x133eaf5f, 0xed1fcd1a2, 0x0, 0xc420443880, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:212 +0x226
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc420443880, 0xc4204c2701)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:274 +0x87
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc4204c2750, 0xc420a64ac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:314 +0x1ba
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x2e2a480, 0xc420758030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:272 +0x1dc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c0a0, 0xc420b50000, 0xc4202fc120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3148 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000149)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026eb40, 0xc420b50000, 0xc42026eb30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3161 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000140)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ee00, 0xc420b50000, 0xc42026edf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3173 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc420ce6400, 0x2e2a480, 0xc420abcb10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3876 +0x16d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x2e2a480, 0xc420abcb10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3822 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ef90, 0xc420b50000, 0xc42026ef80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3101 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x2e2a480, 0xc420abcba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:698 +0x2dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203202f0, 0xc420b50000, 0xc42035df00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2906 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2e2a480, 0xc4210d8660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042c040, 0xc420b50000, 0xc4202fc080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3013 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2e2a480, 0xc4212c42a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203fa970, 0xc420b50000, 0xc4203fa950)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3222 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42009ac80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1234 +0x25f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc42009ac80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0xeb

goroutine 3139 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000134)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc420abc180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc420abc180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ea10, 0xc420b50000, 0xc42026ea00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3165 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000013a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc4212c4c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc4212c4c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ee80, 0xc420b50000, 0xc42026ee70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3183 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420ce4a00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1234 +0x25f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420ce4a00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0xeb

goroutine 3100 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x2e2a480, 0xc420abcb70)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:736 +0x14f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203202e0, 0xc420b50000, 0xc42035dee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2984 [semacquire]:
sync.runtime_notifyListWait(0xc420362ac0, 0xc400000002)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420362ab0)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc420362a80, 0xc420b3b800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc42101e010, 0xc420b3b800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc421116bd0, 0x7f3e896423b0, 0xc42026e0f0, 0x5, 0xc42026e0f0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc420033c00, 0x1fec017, 0xc420033d20, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc420033c00, 0xc42145e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc420e28840, 0xc420364578, 0x9, 0x9, 0x28, 0x1e066a0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2e0a280, 0xc420e28840, 0xc420364578, 0x9, 0x9, 0x9, 0xc420c23de0, 0x7d971e, 0xc420c23e8f)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2e0a280, 0xc420e28840, 0xc420364578, 0x9, 0x9, 0xc81214, 0xc42090a840, 0xc420900004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420364578, 0x9, 0x9, 0x2e0a280, 0xc420e28840, 0x0, 0x0, 0xc42090a840, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420364540, 0xc42090a840, 0xc42090a800, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc42071d680)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1173 +0xe1
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:273 +0xc73

goroutine 3172 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc420ce6400, 0x2e2a480, 0xc420abcae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3835 +0x205
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x2e2a480, 0xc420abcae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3821 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026ef70, 0xc420b50000, 0xc42026ef60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3138 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc40000012f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc420abc150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc420abc150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e9f0, 0xc420b50000, 0xc42026e9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 3237 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420773080)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:956 +0x264
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:264 +0xf0e

goroutine 3134 [semacquire]:
sync.runtime_notifyListWait(0xc421434350, 0xc400000169)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc421434340)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42067c000, 0x2e2a480, 0xc420abc090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e2a480, 0xc420abc090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026e920, 0xc420b50000, 0xc42026e910)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 2907 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2e2a480, 0xc4210d8720)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1b2

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2018-01-26T08:47:03Z

SHA: https://github.com/cockroachdb/cockroach/commits/68ab61a20e5408febf3471dbef36f07c9ab87fb4

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=493783&tab=buildLog

cockroach-teamcity · 2018-01-26T10:00:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/68ab61a20e5408febf3471dbef36f07c9ab87fb4

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=493955&tab=buildLog

cockroach-teamcity · 2018-01-27T09:12:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/2c85c61e7bc64c7c62aa7e25e9e762180f4e345a

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495073&tab=buildLog

cockroach-teamcity · 2018-01-27T10:35:04Z

SHA: https://github.com/cockroachdb/cockroach/commits/2c85c61e7bc64c7c62aa7e25e9e762180f4e345a

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495246&tab=buildLog

cockroach-teamcity · 2018-01-28T09:05:47Z

SHA: https://github.com/cockroachdb/cockroach/commits/40a7b4e6677987d8235d5b607832c3dbf20a8f2b

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495730&tab=buildLog

cockroach-teamcity · 2018-01-28T10:22:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/40a7b4e6677987d8235d5b607832c3dbf20a8f2b

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495905&tab=buildLog

cockroach-teamcity · 2018-01-29T10:34:57Z

SHA: https://github.com/cockroachdb/cockroach/commits/6fadcd50dd42272a9579fb950e37aa7e408bc2f2

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=496606&tab=buildLog

cockroach-teamcity · 2018-01-31T07:48:32Z

SHA: https://github.com/cockroachdb/cockroach/commits/07890702e2faabd34b7885470b141a5612d862cc

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=499851&tab=buildLog

cockroach-teamcity · 2018-02-01T07:54:16Z

SHA: https://github.com/cockroachdb/cockroach/commits/b9416fc1db1f87ef8871c75980efcdb1e249474b

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=501641&tab=buildLog

cockroach-teamcity · 2018-02-01T09:35:09Z

SHA: https://github.com/cockroachdb/cockroach/commits/b9416fc1db1f87ef8871c75980efcdb1e249474b

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=501816&tab=buildLog

cockroach-teamcity · 2018-02-02T10:19:31Z

SHA: https://github.com/cockroachdb/cockroach/commits/aa73c6491abcee69c9ad3da87ea138e9da0dfbe2

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=503699&tab=buildLog

Fixes cockroachdb#21828. Fixes cockroachdb#21846. Fixes cockroachdb#21844. Fixes cockroachdb#21808. See discussion in cockroachdb#21802. We were previously ignoring the errors returned by job operations in a few places. This meant that we would fail to propagate retryable txn errors when using the jobs API and we could think a txn succeeded when it didn't. This change fixes that. In doing so, it makes schema changes fail when they see errors due to job status updates. I don't see a compelling reason why the old behavior was better and I suspect it was allowing subtle inconsistencies. Release note: None

cockroach-teamcity · 2018-02-03T07:50:09Z

SHA: https://github.com/cockroachdb/cockroach/commits/cb007b412e551809db05d8e24de129bac056c8d6

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=504731&tab=buildLog

cockroach-teamcity · 2018-02-04T10:28:14Z

SHA: https://github.com/cockroachdb/cockroach/commits/6a1cc888163fd154f80c1748837f2a6e10e62fe4

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=505905&tab=buildLog

cockroach-teamcity · 2018-02-05T07:48:38Z

SHA: https://github.com/cockroachdb/cockroach/commits/c6e9dd02cb671503e2618ee713bee8bbfdabaab3

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=506267&tab=buildLog

cockroach-teamcity · 2018-02-05T10:27:14Z

SHA: https://github.com/cockroachdb/cockroach/commits/c6e9dd02cb671503e2618ee713bee8bbfdabaab3

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=506621&tab=buildLog

cockroach-teamcity · 2018-02-06T09:37:41Z

SHA: https://github.com/cockroachdb/cockroach/commits/4b2c64a7107e392229939b24b5bda303c1950a1f

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=508091&tab=buildLog

cockroach-teamcity · 2018-02-07T08:00:15Z

SHA: https://github.com/cockroachdb/cockroach/commits/27384c5c82a6f20679aa7f833875eda6f068e4de

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=509730&tab=buildLog

cockroach-teamcity · 2018-02-08T07:46:02Z

SHA: https://github.com/cockroachdb/cockroach/commits/f45bcea79f3d26f9484075ee174e270e9ccdc714

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=511609&tab=buildLog

cockroach-teamcity · 2018-02-08T09:06:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/f45bcea79f3d26f9484075ee174e270e9ccdc714

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=511790&tab=buildLog

cockroach-teamcity · 2018-02-09T07:57:15Z

SHA: https://github.com/cockroachdb/cockroach/commits/c93d304f8b31b5b0678a46d36a7b751cf50fedb1

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=513483&tab=buildLog

cockroach-teamcity · 2018-02-10T07:56:38Z

SHA: https://github.com/cockroachdb/cockroach/commits/fb37bf0c50592df50959fdb25903616a682e3c97

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=514679&tab=buildLog

cockroach-teamcity · 2018-02-10T10:54:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/fb37bf0c50592df50959fdb25903616a682e3c97

Parameters:

TAGS=deadlock
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=515041&tab=buildLog

cockroach-teamcity · 2018-02-11T08:05:03Z

SHA: https://github.com/cockroachdb/cockroach/commits/2ced98d8cd4d13abae6331e73effc5b13f369c9a

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=515606&tab=buildLog

Fixes cockroachdb#21828. Fixes cockroachdb#21846. Fixes cockroachdb#21844. Fixes cockroachdb#21808. See discussion in cockroachdb#21802. We were previously ignoring the errors returned by job operations in a few places. This meant that we would fail to propagate retryable txn errors when using the jobs API and we could think a txn succeeded when it didn't. This change fixes that. In doing so, it makes schema changes fail when they see errors due to job status updates. I don't see a compelling reason why the old behavior was better and I suspect it was allowing subtle inconsistencies. Release note: None

cockroach-teamcity assigned benesch Jan 26, 2018

cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 26, 2018

mberhault mentioned this issue Feb 2, 2018

jobs: runInTxn cannot AutoRetry, could result in partial transactions #21802

Merged

nvanbenschoten mentioned this issue Feb 2, 2018

sql: don't ignore job status update errors #22337

Merged

nvanbenschoten closed this as completed in #22337 Feb 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql/jobs: TestRegistryResumeActiveLease failed under stress #21808

sql/jobs: TestRegistryResumeActiveLease failed under stress #21808

cockroach-teamcity commented Jan 26, 2018

cockroach-teamcity commented Jan 26, 2018

cockroach-teamcity commented Jan 26, 2018

cockroach-teamcity commented Jan 27, 2018

cockroach-teamcity commented Jan 27, 2018

cockroach-teamcity commented Jan 28, 2018

cockroach-teamcity commented Jan 28, 2018

cockroach-teamcity commented Jan 29, 2018

cockroach-teamcity commented Jan 31, 2018

cockroach-teamcity commented Feb 1, 2018

cockroach-teamcity commented Feb 1, 2018

cockroach-teamcity commented Feb 2, 2018

cockroach-teamcity commented Feb 3, 2018

cockroach-teamcity commented Feb 4, 2018

cockroach-teamcity commented Feb 5, 2018

cockroach-teamcity commented Feb 5, 2018

cockroach-teamcity commented Feb 6, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 11, 2018

sql/jobs: TestRegistryResumeActiveLease failed under stress #21808

sql/jobs: TestRegistryResumeActiveLease failed under stress #21808

Comments

cockroach-teamcity commented Jan 26, 2018

cockroach-teamcity commented Jan 26, 2018

cockroach-teamcity commented Jan 26, 2018

cockroach-teamcity commented Jan 27, 2018

cockroach-teamcity commented Jan 27, 2018

cockroach-teamcity commented Jan 28, 2018

cockroach-teamcity commented Jan 28, 2018

cockroach-teamcity commented Jan 29, 2018

cockroach-teamcity commented Jan 31, 2018

cockroach-teamcity commented Feb 1, 2018

cockroach-teamcity commented Feb 1, 2018

cockroach-teamcity commented Feb 2, 2018

cockroach-teamcity commented Feb 3, 2018

cockroach-teamcity commented Feb 4, 2018

cockroach-teamcity commented Feb 5, 2018

cockroach-teamcity commented Feb 5, 2018

cockroach-teamcity commented Feb 6, 2018

cockroach-teamcity commented Feb 7, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 8, 2018

cockroach-teamcity commented Feb 9, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 10, 2018

cockroach-teamcity commented Feb 11, 2018