Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestTimestampMismatch, testrace/TestTimestampMismatch #21235

Closed
cockroach-teamcity opened this issue Jan 4, 2018 · 0 comments
Closed

teamcity: failed tests on master: test/TestTimestampMismatch, testrace/TestTimestampMismatch #21235

cockroach-teamcity opened this issue Jan 4, 2018 · 0 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#467083:

--- FAIL: test/TestTimestampMismatch (0.000s)
Test ended in panic.

------- Stdout: -------
W180104 18:29:22.854647 139653 server/status/runtime.go:109  Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I180104 18:29:22.856565 139653 server/config.go:516  [n?] 1 storage engine initialized
I180104 18:29:22.856589 139653 server/config.go:519  [n?] RocksDB cache size: 128 MiB
I180104 18:29:22.856598 139653 server/config.go:519  [n?] store 0: in-memory, size 0 B
I180104 18:29:22.859327 139653 server/node.go:361  [n?] **** cluster 4e44d876-f731-4aba-9209-320fc8fd952d has been created
I180104 18:29:22.859365 139653 server/server.go:934  [n?] **** add additional nodes by specifying --join=127.0.0.1:40651
I180104 18:29:22.860088 139653 storage/store.go:1209  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180104 18:29:22.860136 139653 server/node.go:486  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=3.2 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=3322.00 p25=3322.00 p50=3322.00 p75=3322.00 p90=3322.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00}
I180104 18:29:22.860158 139653 server/node.go:339  [n1] node ID 1 initialized
I180104 18:29:22.860235 139653 gossip/gossip.go:332  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:40651" > attrs:<> locality:<> ServerVersion:<major_val:1 minor_val:1 patch:0 unstable:8 > 
I180104 18:29:22.861994 139653 storage/stores.go:331  [n1] read 0 node addresses from persistent storage
I180104 18:29:22.862160 139653 server/node.go:627  [n1] connecting to gossip network to verify cluster ID...
I180104 18:29:22.862359 139653 server/node.go:652  [n1] node connected via gossip and verified as part of cluster "4e44d876-f731-4aba-9209-320fc8fd952d"
I180104 18:29:22.862425 139653 server/node.go:428  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180104 18:29:22.862523 139653 sql/distsql_physical_planner.go:122  [n1] creating DistSQLPlanner with address {tcp 127.0.0.1:40651}
I180104 18:29:22.864672 139734 storage/replica_command.go:819  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180104 18:29:22.866084 139653 server/server.go:1161  [n1] starting https server at 127.0.0.1:38353
I180104 18:29:22.866111 139653 server/server.go:1162  [n1] starting grpc/postgres server at 127.0.0.1:40651
I180104 18:29:22.866120 139653 server/server.go:1163  [n1] advertising CockroachDB node at 127.0.0.1:40651
I180104 18:29:22.874728 139734 storage/replica_command.go:819  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180104 18:29:22.878164 139734 storage/replica_command.go:819  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
W180104 18:29:22.878671 139632 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "sql txn implicit" id=519fa74f key=/Table/SystemConfigSpan/Start rw=true pri=0.04014126 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515090562.875722841,0 orig=1515090562.875722841,0 max=1515090562.875722841,0 wto=false rop=false seq=7
I180104 18:29:22.880169 139653 sql/event_log.go:115  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180104 18:29:22.882912 139734 storage/replica_command.go:819  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180104 18:29:22.884294 139653 sql/lease.go:348  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-04 18:29:22.884133665 +0000 UTC
I180104 18:29:22.887308 139734 storage/replica_command.go:819  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
I180104 18:29:22.894996 139734 storage/replica_command.go:819  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180104 18:29:22.899290 139734 storage/replica_command.go:819  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
W180104 18:29:22.899556 139989 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "sql txn implicit" id=6a1d46ad key=/Table/SystemConfigSpan/Start rw=true pri=0.07436965 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515090562.896734615,0 orig=1515090562.896734615,0 max=1515090562.896734615,0 wto=false rop=false seq=7
I180104 18:29:22.901400 139653 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
I180104 18:29:22.903668 139734 storage/replica_command.go:819  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180104 18:29:22.907364 139734 storage/replica_command.go:819  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180104 18:29:22.909906 139653 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
I180104 18:29:22.912117 139734 storage/replica_command.go:819  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
I180104 18:29:22.913886 139653 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180104 18:29:22.916670 139734 storage/replica_command.go:819  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180104 18:29:22.919848 139734 storage/replica_command.go:819  [split,n1,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180104 18:29:22.924563 139734 storage/replica_command.go:819  [split,n1,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
I180104 18:29:22.928232 139653 sql/event_log.go:115  [n1] Event: "alter_table", target: 4, info: {TableName:users Statement:ALTER TABLE system.users ADD COLUMN IF NOT EXISTS "isRole" BOOL NOT NULL DEFAULT false User:node MutationID:1 CascadeDroppedViews:[]}
I180104 18:29:22.928740 139734 storage/replica_command.go:819  [split,n1,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
I180104 18:29:22.930784 139653 sql/lease.go:348  [n1] publish: descID=4 (users) version=2 mtime=2018-01-04 18:29:22.930514239 +0000 UTC
I180104 18:29:22.931906 139734 storage/replica_command.go:819  [split,n1,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
I180104 18:29:22.937237 139734 storage/replica_command.go:819  [split,n1,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
I180104 18:29:22.937967 139653 sql/lease.go:348  [n1] publish: descID=4 (users) version=3 mtime=2018-01-04 18:29:22.93787144 +0000 UTC
I180104 18:29:22.941348 139734 storage/replica_command.go:819  [split,n1,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
I180104 18:29:22.942696 139653 sql/backfill.go:132  [n1] Running backfill for "users", v=3, m=1
I180104 18:29:22.944687 139734 storage/replica_command.go:819  [split,n1,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
I180104 18:29:22.948168 139734 storage/replica_command.go:819  [split,n1,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
I180104 18:29:22.951871 139653 sql/lease.go:348  [n1] publish: descID=4 (users) version=4 mtime=2018-01-04 18:29:22.951762653 +0000 UTC
I180104 18:29:22.955360 139653 sql/event_log.go:115  [n1] Event: "finish_schema_change", target: 4, info: {MutationID:1}
I180104 18:29:22.956183 139653 sql/lease.go:274  publish (count leases): descID=4 name=users version=3 count=1
panic: test timed out after 4m0s

goroutine 139984 [running]:
testing.startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1145 +0xf9
created by time.goFunc
	/usr/local/go/src/time/sleep.go:170 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200a0780, 0x22d7222, 0x15, 0x23a2ad0, 0x8c7a01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4200a0780)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4200a0780, 0xc4205a9db8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc4204525c0, 0x33946e0, 0x38, 0x38, 0x0)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420cf7f18, 0xc42022d5b0)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl.TestMain(0xc4205a9f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/main_test.go:31 +0xda
main.main()
	github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/_test/_testmain.go:172 +0xdb

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x35e19e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 14 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 139687 [select]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x33bf220, 0xc42421c2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x6b4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204120a0, 0xc423bf45a0, 0xc420412090)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 50 [select, locked to thread]:
runtime.gopark(0x23a6808, 0x0, 0x22c0338, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc420950f50, 0xc420440480)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 139902 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000126)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc420be4990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc420be4990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202252a0, 0xc423bf45a0, 0xc420225290)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139933 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x33bf220, 0xc422ab5e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba58e0, 0xc423bf45a0, 0xc424ba58d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139705 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x33bf220, 0xc420f5ec90)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:718 +0x14f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba55e0, 0xc423bf45a0, 0xc421913e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139924 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x33bf220, 0xc423a2acf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba57c0, 0xc423bf45a0, 0xc424ba57b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139895 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000129)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc420be4a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc420be4a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202251c0, 0xc423bf45a0, 0xc4202251b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139869 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc40000010a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4ae70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4ae70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abdc0, 0xc423bf45a0, 0xc4203abdb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139660 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x33bf220, 0xc4238bae10)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:748 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ef280, 0xc423bf45a0, 0xc422570ee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139928 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x33bf220, 0xc420be4ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba5840, 0xc423bf45a0, 0xc424ba5830)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139627 [IO wait]:
internal/poll.runtime_pollWait(0x7f034a6bba08, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420c42118, 0x72, 0xffffffffffffff00, 0x33a97a0, 0x339c760)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420c42118, 0xc421c88400, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420c42100, 0xc421c88400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420c42100, 0xc421c88400, 0x400, 0x400, 0x26, 0x8, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc422114048, 0xc421c88400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
crypto/tls.(*block).readFromUntil(0xc421886e70, 0x7f034d949498, 0xc422114048, 0x5, 0xc422114048, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc423738a80, 0x23a6917, 0xc423738ba0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc423738a80, 0xc423510000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc4249d61e0, 0xc421e20118, 0x9, 0x9, 0x28, 0x2191300, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x339e620, 0xc4249d61e0, 0xc421e20118, 0x9, 0x9, 0x9, 0xc421cc1de8, 0x839e1e, 0xc421cc1e97)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x339e620, 0xc4249d61e0, 0xc421e20118, 0x9, 0x9, 0xcff5f4, 0xc424c520c0, 0xc424c50004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc421e20118, 0x9, 0x9, 0x339e620, 0xc4249d61e0, 0x0, 0x0, 0xc424c520c0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc421e200e0, 0xc424c520c0, 0xc424c520c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4220a4000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1118 +0xbe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:266 +0xc80

goroutine 139707 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x33bf220, 0xc422068c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:112 +0x2db
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba5640, 0xc423bf45a0, 0xc421913e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139892 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000125)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc420be49c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc420be49c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420225160, 0xc423bf45a0, 0xc420225150)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139692 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc421364000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1002 +0x166
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc421364000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:703 +0x1ce
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:694 +0x40d

goroutine 139903 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000132)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc422ab54d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc422ab54d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202252c0, 0xc423bf45a0, 0xc4202252b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139644 [IO wait]:
internal/poll.runtime_pollWait(0x7f034d9ebdf0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc423c98718, 0x72, 0xc42003cc00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc423c98718, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc423c98700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc423c98700, 0xc420516030, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4203562b0, 0x14b30fa, 0x893100, 0xc42003ce80)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4203562b0, 0x23a4c80, 0xc422157580, 0x33cb1c0, 0xc420516030)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc422157580, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x95
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func10.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:965 +0x2f
sync.(*Once).Do(0xc4202b1d90, 0xc42146e758)
	/usr/local/go/src/sync/once.go:44 +0xbe
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func10(0x33bf220, 0xc420b1dbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:964 +0x5b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ab5a0, 0xc423bf45a0, 0xc422574dc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139853 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc40000010d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abb80, 0xc423bf45a0, 0xc4203abb70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139879 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000123)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abff0, 0xc423bf45a0, 0xc4203abfe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139849 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000106)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4ae40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4ae40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abb00, 0xc423bf45a0, 0xc4203abaf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139843 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000101)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4ad20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4ad20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ab9a0, 0xc423bf45a0, 0xc4203ab990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139955 [select]:
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations.func2(0x33bf220, 0xc4242f14a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:323 +0x100
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc423bf45a0, 0x33bf220, 0xc4242f14a0, 0xc422ed9950, 0x29, 0x0, 0x0, 0xc422574a60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x133

goroutine 139935 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x33bf220, 0xc421886090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1023 +0x9b6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba5910, 0xc423bf45a0, 0xc421913ee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139873 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000120)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abe80, 0xc423bf45a0, 0xc4203abe60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139740 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x33bf220, 0xc422ab4000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210dedd0, 0xc423bf45a0, 0xc4210b20c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139764 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x33bf220, 0xc422068450)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1151 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ef570, 0xc423bf45a0, 0xc4202ef550)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139841 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddToPurgatory.func2(0x33bf220, 0xc422de48d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:705 +0x434
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038c900, 0xc423bf45a0, 0xc420cb6f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139910 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x33bf220, 0xc423a2ac90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1179 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202253b0, 0xc423bf45a0, 0xc422575b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139890 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000139)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b560)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b560)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420225120, 0xc423bf45a0, 0xc420225110)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139877 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000124)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b380)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b380)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abfb0, 0xc423bf45a0, 0xc4203abfa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139867 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc40000011a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abd60, 0xc423bf45a0, 0xc4203abd50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139932 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x33bf220, 0xc422ab5dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x167
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ba58c0, 0xc423bf45a0, 0xc424ba58b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139885 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc40000013a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc422ab55f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc422ab55f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420224e20, 0xc423bf45a0, 0xc420224e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139983 [runnable]:
github.com/cockroachdb/cockroach/pkg/sql/parser.(*sqlParserImpl).Parse(0xc420b492b8, 0x33abd20, 0xc420b49200, 0x480)
	yaccpar:156 +0x65983
github.com/cockroachdb/cockroach/pkg/sql/parser.(*Parser).parseWithDepth(0xc420b49200, 0x2, 0x233efd2, 0x5b, 0x0, 0x0, 0x0, 0x0, 0xc421f57518)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/parser/parse.go:51 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/parser.parseWithDepth(0x1, 0x233efd2, 0x5b, 0x0, 0x0, 0x0, 0xc421f57540, 0xe1fcd2)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/parser/parse.go:72 +0x60
github.com/cockroachdb/cockroach/pkg/sql/parser.ParseOne(0x233efd2, 0x5b, 0x0, 0x0, 0x22cf38c, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/parser/parse.go:78 +0x42
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeInternalPlan(0xc42334f500, 0x33bf160, 0xc420d30ec0, 0x233efd2, 0x5b, 0xc420d30f00, 0x4, 0x4, 0x3e8, 0xc421f57700, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planner.go:255 +0x69
github.com/cockroachdb/cockroach/pkg/sql.(*planner).exec(0xc42334f500, 0x33bf160, 0xc420d30ec0, 0x233efd2, 0x5b, 0xc420d30f00, 0x4, 0x4, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planner.go:333 +0x105
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.release.func1(0x33bf160, 0xc420d30ec0, 0xc424102780, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:218 +0x361
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x33bf160, 0xc420d30ec0, 0xc424102780, 0xc4225eeeaa, 0x0, 0x22c2453)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:508 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc424102780, 0x33bf160, 0xc420d30ec0, 0xc420d30101, 0xc42201a480, 0x3b9aca00, 0x4000000000000000)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:753 +0xfe
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc420a70150, 0x33bf160, 0xc420d30ec0, 0xc421c54020, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:507 +0x140
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.release(0x339efe0, 0xc4219a0b60, 0xc422156600, 0x3ff0000000000000, 0xc422156600, 0xc421ba4500, 0x45d964b800, 0x3fd0000000000000, 0xdf8475800, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:207 +0x2d1
github.com/cockroachdb/cockroach/pkg/sql.(*tableState).releaseLease.func1(0x33bf1a0, 0xc420016110)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:921 +0xaa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc423bf45a0, 0x33bf1a0, 0xc420016110, 0xc420302600, 0x32, 0x0, 0x0, 0xc420403ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x133

goroutine 139658 [semacquire]:
sync.runtime_notifyListWait(0xc4200c6d78, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4200c6d68)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4200c6c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:625 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:612 +0x4a3

goroutine 139734 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x33bf220, 0xc42421c030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210ded70, 0xc423bf45a0, 0xc4210b2000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139653 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc421e71e78, 0xc420016110)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:119 +0x189
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.WaitForOneVersion(0x339efe0, 0xc4219a0b60, 0xc422156600, 0x3ff0000000000000, 0xc422156600, 0xc421ba4500, 0x45d964b800, 0x3fd0000000000000, 0xdf8475800, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:253 +0x2bc
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).waitToUpdateLeases(0xc421e7a848, 0x33bf160, 0xc4214d8240, 0xc400000004, 0x23a3118, 0xc42059a820)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:602 +0x108
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec.func2(0xc421e7a848, 0x33bf160, 0xc4214d8240)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:435 +0x49
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec(0xc421e7a848, 0x33bf160, 0xc4214d8240, 0x1, 0x0, 0x0, 0x0, 0x0, 0x375438d9, 0xed1e06982, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:485 +0x444
github.com/cockroachdb/cockroach/pkg/sql.(*schemaChangerCollection).execSchemaChanges(0xc4238a90f0, 0x33bf160, 0xc4214d8240, 0xc423e706c0, 0xc4238a9000, 0xed1e00101, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/session.go:1357 +0x4f5
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execParsed(0xc423e706c0, 0xc4238a9000, 0xc4238d4690, 0x1, 0x1, 0x0, 0x0, 0xc422081d40, 0x9f5be235370)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:923 +0x449
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execRequest(0xc423e706c0, 0xc4238a9000, 0x23404be, 0x62, 0x0, 0x0, 0x62, 0x21c0ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:744 +0x49c
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).ExecuteStatements(0xc423e706c0, 0xc4238a9000, 0x23404be, 0x62, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:629 +0x1bd
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).ExecuteStatementsBuffered(0xc423e706c0, 0xc4238a9000, 0x23404be, 0x62, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:577 +0xe5
github.com/cockroachdb/cockroach/pkg/sqlmigrations.addRoles(0x33bf220, 0xc4242f14a0, 0x33b0ee0, 0xc422570b00, 0xc423e706c0, 0xc421ba4958, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:710 +0x10c
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations(0xc4225362c0, 0x33bf220, 0xc4242f14a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:378 +0xac4
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc421ba4500, 0x33bf220, 0xc4242f14a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1223 +0x32f9
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc4234aba40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x137
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:206 +0xef
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x33cfe00, 0xc4204880f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:176 +0x55
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).doAddServer(0xc423ecaaf0, 0x33cfe00, 0xc4204880f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:247 +0x112
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster(0x33cfe00, 0xc4204880f0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:155 +0x55a
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetupWithParams(0x33cfe00, 0xc4204880f0, 0x1, 0x1, 0x23a2ae0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:86 +0xfe
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.backupRestoreTestSetup(0x33cfe00, 0xc4204880f0, 0x1, 0x1, 0x23a2ae0, 0x27d9480, 0xf, 0x3054120, 0x24, 0x2b3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:151 +0x96
github.com/cockroachdb/cockroach/pkg/ccl/sqlccl_test.TestTimestampMismatch(0xc4204880f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/backup_test.go:1876 +0xa8
testing.tRunner(0xc4204880f0, 0x23a2ad0)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 139878 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000127)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abfd0, 0xc423bf45a0, 0xc4203abfc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139719 [semacquire]:
sync.runtime_notifyListWait(0xc424829240, 0xc400000004)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc424829230)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc424829200, 0xc421e64000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc420fb42b8, 0xc421e64000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc420f5e060, 0x7f034d949470, 0xc4203ab610, 0x5, 0xc4203ab610, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc422d2fc00, 0x23a6917, 0xc422d2fd20, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc422d2fc00, 0xc4215e0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc423ce8480, 0xc4219822d8, 0x9, 0x9, 0x28, 0x2191300, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x339e620, 0xc423ce8480, 0xc4219822d8, 0x9, 0x9, 0x9, 0xc42003ede8, 0x839e1e, 0xc42003ee97)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x339e620, 0xc423ce8480, 0xc4219822d8, 0x9, 0x9, 0xcff5f4, 0xc422af02d0, 0xc422af0004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4219822d8, 0x9, 0x9, 0x339e620, 0xc423ce8480, 0x0, 0x0, 0xc422af02d0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4219822a0, 0xc422af02d0, 0xc422af02d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc422a7ea80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1118 +0xbe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:266 +0xc80

goroutine 139888 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000138)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420225040, 0xc423bf45a0, 0xc420224fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139875 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000121)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203abee0, 0xc423bf45a0, 0xc4203abec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139733 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x33bf220, 0xc420be4c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210ded60, 0xc423bf45a0, 0xc4205d5fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139743 [IO wait]:
internal/poll.runtime_pollWait(0x7f034a6bbd08, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4216dc018, 0x72, 0xffffffffffffff00, 0x33a97a0, 0x339c760)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4216dc018, 0xc420e49800, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4216dc000, 0xc420e49800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4216dc000, 0xc420e49800, 0x400, 0x400, 0x301, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc420516030, 0xc420e49800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc422dba500, 0xc420e49800, 0x400, 0x400, 0x21bcfc0, 0xc422ab5000, 0x7f034d9494c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x121
crypto/tls.(*block).readFromUntil(0xc422ab50e0, 0x7f034d9494c0, 0xc422dba500, 0x5, 0xc422dba500, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc421c62e00, 0x23a6917, 0xc421c62f20, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc421c62e00, 0xc4235d0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc4243d6300, 0xc423dbeab8, 0x9, 0x9, 0x28, 0x2191300, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x339e620, 0xc4243d6300, 0xc423dbeab8, 0x9, 0x9, 0x9, 0xc421d49bb8, 0x839e1e, 0xc421d49c67)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x339e620, 0xc4243d6300, 0xc423dbeab8, 0x9, 0x9, 0xcff5f4, 0xc422af0300, 0xc422af0004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc423dbeab8, 0x9, 0x9, 0x339e620, 0xc4243d6300, 0x0, 0x0, 0xc422af0300, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc423dbea80, 0xc422af0300, 0xc422af0300, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4200c66e0, 0xc424c52060, 0x23a5c20)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x315
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc421ff8640, 0x33ca2c0, 0xc4200c66e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:568 +0x140
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc421ff8640, 0x33ca740, 0xc421c62e00, 0x33a8b20, 0xc4209ead10)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:561 +0x471
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc421ff8640, 0x33cafe0, 0xc422dba500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:526 +0x497
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:495 +0x5b9

goroutine 139880 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc400000128)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc423e4b410)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc423e4b410)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420224110, 0xc423bf45a0, 0xc4202240e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139909 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc420277800, 0x33bf220, 0xc420f5e8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3788 +0x16d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x33bf220, 0xc420f5e8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3734 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420225380, 0xc423bf45a0, 0xc420225370)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139657 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x33bf220, 0xc420be4420)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1469 +0x912
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202eef20, 0xc423bf45a0, 0xc42320e630)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139893 [semacquire]:
sync.runtime_notifyListWait(0xc423ceecd0, 0xc40000012b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423ceecc0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421f830e0, 0x33bf220, 0xc420be49f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x33bf220, 0xc420be49f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420225180, 0xc423bf45a0, 0xc420225170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xad

goroutine 139837 [semacquire]:
sync.runtime_notifyListWait(0xc424829180, 0xc400000004)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc424829170)
	/usr/local/go/src/sync/cond.go:56 +0x80
io.(*pipe).read(0xc424829140, 0xc423841000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:47 +0xc6
io.(*PipeReader).Read(0xc420fb42a0, 0xc423841000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:130 +0x4c
crypto/tls.(*block).readFromUntil(0xc420b1dc80, 0x7f034d949470, 0xc4203ab600, 0x5, 0xc4203ab600, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc422d2f880, 0x23a6917, 0xc422d2f9a0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc422d2f880, 0xc421f60000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc421f7e540, 0xc42497fa78, 0x9, 0x9, 0x28, 0x2191300, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x339e620, 0xc421f7e540, 0xc42497fa78, 0x9, 0x9, 0x9, 0xc423419bb8, 0x839e1e, 0xc423419c67)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x339e620, 0xc421f7e540, 0xc42497fa78, 0x9, 0x9, 0xcff5f4, 0xc422af02a0, 0xc422af0004)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42497fa78, 0x9, 0x9, 0x339e620, 0xc421f7e540, 0x0, 0x0, 0xc422af02a0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42497fa40, 0xc422af02a0, 0xc422af02a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420cba840, 0xc422af0240, 0x23a5c20)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:393 +0x315
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc421ff8640, 0x33ca2c0, 0xc420cba840)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:568 +0x140
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc421ff8640, 0x33ca740, 0xc422d2f880, 0x33a8b20, 0xc420eee9a0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:561 +0x471
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc421ff8640,

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 4, 2018
@benesch benesch mentioned this issue Jan 16, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity