Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: acceptance/TestAdminLossOfQuorum #10436

Closed
cockroach-teamcity opened this issue Nov 3, 2016 · 1 comment
Closed

teamcity: failed tests on master: acceptance/TestAdminLossOfQuorum #10436

cockroach-teamcity opened this issue Nov 3, 2016 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
Q2

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#41280:

--- FAIL: acceptance/TestAdminLossOfQuorum (599.698s)
Test ended in panic.
I161103 15:59:06.287817 22 acceptance/cluster/localcluster.go:291  creating docker network with name: cockroachdb_acceptance-37eb9fa0
I161103 15:59:06.384153 22 acceptance/cluster/localcluster.go:323  Initializing Cluster AdHoc 3x1:
{"name":"AdHoc 3x1","nodes":[{"count":3,"stores":[{"count":1,"max_ranges":0}]}],"duration":5000000000}
I161103 15:59:06.387985 22 acceptance/cluster/docker.go:107  ImagePull cockroachdb/builder:20161024-143446 already exists
I161103 15:59:07.537208 22 acceptance/cluster/localcluster.go:642  creating certs (1024bit) in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613
I161103 15:59:07.627751 22 acceptance/cluster/localcluster.go:426  creating docker container with name: roach-37eb9fa0-0
I161103 15:59:08.416595 22 acceptance/cluster/localcluster.go:537  *** started roach-37eb9fa0-0 ***
  ui:        https://127.0.0.1:32769
  trace:     https://127.0.0.1:32769/debug/requests
  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestAdminLossOfQuorum-37eb9fa0/roach-37eb9fa0-0/cockroach.INFO
  pprof:     docker exec -it 7f913 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'
  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach-37eb9fa0-0 --alsologtostderr=INFO --verbosity=1 --store=path=/data1.0 --log-dir=/logs/roach-37eb9fa0-0 --logtostderr=false]

  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=32769
I161103 15:59:08.416662 22 acceptance/cluster/localcluster.go:426  creating docker container with name: roach-37eb9fa0-1
I161103 15:59:09.112564 22 acceptance/cluster/localcluster.go:537  *** started roach-37eb9fa0-1 ***
  ui:        https://127.0.0.1:32771
  trace:     https://127.0.0.1:32771/debug/requests
  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestAdminLossOfQuorum-37eb9fa0/roach-37eb9fa0-1/cockroach.INFO
  pprof:     docker exec -it 686a9 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'
  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach-37eb9fa0-1 --alsologtostderr=INFO --verbosity=1 --store=path=/data2.0 --join=roach-37eb9fa0-0:26257 --log-dir=/logs/roach-37eb9fa0-1 --logtostderr=false]

  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=32771
I161103 15:59:09.112639 22 acceptance/cluster/localcluster.go:426  creating docker container with name: roach-37eb9fa0-2
I161103 15:59:09.858748 22 acceptance/cluster/localcluster.go:537  *** started roach-37eb9fa0-2 ***
  ui:        https://127.0.0.1:32773
  trace:     https://127.0.0.1:32773/debug/requests
  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestAdminLossOfQuorum-37eb9fa0/roach-37eb9fa0-2/cockroach.INFO
  pprof:     docker exec -it 417d2 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'
  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach-37eb9fa0-2 --alsologtostderr=INFO --verbosity=1 --store=path=/data3.0 --join=roach-37eb9fa0-0:26257 --log-dir=/logs/roach-37eb9fa0-2 --logtostderr=false]

  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.239558613/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=32773
I161103 15:59:09.858789 22 acceptance/replication_test.go:55  waiting for first range to have 3 replicas
I161103 15:59:10.861343 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
I161103 15:59:11.864291 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
I161103 15:59:12.867879 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
I161103 15:59:13.870910 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
I161103 15:59:14.873765 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
I161103 15:59:15.876509 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
I161103 15:59:16.878888 22 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks
panic: test timed out after 10m0s

goroutine 118 [running]:
panic(0x13d3160, 0xc420510a10)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
testing.startAlarm.func1()
	/usr/local/go/src/testing/testing.go:918 +0x10b
created by time.goFunc
	/usr/local/go/src/time/sleep.go:154 +0x44

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc420454180, 0x156ae53, 0x15, 0x16ac170, 0xc4202a9c58)
	/usr/local/go/src/testing/testing.go:647 +0x316
testing.RunTests.func1(0xc420454180)
	/usr/local/go/src/testing/testing.go:793 +0x6d
testing.tRunner(0xc420454180, 0xc420054da8)
	/usr/local/go/src/testing/testing.go:610 +0x81
testing.RunTests(0x16aeb48, 0x1e8e220, 0x24, 0x24, 0xc4202a9df0)
	/usr/local/go/src/testing/testing.go:799 +0x2f5
testing.(*M).Run(0xc420054ef8, 0x16ac2c0)
	/usr/local/go/src/testing/testing.go:743 +0x85
github.com/cockroachdb/cockroach/pkg/acceptance.runTests(0xc420054ef8)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:138 +0x48
github.com/cockroachdb/cockroach/pkg/acceptance.TestMain(0xc420054ef8)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/main_test.go:38 +0x45
main.main()
	github.com/cockroachdb/cockroach/pkg/acceptance/_test/_testmain.go:122 +0xc6

goroutine 17 [syscall, 10 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 8 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc42015cb00)
	/usr/local/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1063 +0x50e

goroutine 7 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7f486d92f140, 0x72, 0x4)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4203fcdf0, 0x72, 0xc42003e9d0, 0xc420014068)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4203fcdf0, 0x1e9ebc0, 0xc420014068)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4203fcd90, 0xc420266000, 0x1000, 0x1000, 0x0, 0x1e9ebc0, 0xc420014068)
	/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420506020, 0xc420266000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc42015cb00, 0xc420266000, 0x1000, 0x1000, 0x8a0ec0, 0xc42003eb58, 0x5c90ad)
	/usr/local/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc4202f4420)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4202f4420, 0x1, 0xc42003ebbd, 0x1, 0x0, 0xc4202992c0, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc42015cb00)
	/usr/local/go/src/net/http/transport.go:1418 +0x1a1
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1062 +0x4e9

goroutine 34 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x21f4dc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1001 +0x77
created by github.com/cockroachdb/cockroach/pkg/util/log.init.1
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:579 +0x95

goroutine 35 [syscall, 10 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 9 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7f486d92f080, 0x72, 0x6)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4203fce60, 0x72, 0xc420145818, 0xc420014068)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4203fce60, 0x1e9ebc0, 0xc420014068)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4203fce00, 0xc420164000, 0x1000, 0x1000, 0x0, 0x1e9ebc0, 0xc420014068)
	/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420032178, 0xc420164000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc4204ea100, 0xc420164000, 0x1000, 0x1000, 0x9cca38, 0xc4204e0388, 0x7d)
	/usr/local/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc4201dc5a0)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).ReadSlice(0xc4201dc5a0, 0x10a, 0x2, 0x0, 0x199, 0x186, 0x186)
	/usr/local/go/src/bufio/bufio.go:330 +0xb5
net/http/internal.readChunkLine(0xc4201dc5a0, 0x5d3659, 0xc4201dc058, 0x0, 0x1, 0xc420145a88)
	/usr/local/go/src/net/http/internal/chunked.go:110 +0x34
net/http/internal.(*chunkedReader).beginChunk(0xc420012840)
	/usr/local/go/src/net/http/internal/chunked.go:47 +0x32
net/http/internal.(*chunkedReader).Read(0xc420012840, 0xc420078001, 0x5ff, 0x5ff, 0x619db0, 0xc420145b70, 0xc420145b78)
	/usr/local/go/src/net/http/internal/chunked.go:77 +0x7b
net/http.(*body).readLocked(0xc420098040, 0xc420078001, 0x5ff, 0x5ff, 0x2, 0x2, 0x0)
	/usr/local/go/src/net/http/transfer.go:651 +0x61
net/http.(*body).Read(0xc420098040, 0xc420078001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transfer.go:643 +0xf6
net/http.(*bodyEOFSignal).Read(0xc420098080, 0xc420078001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:1974 +0x116
encoding/json.(*Decoder).refill(0xc4204e0340, 0xa, 0x9)
	/usr/local/go/src/encoding/json/stream.go:152 +0xfa
encoding/json.(*Decoder).readValue(0xc4204e0340, 0x0, 0x0, 0x80)
	/usr/local/go/src/encoding/json/stream.go:128 +0x198
encoding/json.(*Decoder).Decode(0xc4204e0340, 0x13a1e60, 0xc42013ce00, 0x0, 0x0)
	/usr/local/go/src/encoding/json/stream.go:57 +0x8e
github.com/docker/docker/client.(*Client).Events.func1(0xc4201dc060, 0xc420493500, 0x0, 0x0, 0x0, 0x0, 0xc4201ee060, 0x7f486d8ee7e8, 0xc4202f1480, 0xc4201dc000)
	/go/src/github.com/docker/docker/client/events.go:50 +0x2a7
created by github.com/docker/docker/client.(*Client).Events
	/go/src/github.com/docker/docker/client/events.go:63 +0xe2

goroutine 21 [chan receive, 10 minutes]:
github.com/cockroachdb/cockroach/pkg/acceptance.runTests.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:129 +0xb6
created by github.com/cockroachdb/cockroach/pkg/acceptance.runTests
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:137 +0x3a

goroutine 22 [select, 9 minutes]:
google.golang.org/grpc/transport.(*Stream).Header(0xc4205361e0, 0x16ae980, 0xc4205a51e8, 0x1eb0300)
	/go/src/google.golang.org/grpc/transport/transport.go:239 +0x299
google.golang.org/grpc.recvResponse(0x1eaa280, 0x222de70, 0x0, 0x0, 0x0, 0x0, 0x1e9ddc0, 0xc420582020, 0x0, 0x0, ...)
	/go/src/google.golang.org/grpc/call.go:62 +0xad
google.golang.org/grpc.Invoke(0x7f486d8b3648, 0xc420014558, 0x15767ac, 0x21, 0x1541320, 0xc4203b60e0, 0x14e16a0, 0xc420376060, 0xc42015d200, 0x0, ...)
	/go/src/google.golang.org/grpc/call.go:202 +0x8a1
github.com/cockroachdb/cockroach/pkg/roachpb.(*externalClient).Batch(0xc4201262f8, 0x7f486d8b3648, 0xc420014558, 0xc4203b60e0, 0x0, 0x0, 0x0, 0x0, 0x16a8881, 0xc4203a7700)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:1562 +0xd2
github.com/cockroachdb/cockroach/pkg/internal/client.sender.Send(0x1e97bc0, 0xc4201262f8, 0x7f486d8b3648, 0xc420014558, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/rpc_sender.go:44 +0xbf
github.com/cockroachdb/cockroach/pkg/internal/client.(*sender).Send(0xc420341180, 0x7f486d8b3648, 0xc420014558, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	<autogenerated>:4 +0xb0
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc4205806e0, 0x7f486d8b3648, 0xc420014558, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:511 +0xbf
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:445 +0x6e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0xc4202480a0, 0xc420078600, 0x0, 0x1eb0d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:416 +0x168
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc4205806e0, 0x7f486d8b3648, 0xc420014558, 0xc420078600, 0x8, 0x18)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:447 +0xac
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Get(0xc4205806e0, 0x7f486d8b3648, 0xc420014558, 0x14b9060, 0xc420248080, 0xc420248080, 0x18, 0x18, 0x1, 0xc420248080, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:214 +0x9a
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).GetProto(0xc4205806e0, 0x7f486d8b3648, 0xc420014558, 0x14b9060, 0xc420248080, 0x1ea4300, 0xc420376000, 0xc4202d19e0, 0x60033c)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:222 +0x69
github.com/cockroachdb/cockroach/pkg/acceptance.countRangeReplicas(0xc4205806e0, 0xc42023d3e0, 0x0, 0xc420580601)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/replication_test.go:36 +0x157
github.com/cockroachdb/cockroach/pkg/acceptance.checkRangeReplication.func1(0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/replication_test.go:71 +0x1f4
github.com/cockroachdb/cockroach/pkg/util.RetryForDuration(0xa7a358200, 0xc4205a5ce0, 0x2, 0x158337f)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/testing.go:129 +0xcd
github.com/cockroachdb/cockroach/pkg/util.SucceedsSoonDepth(0x1, 0x1ead6c0, 0xc420454240, 0xc4205a5ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/testing.go:115 +0x41
github.com/cockroachdb/cockroach/pkg/util.SucceedsSoon(0x1ead6c0, 0xc420454240, 0xc4205a5ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/testing.go:109 +0x48
github.com/cockroachdb/cockroach/pkg/acceptance.checkRangeReplication(0xc420454240, 0x1eb1e20, 0xc4204f8100, 0x4a817c800)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/replication_test.go:83 +0x2d3
github.com/cockroachdb/cockroach/pkg/acceptance.StartCluster(0xc420454240, 0xc420125630, 0x9, 0xc420128dc0, 0x1, 0x1, 0x12a05f200, 0x1eb1e20, 0xc4204f8100)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:345 +0x188
github.com/cockroachdb/cockroach/pkg/acceptance.runTestOnConfigs.func1(0xc420454240, 0xc4205a5ec8, 0x16ac2d8)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:312 +0x4e
github.com/cockroachdb/cockroach/pkg/acceptance.runTestOnConfigs(0xc420454240, 0x16ac2d8)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:315 +0xe2
github.com/cockroachdb/cockroach/pkg/acceptance.TestAdminLossOfQuorum(0xc420454240)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/admin_test.go:34 +0x37
testing.tRunner(0xc420454240, 0x16ac170)
	/usr/local/go/src/testing/testing.go:610 +0x81
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:646 +0x2ec

goroutine 37 [select, 10 minutes, locked to thread]:
runtime.gopark(0x16af400, 0x0, 0x155b965, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc4203b3730, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc4203b3730)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 31 [select, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/acceptance/cluster.(*LocalCluster).monitor.func1(0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/cluster/localcluster.go:609 +0x4aa
github.com/cockroachdb/cockroach/pkg/acceptance/cluster.(*LocalCluster).monitor(0xc4204f8100)
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/cluster/localcluster.go:629 +0x61
created by github.com/cockroachdb/cockroach/pkg/acceptance/cluster.(*LocalCluster).Start
	/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/cluster/localcluster.go:653 +0x57e

goroutine 11 [select, 9 minutes]:
net/http.(*persistConn).readLoop(0xc4204ea100)
	/usr/local/go/src/net/http/transport.go:1541 +0x9c9
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1062 +0x4e9

goroutine 12 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc4204ea100)
	/usr/local/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1063 +0x50e

goroutine 92 [select, 9 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420487b80)
	/go/src/google.golang.org/grpc/clientconn.go:720 +0x63e
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420487b80)
	/go/src/google.golang.org/grpc/clientconn.go:479 +0x1de
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/google.golang.org/grpc/clientconn.go:480 +0x3c7

goroutine 93 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc42013ca80, 0xc42015d200, 0xc420340380, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:291 +0x617
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:226 +0x69
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42018b950, 0xc4204a35c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66

goroutine 41 [IO wait]:
net.runtime_pollWait(0x7f486d92efc0, 0x72, 0x7)
	/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420379f70, 0x72, 0xc420043828, 0xc420014068)
	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420379f70, 0x1e9ebc0, 0xc420014068)
	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc420379f10, 0xc420066800, 0x400, 0x400, 0x0, 0x1e9ebc0, 0xc420014068)
	/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420506158, 0xc420066800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc4204a35f0, 0x7f486d8eec38, 0xc420506158, 0x5, 0xc420506158, 0xc4205100d0)
	/usr/local/go/src/crypto/tls/conn.go:476 +0x91
crypto/tls.(*Conn).readRecord(0xc4202e4700, 0x16af517, 0xc4202e4808, 0xc4205109d0)
	/usr/local/go/src/crypto/tls/conn.go:578 +0xc4
crypto/tls.(*Conn).Read(0xc4202e4700, 0xc42059a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1113 +0x116
bufio.(*Reader).fill(0xc420596840)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc420596840, 0xc42024a0f8, 0x9, 0x9, 0xc420043ca8, 0x619eb0, 0xc4205033c0)
	/usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1e97240, 0xc420596840, 0xc42024a0f8, 0x9, 0x9, 0x9, 0xc420153c00, 0xc420153bf8, 0x1e9b000)
	/usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1e97240, 0xc420596840, 0xc42024a0f8, 0x9, 0x9, 0xc420043eb0, 0xee1df0, 0xc4204a4ff0)
	/usr/local/go/src/io/io.go:325 +0x58
golang.org/x/net/http2.readFrameHeader(0xc42024a0f8, 0x9, 0x9, 0x1e97240, 0xc420596840, 0x0, 0xc400000000, 0xc4205100d0, 0xc)
	/go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc42024a0c0, 0x0, 0x0, 0x0, 0x0)
	/go/src/golang.org/x/net/http2/frame.go:464 +0xa4
google.golang.org/grpc/transport.(*framer).readFrame(0xc4201f2300, 0xc4201ee0c0, 0xc4201ee0c0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f
google.golang.org/grpc/transport.(*http2Client).reader(0xc42042cb00)
	/go/src/google.golang.org/grpc/transport/http2_client.go:921 +0xb6
created by google.golang.org/grpc/transport.newHTTP2Client
	/go/src/google.golang.org/grpc/transport/http2_client.go:203 +0x8b1

goroutine 103 [chan receive, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:137 +0x76
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42018b950, 0xc4202e97e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66

goroutine 130 [select, 9 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc42042cb00)
	/go/src/google.golang.org/grpc/transport/http2_client.go:999 +0x59d
created by google.golang.org/grpc/transport.newHTTP2Client
	/go/src/google.golang.org/grpc/transport/http2_client.go:233 +0xe78

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added this to the Q2 milestone Nov 3, 2016
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Nov 3, 2016
@tamird
Copy link
Contributor

tamird commented Nov 3, 2016

#10409.

@tamird tamird closed this as completed Nov 3, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
Q2
Development

No branches or pull requests
2 participants
@tamird @cockroach-teamcity