Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

acceptance: TestClusterRecovery is skipped #8538

Closed
cockroach-teamcity opened this issue Aug 15, 2016 · 7 comments · Fixed by #10514
Closed

acceptance: TestClusterRecovery is skipped #8538

cockroach-teamcity opened this issue Aug 15, 2016 · 7 comments · Fixed by #10514
Assignees
@jordanlewis
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
Q2

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#11286:

--- FAIL: acceptance/TestClusterRecovery (597.264s)
Test ended in panic.
I160815 03:12:19.284514 acceptance/cluster/localcluster.go:299  Initializing Cluster AdHoc 3x1:
{"name":"AdHoc 3x1","nodes":[{"count":3,"stores":[{"count":1,"max_ranges":0}]}],"duration":5000000000}
I160815 03:12:19.289270 acceptance/cluster/docker.go:106  ImagePull cockroachdb/builder:20160808-065131 already exists
I160815 03:12:19.668890 acceptance/cluster/localcluster.go:605  creating certs (1024bit) in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299
I160815 03:12:19.842386 acceptance/cluster/localcluster.go:556  events monitor starts
I160815 03:12:20.436597 acceptance/cluster/localcluster.go:505  *** started roach0 ***
  ui:        https://127.0.0.1:38208
  trace:     https://127.0.0.1:38208/debug/requests
  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestClusterRecovery/roach0/cockroach.INFO
  pprof:     docker exec -it 9f34c /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'
  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach0 --alsologtostderr=INFO --verbosity=1 --store=path=/data1.0 --log-dir=/logs/roach0 --logtostderr=false]

  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=38208
I160815 03:12:20.842579 acceptance/cluster/localcluster.go:505  *** started roach1 ***
  ui:        https://127.0.0.1:38210
  trace:     https://127.0.0.1:38210/debug/requests
  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestClusterRecovery/roach1/cockroach.INFO
  pprof:     docker exec -it 52529 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'
  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach1 --alsologtostderr=INFO --verbosity=1 --store=path=/data2.0 --join=roach0:26257 --log-dir=/logs/roach1 --logtostderr=false]

  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=38210
I160815 03:12:21.257610 acceptance/cluster/localcluster.go:505  *** started roach2 ***
  ui:        https://127.0.0.1:38212
  trace:     https://127.0.0.1:38212/debug/requests
  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestClusterRecovery/roach2/cockroach.INFO
  pprof:     docker exec -it c9513 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'
  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach2 --alsologtostderr=INFO --verbosity=1 --store=path=/data3.0 --join=roach0:26257 --log-dir=/logs/roach2 --logtostderr=false]

  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/.localcluster.certs.315781299/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=38212
I160815 03:12:21.257628 acceptance/replication_test.go:55  waiting for first range to have 3 replicas
I160815 03:12:21.259681 rpc/context.go:170  dialing 127.0.0.1:38207
I160815 03:12:22.261237 acceptance/replication_test.go:77  found 3 replicas
I160815 03:12:22.261260 rpc/context.go:141  closing 127.0.0.1:38207
I160815 03:12:22.261324 acceptance/replication_test.go:85  found 3 replicas
W160815 03:12:22.415657 acceptance/chaos_test.go:343  monkey starts (seed -5308280896782964865)
I160815 03:12:22.488959 acceptance/chaos_test.go:202  round 1: restarting nodes [0 2 1]
I160815 03:12:22.488978 acceptance/chaos_test.go:213  round 1: restarting 0
E160815 03:12:22.511643 acceptance/cluster/localcluster.go:521  node=0 status=die
I160815 03:12:22.817920 acceptance/chaos_test.go:213  round 1: restarting 2
E160815 03:12:22.819392 acceptance/cluster/localcluster.go:521  node=0 status=restart
E160815 03:12:22.848966 acceptance/cluster/localcluster.go:521  node=2 status=die
I160815 03:12:23.240440 acceptance/chaos_test.go:213  round 1: restarting 1
E160815 03:12:23.241920 acceptance/cluster/localcluster.go:521  node=2 status=restart
E160815 03:12:23.275051 acceptance/cluster/localcluster.go:521  node=1 status=die
I160815 03:12:23.415848 acceptance/chaos_test.go:300  round 1: client counts: (1, 1, 4)
E160815 03:12:23.660951 acceptance/cluster/localcluster.go:521  node=1 status=restart
W160815 03:12:23.662322 acceptance/chaos_test.go:244  round 1: monkey sleeping while cluster recovers...
I160815 03:12:26.416275 acceptance/chaos_test.go:300  round 1: client counts: (9, 275, 12)
W160815 03:12:26.769379 acceptance/chaos_test.go:252  round 1: cluster recovered
I160815 03:12:26.795449 acceptance/chaos_test.go:202  round 2: restarting nodes [1 2 0]
I160815 03:12:26.795471 acceptance/chaos_test.go:213  round 2: restarting 1
E160815 03:12:26.826121 acceptance/cluster/localcluster.go:521  node=1 status=die
I160815 03:12:27.140248 acceptance/chaos_test.go:213  round 2: restarting 2
E160815 03:12:27.146099 acceptance/cluster/localcluster.go:521  node=1 status=restart
E160815 03:12:27.182360 acceptance/cluster/localcluster.go:521  node=2 status=die
I160815 03:12:27.416473 acceptance/chaos_test.go:300  test finished, waiting for shutdown of 3 clients
W160815 03:12:27.578464 acceptance/chaos_test.go:244  round 2: monkey sleeping while cluster recovers...
E160815 03:12:27.581176 acceptance/cluster/localcluster.go:521  node=2 status=restart
I160815 03:12:27.582390 acceptance/chaos_test.go:177  client 2 shutting down
W160815 03:12:27.681958 acceptance/chaos_test.go:252  round 2: cluster recovered
I160815 03:12:33.693510 acceptance/chaos_test.go:177  client 1 shutting down
panic: test timed out after 10m0s

goroutine 655 [running]:
panic(0x12cca20, 0xc420192010)
    /usr/local/go/src/runtime/panic.go:500 +0x1a1
testing.startAlarm.func1()
    /usr/local/go/src/testing/testing.go:918 +0x10b
created by time.goFunc
    /usr/local/go/src/time/sleep.go:154 +0x44

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc420074480, 0x143ac97, 0x13, 0x1557528, 0xc420241c01)
    /usr/local/go/src/testing/testing.go:647 +0x316
testing.RunTests.func1(0xc420074480)
    /usr/local/go/src/testing/testing.go:793 +0x6d
testing.tRunner(0xc420074480, 0xc42003fda8)
    /usr/local/go/src/testing/testing.go:610 +0x81
testing.RunTests(0x15598a0, 0x1ca8920, 0x22, 0x22, 0xc420241df0)
    /usr/local/go/src/testing/testing.go:799 +0x2f5
testing.(*M).Run(0xc42003fef8, 0x1557650)
    /usr/local/go/src/testing/testing.go:743 +0x85
github.com/cockroachdb/cockroach/acceptance.runTests(0xc42003fef8)
    /go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:137 +0x48
github.com/cockroachdb/cockroach/acceptance.TestMain(0xc42003fef8)
    /go/src/github.com/cockroachdb/cockroach/acceptance/main_test.go:38 +0x45
main.main()
    github.com/cockroachdb/cockroach/acceptance/_test/_testmain.go:118 +0xc6

goroutine 17 [syscall, 10 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x1fed840)
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1016 +0x77
created by github.com/cockroachdb/cockroach/util/log.init.1
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:599 +0x95

goroutine 19 [select, 10 minutes, locked to thread]:
runtime.gopark(0x155a168, 0x0, 0x142e5ae, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc42002b730, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc42002b730)
    /usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 11 [syscall, 10 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 12 [chan receive, 10 minutes]:
github.com/cockroachdb/cockroach/acceptance.runTests.func1()
    /go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:128 +0xb6
created by github.com/cockroachdb/cockroach/acceptance.runTests
    /go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:136 +0x3a

goroutine 481 [runnable]:
github.com/cockroachdb/cockroach/acceptance.waitClientsStop(0x3, 0xc4204c17a0, 0x1bf08eb000)
    /go/src/github.com/cockroachdb/cockroach/acceptance/chaos_test.go:265 +0x289
github.com/cockroachdb/cockroach/acceptance.testClusterRecoveryInner(0xc420074240, 0x1cc9a40, 0xc4200ca0e0, 0xc420017d10, 0x9, 0xc420117240, 0x1, 0x1, 0x12a05f200)
    /go/src/github.com/cockroachdb/cockroach/acceptance/chaos_test.go:349 +0x5d6
github.com/cockroachdb/cockroach/acceptance.runTestOnConfigs.func1(0xc420074240, 0xc4204b7ed8, 0x1557680)
    /go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:305 +0xd0
github.com/cockroachdb/cockroach/acceptance.runTestOnConfigs(0xc420074240, 0x1557680)
    /go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:306 +0xe2
github.com/cockroachdb/cockroach/acceptance.TestClusterRecovery(0xc420074240)
    /go/src/github.com/cockroachdb/cockroach/acceptance/chaos_test.go:312 +0x37
testing.tRunner(0xc420074240, 0x1557528)
    /usr/local/go/src/testing/testing.go:610 +0x81
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:646 +0x2ec

goroutine 401 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc4202e8300)
    /usr/local/go/src/net/http/transport.go:1632 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1049 +0x50e

goroutine 558 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc4201a1ce0)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 16 [IO wait, 10 minutes]:
net.runtime_pollWait(0x7fb0e915f490, 0x72, 0x4)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4203a5e90, 0x72, 0xc42003d9d0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4203a5e90, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4203a5e30, 0xc4203ce000, 0x1000, 0x1000, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002ca30, 0xc4203ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc42038af00, 0xc4203ce000, 0x1000, 0x1000, 0x30, 0xc42003db58, 0x5fb00c)
    /usr/local/go/src/net/http/transport.go:1247 +0x154
bufio.(*Reader).fill(0xc4203be5a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4203be5a0, 0x1, 0x0, 0x1, 0x0, 0xc4204f06c0, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc42038af00)
    /usr/local/go/src/net/http/transport.go:1404 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1048 +0x4e9

goroutine 18 [select, 10 minutes]:
net/http.(*persistConn).writeLoop(0xc42038af00)
    /usr/local/go/src/net/http/transport.go:1632 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1049 +0x50e

goroutine 506 [select, 9 minutes]:
net/http.(*persistConn).readLoop(0xc42000b400)
    /usr/local/go/src/net/http/transport.go:1527 +0x9c9
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1048 +0x4e9

goroutine 302 [IO wait, 10 minutes]:
net.runtime_pollWait(0x7fb0e915f0d0, 0x72, 0x7)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4204e2370, 0x72, 0xc4205459d0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4204e2370, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4204e2310, 0xc42026a000, 0x1000, 0x1000, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c070, 0xc42026a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc4202e8100, 0xc42026a000, 0x1000, 0x1000, 0x30, 0xc420545b58, 0x5fb00c)
    /usr/local/go/src/net/http/transport.go:1247 +0x154
bufio.(*Reader).fill(0xc4204c16e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4204c16e0, 0x1, 0x0, 0x1, 0x0, 0xc4204f09c0, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc4202e8100)
    /usr/local/go/src/net/http/transport.go:1404 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1048 +0x4e9

goroutine 303 [select, 10 minutes]:
net/http.(*persistConn).writeLoop(0xc4202e8100)
    /usr/local/go/src/net/http/transport.go:1632 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1049 +0x50e

goroutine 191 [select, 10 minutes]:
net/http.(*persistConn).writeLoop(0xc4202e8000)
    /usr/local/go/src/net/http/transport.go:1632 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1049 +0x50e

goroutine 554 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc4201a1a20)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 400 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7fb0e915f250, 0x72, 0x8)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4204e23e0, 0x72, 0xc4205409d0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4204e23e0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4204e2380, 0xc420268000, 0x1000, 0x1000, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c080, 0xc420268000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc4202e8300, 0xc420268000, 0x1000, 0x1000, 0x30, 0xc420540b58, 0x5fb00c)
    /usr/local/go/src/net/http/transport.go:1247 +0x154
bufio.(*Reader).fill(0xc4204f1260)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4204f1260, 0x1, 0x0, 0x1, 0x0, 0xc4204f0960, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc4202e8300)
    /usr/local/go/src/net/http/transport.go:1404 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1048 +0x4e9

goroutine 190 [IO wait, 10 minutes]:
net.runtime_pollWait(0x7fb0e915f310, 0x72, 0x6)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420330760, 0x72, 0xc42003b9d0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420330760, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc420330700, 0xc4203aa000, 0x1000, 0x1000, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c058, 0xc4203aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc4202e8000, 0xc4203aa000, 0x1000, 0x1000, 0x30, 0xc42003bb58, 0x5fb00c)
    /usr/local/go/src/net/http/transport.go:1247 +0x154
bufio.(*Reader).fill(0xc4204f14a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4204f14a0, 0x1, 0x0, 0x1, 0x0, 0xc4204c0ae0, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc4202e8000)
    /usr/local/go/src/net/http/transport.go:1404 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1048 +0x4e9

goroutine 485 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc4202e8400)
    /usr/local/go/src/net/http/transport.go:1632 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1049 +0x50e

goroutine 508 [select, 9 minutes]:
github.com/docker/engine-api/client/transport/cancellable.Do.func3(0x7fb0e91667b0, 0xc4204cba00, 0xc4203f2240, 0xc42029a9c0)
    /go/src/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:79 +0x111
created by github.com/docker/engine-api/client/transport/cancellable.Do
    /go/src/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:85 +0x2c3

goroutine 484 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7fb0e915f190, 0x72, 0x9)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4204e2450, 0x72, 0xc4205429d0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4204e2450, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4204e23f0, 0xc4204ec000, 0x1000, 0x1000, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c090, 0xc4204ec000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc4202e8400, 0xc4204ec000, 0x1000, 0x1000, 0x88ef00, 0xc420542b58, 0x5c3d7d)
    /usr/local/go/src/net/http/transport.go:1247 +0x154
bufio.(*Reader).fill(0xc4204f19e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4204f19e0, 0x1, 0xc420542bbd, 0x1, 0x0, 0xc4201c3bc0, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc4202e8400)
    /usr/local/go/src/net/http/transport.go:1404 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1048 +0x4e9

goroutine 551 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7fb0e915ec50, 0x72, 0x10)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420331b10, 0x72, 0xc42051f438, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420331b10, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc420331ab0, 0xc4204fec00, 0x400, 0x400, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c980, 0xc4204fec00, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc4202a2720, 0x7fb0e9166c00, 0xc42002c980, 0x5, 0xc42002c980, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:472 +0x91
crypto/tls.(*Conn).readRecord(0xc420348a80, 0x155a217, 0xc420348b80, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:574 +0xc4
crypto/tls.(*Conn).Read(0xc420348a80, 0xc42032b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1109 +0x116
bufio.(*Reader).fill(0xc4200132c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc4200132c0, 0xc42022e020, 0x5, 0x200, 0x2, 0x1556eb0, 0x0)
    /usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1cb1700, 0xc4200132c0, 0xc42022e020, 0x5, 0x200, 0x5, 0xc42023b220, 0x20, 0x18)
    /usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1cb1700, 0xc4200132c0, 0xc42022e020, 0x5, 0x200, 0xc4203d8000, 0xc42001e0a0, 0x7fb0e91ae000)
    /usr/local/go/src/io/io.go:325 +0x58
github.com/cockroachdb/pq.(*conn).recvMessage(0xc42022e000, 0xc42023b220, 0x12d0c00, 0x201, 0x2026878)
    /go/src/github.com/cockroachdb/pq/conn.go:953 +0x117
github.com/cockroachdb/pq.(*conn).recv1Buf(0xc42022e000, 0xc42023b220, 0x200)
    /go/src/github.com/cockroachdb/pq/conn.go:1003 +0x39
github.com/cockroachdb/pq.(*conn).recv1(0xc42022e000, 0xa0737a, 0xc420348a80)
    /go/src/github.com/cockroachdb/pq/conn.go:1024 +0x87
github.com/cockroachdb/pq.(*conn).readBindResponse(0xc42022e000)
    /go/src/github.com/cockroachdb/pq/conn.go:1771 +0x2f
github.com/cockroachdb/pq.(*stmt).exec(0xc4203a2320, 0xc42050b350, 0x3, 0x3)
    /go/src/github.com/cockroachdb/pq/conn.go:1414 +0xacf
github.com/cockroachdb/pq.(*stmt).Exec(0xc4203a2320, 0xc42050b350, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/pq/conn.go:1368 +0x11a
github.com/cockroachdb/pq.(*conn).Exec(0xc42022e000, 0x146ab0a, 0xd0, 0xc42050b350, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/pq/conn.go:884 +0x2c7
database/sql.(*DB).exec(0xc420072c60, 0x146ab0a, 0xd0, 0xc42051fe88, 0x3, 0x3, 0xc420193001, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/database/sql/sql.go:1036 +0x4ca
database/sql.(*DB).Exec(0xc420072c60, 0x146ab0a, 0xd0, 0xc42051fe88, 0x3, 0x3, 0x61eed3, 0x57b1331a, 0x2efe9e13, 0xc420521e68)
    /usr/local/go/src/database/sql/sql.go:1010 +0xa2
github.com/cockroachdb/cockroach/acceptance.transferMoney(0xc4201ad080, 0x3e7, 0x3e7, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/acceptance/chaos_test.go:137 +0x235
github.com/cockroachdb/cockroach/acceptance.transferMoneyLoop(0x0, 0xc4204c17a0, 0x3e7, 0x3e7)
    /go/src/github.com/cockroachdb/cockroach/acceptance/chaos_test.go:165 +0xa5
created by github.com/cockroachdb/cockroach/acceptance.testClusterRecoveryInner
    /go/src/github.com/cockroachdb/cockroach/acceptance/chaos_test.go:334 +0x3a8

goroutine 507 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc42000b400)
    /usr/local/go/src/net/http/transport.go:1632 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1049 +0x50e

goroutine 502 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7fb0e915f3d0, 0x72, 0xa)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc42028cfb0, 0x72, 0xc4204b5870, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc42028cfb0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc42028cf50, 0xc420170000, 0x1000, 0x1000, 0x0, 0x1cb85c0, 0xc420016068)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc42002c0d0, 0xc420170000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc42000b400, 0xc420170000, 0x1000, 0x1000, 0x0, 0x1559cd8, 0xa523c0)
    /usr/local/go/src/net/http/transport.go:1247 +0x154
bufio.(*Reader).fill(0xc42029a7e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).ReadSlice(0xc42029a7e0, 0xc4204b5a0a, 0x614c90, 0xc420404ea0, 0x4, 0xc4204b5a88, 0x5c45de)
    /usr/local/go/src/bufio/bufio.go:330 +0xb5
net/http/internal.readChunkLine(0xc42029a7e0, 0xc4204f0d80, 0xc4204bc230, 0xc4204b5ca8, 0xc4204b5b00, 0x715d5c)
    /usr/local/go/src/net/http/internal/chunked.go:110 +0x34
net/http/internal.(*chunkedReader).beginChunk(0xc4203958f0)
    /usr/local/go/src/net/http/internal/chunked.go:47 +0x32
net/http/internal.(*chunkedReader).Read(0xc4203958f0, 0xc4202fe001, 0x5ff, 0x5ff, 0x614a80, 0xc4204b5bc8, 0xc4204b5bd0)
    /usr/local/go/src/net/http/internal/chunked.go:77 +0x7b
net/http.(*body).readLocked(0xc4204cbc00, 0xc4202fe001, 0x5ff, 0x5ff, 0xc42001e0f0, 0x2, 0x3)
    /usr/local/go/src/net/http/transfer.go:651 +0x61
net/http.(*body).Read(0xc4204cbc00, 0xc4202fe001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:643 +0xf6
net/http.(*bodyEOFSignal).Read(0xc4204cbc40, 0xc4202fe001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1960 +0x116
github.com/docker/engine-api/client/transport/cancellable.(*notifyingReader).Read(0xc420395920, 0xc4202fe001, 0x5ff, 0x5ff, 0x155409f, 0x155409e, 0xc41fff2953)
    /go/src/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:100 +0x5b
encoding/json.(*Decoder).refill(0xc420177520, 0xa, 0x9)
    /usr/local/go/src/encoding/json/stream.go:152 +0xfa
encoding/json.(*Decoder).readValue(0xc420177520, 0x0, 0x0, 0x80)
    /usr/local/go/src/encoding/json/stream.go:128 +0x198
encoding/json.(*Decoder).Decode(0xc420177520, 0x129f600, 0xc4201ad500, 0xc4204cbf40, 0x40)
    /usr/local/go/src/encoding/json/stream.go:57 +0x8e
github.com/cockroachdb/cockroach/acceptance/cluster.(*LocalCluster).monitor.func1(0x7fb000000000)
    /go/src/github.com/cockroachdb/cockroach/acceptance/cluster/localcluster.go:570 +0x23f
github.com/cockroachdb/cockroach/acceptance/cluster.(*LocalCluster).monitor(0xc4200ca0e0)
    /go/src/github.com/cockroachdb/cockroach/acceptance/cluster/localcluster.go:592 +0x61
created by github.com/cockroachdb/cockroach/acceptance/cluster.(*LocalCluster).Start
    /go/src/github.com/cockroachdb/cockroach/acceptance/cluster/localcluster.go:616 +0x57e

goroutine 550 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc4201a16b0)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 547 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc42043fa20)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 585 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc420072f20)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 573 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc420072c60)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 597 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc4200731e0)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 630 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc42043e580)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 619 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc42043fb80)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added this to the Q2 milestone Aug 15, 2016
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 15, 2016
@jordanlewis
Copy link
Member

This test seems to time out with some frequency.

This was referenced Aug 15, 2016
Closed
Closed
@jordanlewis
Copy link
Member

This is preventing master merges: e.g. https://teamcity.cockroachdb.com/viewLog.html?buildId=11351&tab=buildResultsDiv&buildTypeId=Cockroach_MergeToMaster

This was referenced Aug 15, 2016
Closed
Closed
@d4l3k
Copy link
Contributor

d4l3k commented Aug 15, 2016

Also failed on https://teamcity.cockroachdb.com/viewLog.html?buildId=11500&buildTypeId=Cockroach_UnitTests

@petermattis
Copy link
Collaborator

I thought this was worked around by #8430. As indicated on #8353, I believe the real fix is grpc/grpc-go#818 which was just merged. Note that these failures were easy to reproduce locally by running the acceptance test with the -count flag.

@jordanlewis jordanlewis mentioned this issue Aug 15, 2016
Closed
jordanlewis added a commit to jordanlewis/cockroach that referenced this issue Aug 16, 2016
@jordanlewis
Update to latest grpc.
4778d45 
This should hopefully fix cockroachdb#8538.
This was referenced Aug 16, 2016
Merged
Closed
Closed
@tamird
Copy link
Contributor

tamird commented Sep 1, 2016

Closing for now, since expected fixes from grpc are in.

@tamird tamird closed this as completed Sep 1, 2016
@andreimatei
Copy link
Contributor

reopening as TestClusterRecovery is still skipped. @jordanlewis can we unskip?

@andreimatei andreimatei reopened this Nov 3, 2016
@andreimatei andreimatei changed the title teamcity: failed tests: acceptance/TestClusterRecovery acceptance: TestClusterRecovery is skipped Nov 3, 2016
@jordanlewis
Copy link
Member

Sure, although I'm hesitant to try to unskip this until #10409 is resolved as it will certainly cause this test to flake as well.

@jordanlewis jordanlewis mentioned this issue Nov 7, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jordanlewis jordanlewis

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
Q2
Development

Successfully merging a pull request may close this issue.

Revert "acceptance: skip flaky TestClusterRecovery." jordanlewis/cockroach
6 participants
@jordanlewis @andreimatei @d4l3k @tamird @petermattis @cockroach-teamcity