roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44368

cockroach-teamcity · 2020-01-24T20:06:35Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@8fa3e9023664bde8adb108bea59e8d3e41003eb3:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200124-1708476/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:734: test timed out (10h0m0s)

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44332 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44293 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-2.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44287 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44259 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202001221958_v2.1.11 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-01-25T20:04:10Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@3a68f146d8c59060761b1f525801b1e1b6e22fe8:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200125-1710879/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:734: test timed out (10h0m0s)

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44332 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44293 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-2.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44287 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44259 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202001221958_v2.1.11 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-29T20:19:11Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@359e1940c013729337a23221c341177739d8cccd:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200129-1716020/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:734: test timed out (10h0m0s)

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44332 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-03T20:13:58Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@4c97816a16e185e36a8d269e8842e6cbfefb9985:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200203-1723795/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:749: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:734: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:734
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-04T20:15:28Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@4ac7dc34369b15382b4505ebb65032e9d8717322:

		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		        main.execCmd
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		        main.(*cluster).StartE
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1859
		        main.(*Chaos).Runner.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/chaos.go:121
		        main.(*monitor).Go.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		        github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		        runtime.goexit
		        	/usr/local/go/src/runtime/asm_amd64.s:1357
		      - error with embedded safe details: %s returned:
		        stderr:
		        %s
		        stdout:
		        %s
		        -- arg 1: <string>
		        -- arg 2: <string>
		        -- arg 3: <string>
		      - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1726356-1580802041-26-n10cpu4:1 returned:
		        stderr:
		        
		        stdout:
		        teamcity-1726356-1580802041-26-n10cpu4: starting..
		        teamcity-1726356-1580802041-26-n10cpu4: initializing cluster settings:
		      - context canceled
		    could not restart node :1
		    main.(*Chaos).Runner.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/chaos.go:122
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-05T20:19:36Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@4654024871028124660244c6ffcead27b5bba6c0:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T20:22:31Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@eb883734bcdafd85d98eb0e49126749bc2cc1284:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-08T20:14:01Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@567530c82055309dab055bfa623511e735b0d376:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-10T20:33:20Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@805c31a86e013b577bcb2ba8b33d7b06962a80b6:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-20T20:09:58Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@40ae1c279e2a2b20e8e1acfee1b5b0b87d04dbed:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		        main.execCmd
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		        main.(*cluster).StartE
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1859
		        main.(*Chaos).Runner.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/chaos.go:121
		        main.(*monitor).Go.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		        github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		        runtime.goexit
		        	/usr/local/go/src/runtime/asm_amd64.s:1357
		      - error with embedded safe details: %s returned:
		        stderr:
		        %s
		        stdout:
		        %s
		        -- arg 1: <string>
		        -- arg 2: <string>
		        -- arg 3: <string>
		      - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1757502-1582184184-25-n10cpu4:4 returned:
		        stderr:
		        
		        stdout:
		        teamcity-1757502-1582184184-25-n10cpu4: starting.:
		      - context canceled
		    could not restart node :4
		    main.(*Chaos).Runner.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/chaos.go:122
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-22T20:21:00Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@1b0fbbad3433d12de86a7cd809f8f884976abc2e:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-26T20:23:00Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@c772d373de7ecab68e412341a413e808f2dd0638:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-27T20:12:30Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@b2e9ce2c11961909d8aa62cbace78419536e9b8c:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45470 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-03T20:09:30Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@a7db18b266e6e0f965b74350930f0c31f042b200:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200303-1780235/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45470 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-03-03T23:58:49Z

They're all timeout errors above (test timed out (10h0m0s)).

irfansharif · 2020-03-04T00:30:46Z

One of the workload generators is stuck looping as such.

run_121152.863_n10_workload_run_tpcc: 12:11:52 cluster.go:335: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1780235-1583220814-25-n10cpu4:10 -- ./workload run tpcc --warehouses=2000 --active-warehouses=1050 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 {pgurl:10} --histograms=perf/warehouses=1050/stats.json
Initializing 2100 connections...
Initializing 10500 workers and preparing statements...
I200303 12:12:58.078466 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		UPDATE district
		SET d_next_o_id = d_next_o_id + 1
		WHERE d_w_id = $1 AND d_id = $2
		RETURNING d_tax, d_next_o_id: EOF
Initializing 2100 connections...
Initializing 10500 workers and preparing statements...
I200303 12:14:01.518132 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		UPDATE district
		SET d_next_o_id = d_next_o_id + 1
		WHERE d_w_id = $1 AND d_id = $2
		RETURNING d_tax, d_next_o_id: EOF
[...]
Initializing 2100 connections...
Initializing 10500 workers and preparing statements...
I200303 12:18:13.237013 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		UPDATE district
		SET d_next_o_id = d_next_o_id + 1
		WHERE d_w_id = $1 AND d_id = $2
		RETURNING d_tax, d_next_o_id: EOF
Initializing 2100 connections...
Initializing 10500 workers and preparing statements...
I200303 12:22:40.123554 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		UPDATE district
		SET d_next_o_id = d_next_o_id + 1
		WHERE d_w_id = $1 AND d_id = $2
		RETURNING d_tax, d_next_o_id: EOF

The chaos process waits for the signal that the workload generation process has terminated before ending the chaos loop.

The worker process eventually terminates with the following. Presumably this is the test runner timing out.

Initializing 2100 connections...
Initializing 10500 workers and preparing statements...
I200303 20:04:12.358703 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		UPDATE district
		SET d_next_o_id = d_next_o_id + 1
		WHERE d_w_id = $1 AND d_id = $2
		RETURNING d_tax, d_next_o_id: EOF
Initializing 2100 connections...
run_121152.863_n10_workload_run_tpcc: 20:08:11 cluster.go:2001: > result: error with attached stack trace:
    main.execCmd
    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
    main.(*cluster).RunL
    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
    main.(*cluster).RunE
    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
    main.runTPCCBench.func3.1
    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:815
    main.(*monitor).Go.func1
    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
    runtime.goexit
    	/usr/local/go/src/runtime/asm_amd64.s:1357
  - error with embedded safe details: %s returned:
    stderr:
    %s
    stdout:
    %s
    -- arg 1: <string>
    -- arg 2: <string>
    -- arg 3: <string>
  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1780235-1583220814-25-n10cpu4:10 -- ./workload run tpcc --warehouses=2000 --active-warehouses=1050 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 {pgurl:10} --histograms=perf/warehouses=1050/stats.json returned:
    stderr:
    trict
    		SET d_next_o_id = d_next_o_id + 1
    		WHERE d_w_id = $1 AND d_id = $2
    		RETURNING d_tax, d_next_o_id: EOF
    I200303 20:04:12.358703 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
[...]
    		UPDATE district
    		SET d_next_o_id = d_next_o_id + 1
    		WHERE d_w_id = $1 AND d_id = $2
    		RETURNING d_tax, d_next_o_id: EOF
    I200303 20:07:22.916677 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
    		UPDATE district
    		SET d_next_o_id = d_next_o_id + 1
    		WHERE d_w_id = $1 AND d_id = $2
    		RETURNING d_tax, d_next_o_id: EOF
    
    stdout:
    ...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...
    Initializing 10500 workers and preparing statements...
    Initializing 2100 connections...:
  - context canceled
run_121152.863_n10_workload_run_tpcc: 20:08:11 cluster.go:2003: (note: incoming context was canceled: context canceled

cockroach-teamcity · 2020-03-04T20:08:05Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@48c23df882e3caa12597409b9582d5d7801ef627:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200304-1783236/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:756: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-12T19:21:03Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@826d92a030ff06ed3ef46050470ba92b4cb998cb:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200312-1804482/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:779: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:747: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45820 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-13T19:17:44Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@cddd03b095799dc35c84ea9efd4f17b4e015011e:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200313-1806479/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:785: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45820 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-28T19:14:15Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@9ec209d560b7489451d9107585fd381a559ecd7c:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1834648-1585377863-26-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		8: 22152
		5: 21238
		2: 21680
		3: 17766
		4: 21709
		7: 22298
		6: 19997
		9: 23502
		1: dead
		Error: UNCLASSIFIED_PROBLEM:
		  - 1: dead
		    main.glob..func13
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		    main.wrap.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		    main.main
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:203
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-01T21:09:42Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@63fcb9fa24525c41bc36f1d5dd873828c99a96fe:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200401-1841499/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-03T21:09:33Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@a7a80a8f061e318324cc6167d86697a5e1acb785:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200403-1848273/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-04T20:09:43Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@ee759892738f7f203ff95ec7627b90d7c47b4350:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200404-1851713/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-09T21:09:42Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@7ff60aac0a394a8d43befc9a0f4e89f911aa1294:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200409-1862663/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2366
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2422
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47185 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-14T21:10:22Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@f6f008ff5e2c189051fc826c38ae6f9f678796a0:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200414-1874510/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: monitor failure: monitor task failed: context canceled
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2406
		  | main.runTPCCBench.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2462
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47471 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-16T21:12:50Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@67b32428bef0d6965cb3c03cf0519461092eeed9:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:858,tpcc.go:570,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2366
		    main.runTPCCBench.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:835
		    github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		    github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:746
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2422
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - context canceled

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-19T20:09:32Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@7e24195e28229cfa352227fd4b8b9357c4bea277:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:860,tpcc.go:572,test_runner.go:753: monitor failure: monitor task failed: context canceled
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2426
		  | main.runTPCCBench.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:837
		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2482
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-21T20:13:22Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@3b556da231bb5ac15ac6ff678d61cb1ba516e6f7:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:800: test timed out (10h0m0s)

	tpcc.go:860,tpcc.go:572,test_runner.go:753: monitor failure: monitor task failed: context canceled
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2426
		  | main.runTPCCBench.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:837
		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2482
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-03T19:00:24Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@8085a7edbb294a78b6a7550f05522ec3088754ad:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:837
		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1916055-1588487054-28-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		8: 18395
		2: 14547
		5: dead
		6: dead
		1: 24647
		4: dead
		9: dead
		7: dead
		3: dead
		Error: UNCLASSIFIED_PROBLEM: 5: dead, 6: dead, 4: dead, 9: dead, 7: dead, 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) 5: dead, 6: dead, 4: dead, 9: dead, 7: dead, 3: dead
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Error types: (1) errors.Unclassified (2) *errors.fundamental

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-05T19:37:54Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@6490e49bae3807e8bf823dc3686dc9ab28f05dfa:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:837
		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1920718-1588661390-28-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		3: 21535
		5: 19671
		8: 23557
		6: 23174
		2: 22949
		4: 22685
		1: dead
		9: 23323
		7: 20694
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) 1: dead
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Error types: (1) errors.Unclassified (2) *errors.fundamental

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-11T19:28:50Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@6b3603510631daad6f6d09208b41ebe20dd0f2d7:

		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1934043-1589179638-28-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		8: 21336
		4: 23573
		1: 22393
		3: 22037
		9: 20339
		7: 19983
		5: dead
		2: dead
		6: dead
		Error: UNCLASSIFIED_PROBLEM: 5: dead, 2: dead, 6: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 5: dead, 2: dead, 6: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #48636 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #48635 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202004061746_v19.2.6 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-22T19:21:17Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@0ff02629058261d04b19a77e6159cc534582d225:

		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1960621-1590129586-28-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		8: 23066
		5: 21022
		3: 18893
		2: 21515
		6: 22353
		9: dead
		4: 18533
		1: dead
		7: dead
		Error: UNCLASSIFIED_PROBLEM: 9: dead, 1: dead, 7: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 9: dead, 1: dead, 7: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #49167 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-23T19:16:14Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@b3ccc2171c393e7c43015d090a8dacdbec499e8c:

		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1962586-1590216092-29-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		6: 20221
		2: 21361
		9: 21672
		1: 25976
		3: 21161
		4: 17515
		5: 22442
		8: dead
		7: dead
		Error: UNCLASSIFIED_PROBLEM: 8: dead, 7: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 8: dead, 7: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #49167 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-26T20:14:00Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@91ba6cde389547b13b8fcbdce9eb1ac373b2cd9a:

		  | github.com/cockroachdb/cockroach/pkg/util/search.searchWithSearcher
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:43
		  | github.com/cockroachdb/cockroach/pkg/util/search.(*lineSearcher).Search
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/search/search.go:173
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:748
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:572
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) monitor task failed
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1967613-1590478346-28-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		6: 21800
		9: 19980
		2: 23337
		3: 19752
		5: 20176
		8: 20779
		4: dead
		7: dead
		1: dead
		Error: UNCLASSIFIED_PROBLEM: 4: dead, 7: dead, 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 4: dead, 7: dead, 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #49167 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

knz · 2020-06-04T13:31:45Z

see #45722 (comment)

cockroach-teamcity added branch-release-19.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 24, 2020

cockroach-teamcity added this to the 20.1 milestone Jan 24, 2020

cockroach-teamcity assigned andreimatei Jan 24, 2020

cockroach-teamcity mentioned this issue Jan 25, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44332

Closed

cockroach-teamcity mentioned this issue Feb 5, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44748

Closed

cockroach-teamcity mentioned this issue Feb 26, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45470

Closed

irfansharif assigned irfansharif and unassigned andreimatei Mar 3, 2020

This was referenced Mar 4, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45722

Closed

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #45820

Closed

cockroach-teamcity mentioned this issue Apr 8, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47185

Closed

irfansharif added branch-release-20.1 and removed branch-release-20.1 labels Apr 9, 2020

This was referenced Apr 10, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47317

Closed

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47471

Closed

irfansharif removed their assignment Apr 15, 2020

cockroach-teamcity mentioned this issue Apr 30, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #48209

Closed

This was referenced May 9, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #48635

Closed

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #48636

Closed

cockroach-teamcity mentioned this issue May 17, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #49167

Closed

knz closed this as completed Jun 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44368

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44368

cockroach-teamcity commented Jan 24, 2020

cockroach-teamcity commented Jan 25, 2020

cockroach-teamcity commented Jan 29, 2020

cockroach-teamcity commented Feb 3, 2020

cockroach-teamcity commented Feb 4, 2020

cockroach-teamcity commented Feb 5, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 8, 2020

cockroach-teamcity commented Feb 10, 2020

cockroach-teamcity commented Feb 20, 2020

cockroach-teamcity commented Feb 22, 2020

cockroach-teamcity commented Feb 26, 2020

cockroach-teamcity commented Feb 27, 2020

cockroach-teamcity commented Mar 3, 2020

irfansharif commented Mar 3, 2020 •

edited

Loading

irfansharif commented Mar 4, 2020 •

edited

Loading

cockroach-teamcity commented Mar 4, 2020

cockroach-teamcity commented Mar 12, 2020

cockroach-teamcity commented Mar 13, 2020

cockroach-teamcity commented Mar 28, 2020

cockroach-teamcity commented Apr 1, 2020

cockroach-teamcity commented Apr 3, 2020

cockroach-teamcity commented Apr 4, 2020

cockroach-teamcity commented Apr 9, 2020

cockroach-teamcity commented Apr 14, 2020

cockroach-teamcity commented Apr 16, 2020

cockroach-teamcity commented Apr 19, 2020

cockroach-teamcity commented Apr 21, 2020

cockroach-teamcity commented May 3, 2020

cockroach-teamcity commented May 5, 2020

cockroach-teamcity commented May 11, 2020

cockroach-teamcity commented May 22, 2020

cockroach-teamcity commented May 23, 2020

cockroach-teamcity commented May 26, 2020

knz commented Jun 4, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44368

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44368

Comments

cockroach-teamcity commented Jan 24, 2020

cockroach-teamcity commented Jan 25, 2020

cockroach-teamcity commented Jan 29, 2020

cockroach-teamcity commented Feb 3, 2020

cockroach-teamcity commented Feb 4, 2020

cockroach-teamcity commented Feb 5, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 8, 2020

cockroach-teamcity commented Feb 10, 2020

cockroach-teamcity commented Feb 20, 2020

cockroach-teamcity commented Feb 22, 2020

cockroach-teamcity commented Feb 26, 2020

cockroach-teamcity commented Feb 27, 2020

cockroach-teamcity commented Mar 3, 2020

irfansharif commented Mar 3, 2020 • edited Loading

irfansharif commented Mar 4, 2020 • edited Loading

cockroach-teamcity commented Mar 4, 2020

cockroach-teamcity commented Mar 12, 2020

cockroach-teamcity commented Mar 13, 2020

cockroach-teamcity commented Mar 28, 2020

cockroach-teamcity commented Apr 1, 2020

cockroach-teamcity commented Apr 3, 2020

cockroach-teamcity commented Apr 4, 2020

cockroach-teamcity commented Apr 9, 2020

cockroach-teamcity commented Apr 14, 2020

cockroach-teamcity commented Apr 16, 2020

cockroach-teamcity commented Apr 19, 2020

cockroach-teamcity commented Apr 21, 2020

cockroach-teamcity commented May 3, 2020

cockroach-teamcity commented May 5, 2020

cockroach-teamcity commented May 11, 2020

cockroach-teamcity commented May 22, 2020

cockroach-teamcity commented May 23, 2020

cockroach-teamcity commented May 26, 2020

knz commented Jun 4, 2020

irfansharif commented Mar 3, 2020 •

edited

Loading

irfansharif commented Mar 4, 2020 •

edited

Loading