roachtest: tpccbench/nodes=9/cpu=4/multi-region failed

[cockroach-teamcity]

(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on provisional_202008151325_v19.2.10@13350e2bebf99997730262bce26b843ed3ecb1be:

		Wraps: (4) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2641
		Wraps: (5) 1 safe detail enclosed
		Wraps: (6) monitor task failed
		Wraps: (7) attached stack trace
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:878
		  | [...repeated from below...]
		Wraps: (8) 1 safe detail enclosed
		Wraps: (9) error running tpcc load generator
		Wraps: (10) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:875
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2575
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (11) 2 safe details enclosed
		Wraps: (12) output in run_021815.048_n4_workload_run_tpcc
		Wraps: (13) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2185654-1597595624-83-n12cpu4-geo:4 -- ./workload run tpcc --warehouses=5000 --active-warehouses=1000 --tolerate-errors --scatter --ramp=3m0s --duration=15m0s --partitions=3 --partition-affinity=0 {pgurl:1-3} --histograms=perf/warehouses=1000/stats.json returned
		  | stderr:
		  | .485101 1 workload/cli/run.go:379  Attempt to create load generator failed. It's been more than 10m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: preparing 
		  | 		UPDATE district
		  | 		SET d_next_o_id = d_next_o_id + 1
		  | 		WHERE d_w_id = $1 AND d_id = $2
		  | 		RETURNING d_tax, d_next_o_id: context deadline exceeded
		  | Error: failed to initialize the load generator: preparing 
		  | 		UPDATE district
		  | 		SET d_next_o_id = d_next_o_id + 1
		  | 		WHERE d_w_id = $1 AND d_id = $2
		  | 		RETURNING d_tax, d_next_o_id: context deadline exceeded
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload run tpcc --warehouses=5000 --active-warehouses=1000 --tolerate-errors --scatter --ramp=3m0s --duration=15m0s --partitions=3 --partition-affinity=0 {pgurl:1-3} --histograms=perf/warehouses=1000/stats.json
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Initializing 2000 connections...
		  | Initializing 10000 workers and preparing statements...
		Wraps: (14) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *withstack.withStack (5) *safedetails.withSafeDetails (6) *errutil.withMessage (7) *withstack.withStack (8) *safedetails.withSafeDetails (9) *errutil.withMessage (10) *withstack.withStack (11) *safedetails.withSafeDetails (12) *errutil.withMessage (13) *main.withCommandDetails (14) *exec.ExitError

More

Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #52337 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202008031850_v20.2.0-alpha.3 release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #52206 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #51160 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202007071743_v20.2.0-alpha.2 release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #50698 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #50518 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202006220937_v19.2.8 release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #50142 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202006091546_v20.1.2 release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #48255 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-master release-blocker

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #46343 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[irfansharif]

Same as #52206. We don't know what this is yet.