roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed

[cockroach-teamcity]

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-20.2@7f140b6fdc0c5086353a56016aafcdaa34ed6405:

		  -- stack trace:
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:834
		  | [...repeated from below...]
		Wraps: (6) error running tpcc load generator
		Wraps: (7) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2290
		  | main.runTPCCBench.func3.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:831
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2632
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (8) output in run_130246.379_n10_workload_run_tpcc
		Wraps: (9) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2528209-1608275695-21-n10cpu4:10 -- ./workload run tpcc --warehouses=2000 --active-warehouses=930 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 --method=simple {pgurl:10} --histograms=perf/warehouses=930/stats.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:23:53.634872 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:26:55.525842 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201218 13:29:57.376064 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": EOF
		  | W201218 13:32:59.349181 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201218 13:36:01.442924 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201218 13:39:03.660147 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:42:05.571720 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE customer SCATTER": driver: bad connection
		  | W201218 13:45:07.582571 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:48:09.563885 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:51:11.634884 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201218 13:54:15.958727 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:56:17.922885 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | W201218 13:59:20.127975 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE stock SCATTER": driver: bad connection
		  | W201218 14:02:22.054656 1 workload/cli/run.go:365  retrying after error while creating load: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE order_line SCATTER": driver: bad connection
		  | E201218 14:05:22.097896 1 workload/cli/run.go:379  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE item SCATTER": EOF
		  | Error: failed to initialize the load generator: could not scatter ranges: Couldn't exec "ALTER TABLE item SCATTER": EOF
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 10. Command with error:
		  |   | ```
		  |   | ./workload run tpcc --warehouses=2000 --active-warehouses=930 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 --method=simple {pgurl:10} --histograms=perf/warehouses=930/stats.json
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (10) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.withPrefix (9) *main.withCommandDetails (10) *exec.ExitError

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

Fixed by #58051.