roachtest: schemachange/during/tpcc failed

[cockroach-teamcity]

roachtest.schemachange/during/tpcc failed with artifacts on master @ 771432d1099e516dbc11827c5458886c176e73e3:

		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1962
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1926
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:160
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:176
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCC
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:223
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.makeSchemaChangeDuringTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange.go:431
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_104031.040670813_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) ././cockroach workload fixtures import tpcc --warehouses=1000  returned
		  | stderr:
		  | I220416 10:40:31.909513 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 9 tables
		  | I220416 10:40:35.160106 77 ccl/workloadccl/fixture.go:483  [-] 2  imported 1006 KiB in district table (10000 rows, 0 index entries, took 3.162887721s, 0.31 MiB/s)
		  | I220416 10:40:35.161801 114 ccl/workloadccl/fixture.go:483  [-] 3  imported 7.9 MiB in item table (100000 rows, 0 index entries, took 3.164471859s, 2.49 MiB/s)
		  | I220416 10:40:36.452710 76 ccl/workloadccl/fixture.go:483  [-] 4  imported 53 KiB in warehouse table (1000 rows, 0 index entries, took 4.455563228s, 0.01 MiB/s)
		  | I220416 10:40:43.671891 81 ccl/workloadccl/fixture.go:483  [-] 5  imported 134 MiB in new_order table (9000000 rows, 0 index entries, took 11.674648899s, 11.49 MiB/s)
		  | I220416 10:41:28.727093 79 ccl/workloadccl/fixture.go:483  [-] 6  imported 2.2 GiB in history table (30000000 rows, 0 index entries, took 56.729790121s, 38.88 MiB/s)
		  | I220416 10:41:30.253091 80 ccl/workloadccl/fixture.go:483  [-] 7  imported 1.6 GiB in order table (30000000 rows, 30000000 index entries, took 58.255872154s, 28.36 MiB/s)
		  | I220416 10:42:52.952462 78 ccl/workloadccl/fixture.go:483  [-] 8  imported 17 GiB in customer table (30000000 rows, 30000000 index entries, took 2m20.955251309s, 125.26 MiB/s)
		  | I220416 10:43:59.599320 115 ccl/workloadccl/fixture.go:483  [-] 9  imported 30 GiB in stock table (100000000 rows, 0 index entries, took 3m27.601969968s, 148.55 MiB/s)
		  | I220416 10:44:31.609086 116 ccl/workloadccl/fixture.go:483  [-] 10  imported 17 GiB in order_line table (300011520 rows, 0 index entries, took 3m59.611808457s, 73.06 MiB/s)
		  | I220416 10:44:31.639010 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 68 GiB bytes in 9 tables (took 3m59.729317555s, 292.01 MiB/s)
		  | I220416 10:44:32.883642 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I220416 10:44:32.928472 1 workload/tpcc/tpcc.go:516  [-] 13  check 3.3.2.1 took 44.75893ms
		  | I220416 10:44:47.411561 1 workload/tpcc/tpcc.go:516  [-] 14  check 3.3.2.2 took 14.482988752s
		  | I220416 10:44:49.227739 1 workload/tpcc/tpcc.go:516  [-] 15  check 3.3.2.3 took 1.816126023s
		  | I220416 10:47:10.699386 1 workload/tpcc/tpcc.go:516  [-] 16  check 3.3.2.4 took 2m21.471596378s
		  | I220416 10:47:50.869047 1 workload/tpcc/tpcc.go:516  [-] 17  check 3.3.2.5 took 40.169581639s
		  | I220416 10:51:08.576243 1 workload/tpcc/tpcc.go:516  [-] 18  check 3.3.2.7 took 3m17.70710882s
		  | I220416 10:51:21.544536 1 workload/tpcc/tpcc.go:516  [-] 19  check 3.3.2.8 took 12.96818424s
		  | I220416 10:51:41.789592 1 workload/tpcc/tpcc.go:516  [-] 20  check 3.3.2.9 took 20.244981238s
		  | Error: check failed: 3.3.2.9: pq: inbox communication error: rpc error: code = Canceled desc = context canceled
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ././cockroach workload fixtures import tpcc --warehouses=1000
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-15645

[kvoli]

tracking issue #80072, caused by #76609. Resolved in #80071.