roachtest: import/tpcc/warehouses=4000/geo failed #89616

cockroach-teamcity · 2022-10-09T11:26:37Z

roachtest.import/tpcc/warehouses=4000/geo failed with artifacts on release-22.2 @ fbe8fde57907a776a9ac6f1cfe0c2596b36982bc:

		  | I221009 11:05:40.932235 65 ccl/workloadccl/fixture.go:481  [-] 10  imported 121 GiB in stock table (400000000 rows, 0 index entries, took 53m32.620453915s, 38.43 MiB/s)
		  | I221009 11:05:42.281347 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 274 GiB bytes in 9 tables (took 53m34.117442342s, 87.30 MiB/s)
		  | I221009 11:07:05.073201 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I221009 11:07:06.330696 1 workload/tpcc/tpcc.go:517  [-] 13  check 3.3.2.1 took 1.257402262s
		  | I221009 11:07:51.843767 1 workload/tpcc/tpcc.go:517  [-] 14  check 3.3.2.2 took 45.513003753s
		  | I221009 11:07:59.645981 1 workload/tpcc/tpcc.go:517  [-] 15  check 3.3.2.3 took 7.802145933s
		  | I221009 11:13:12.804341 1 workload/tpcc/tpcc.go:517  [-] 16  check 3.3.2.4 took 5m13.158253015s
		  | I221009 11:13:33.630041 1 workload/tpcc/tpcc.go:517  [-] 17  check 3.3.2.5 took 20.825613536s
		  | Error: check failed: 3.3.2.5: left EXCEPT right returned nonzero results.
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import tpcc --warehouses=4000 --csv-server='http://localhost:8081'
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import.go:154,import.go:181,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:181
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpcc/warehouses=4000/geo failed #89480 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-master release-blocker]

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-20337}

The text was updated successfully, but these errors were encountered:

stevendanna · 2022-10-11T08:45:22Z

Since #89482 wasn't backported to 22.2, I don't think @irfansharif's theory about #89480 applies here.

A search of previous tickets shows we've seen this error | Error: check failed: 3.3.2.5: left EXCEPT right returned nonzero results. as the result of a number of different issues before.

stevendanna · 2022-10-11T14:15:32Z

Hopefully this might make this more illuminating if it happens again: #89748

stevendanna · 2022-10-11T14:24:50Z

Around the time of the failure, we do see this in the crdb logs:

teamcity-6848277-1665292867-52-n8cpu16-geo-0003> W221009 11:13:32.739249 3850 2@rpc/clock_offset.go:223 ⋮ [n3,rnode=5,raddr=‹10.146.15.220:26257›,class=default,heartbeat] 2171  latency jump (prev avg 266.41ms, current 547.29ms)
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.519316 2275 kv/kvserver/raft_transport.go:602 ⋮ [n5] 4323  while processing outgoing Raft queue to node 3: ‹rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout›:
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324  ‹[core]›‹[Channel #64 SubChannel #65] grpc: addrConn.createTransport failed to connect to {›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹  "Addr": "10.164.15.218:26257",›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹  "ServerName": "10.164.15.218:26257",›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹  "Attributes": null,›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹  "BalancerAttributes": null,›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹  "Type": 0,›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹  "Metadata": null›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609816 133651 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 4324 +‹}. Err: connection error: desc = "transport: Error while dialing cannot reuse client connection"›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609918 133622 kv/kvserver/raft_transport.go:597 ⋮ [n5] 4325  creating batch client for node 3 failed: ‹rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing cannot reuse client connection"›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609918 133622 kv/kvserver/raft_transport.go:597 ⋮ [n5] 4325 +(1) ‹rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing cannot reuse client connection"›
teamcity-6848277-1665292867-52-n8cpu16-geo-0005> W221009 11:13:33.609918 133622 kv/kvserver/raft_transport.go:597 ⋮ [n5] 4325 +Error types: (1) *status.Error

stevendanna · 2022-10-11T14:33:15Z

I'm going to remove the release blocker on this for now.

We've seen similar transient failures in other tests (#84799), but unfortunately here we don't have a tsdump to see if it matches a similar pattern.

cockroach-teamcity · 2022-10-15T11:33:04Z

roachtest.import/tpcc/warehouses=4000/geo failed with artifacts on release-22.2 @ cffe9bc440988894abe9a598ea6b2f15e1b7df93:

		  | I221015 11:03:50.719356 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 274 GiB bytes in 9 tables (took 43m57.703702243s, 106.37 MiB/s)
		  | I221015 11:05:49.499516 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I221015 11:05:50.617498 1 workload/tpcc/tpcc.go:517  [-] 13  check 3.3.2.1 took 1.117910329s
		  | I221015 11:06:24.661757 1 workload/tpcc/tpcc.go:517  [-] 14  check 3.3.2.2 took 34.044203866s
		  | I221015 11:06:30.924114 1 workload/tpcc/tpcc.go:517  [-] 15  check 3.3.2.3 took 6.262278033s
		  | I221015 11:11:24.416575 1 workload/tpcc/tpcc.go:517  [-] 16  check 3.3.2.4 took 4m53.492406486s
		  | I221015 11:16:21.018209 1 workload/tpcc/tpcc.go:517  [-] 17  check 3.3.2.5 took 4m56.601550674s
		  | I221015 11:20:04.339743 1 workload/tpcc/tpcc.go:517  [-] 18  check 3.3.2.7 took 3m43.321446218s
		  | Error: check failed: 3.3.2.7: pq: inbox communication error: rpc error: code = Canceled desc = context canceled
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import tpcc --warehouses=4000 --csv-server='http://localhost:8081'
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import.go:155,import.go:182,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:155
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:182
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpcc/warehouses=4000/geo failed #90026 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-22.1 release-blocker]
roachtest: import/tpcc/warehouses=4000/geo failed #89904 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-22.2.0 release-blocker]
roachtest: import/tpcc/warehouses=4000/geo failed #89480 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-master]

_{This test on roachdash | Improve this report!}

irfansharif · 2022-10-17T14:58:47Z

Most recent failure:

		  | I221015 11:03:50.719356 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 274 GiB bytes in 9 tables (took 43m57.703702243s, 106.37 MiB/s)
		  | I221015 11:05:49.499516 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I221015 11:05:50.617498 1 workload/tpcc/tpcc.go:517  [-] 13  check 3.3.2.1 took 1.117910329s
		  | I221015 11:06:24.661757 1 workload/tpcc/tpcc.go:517  [-] 14  check 3.3.2.2 took 34.044203866s
		  | I221015 11:06:30.924114 1 workload/tpcc/tpcc.go:517  [-] 15  check 3.3.2.3 took 6.262278033s
		  | I221015 11:11:24.416575 1 workload/tpcc/tpcc.go:517  [-] 16  check 3.3.2.4 took 4m53.492406486s
		  | I221015 11:16:21.018209 1 workload/tpcc/tpcc.go:517  [-] 17  check 3.3.2.5 took 4m56.601550674s
		  | I221015 11:20:04.339743 1 workload/tpcc/tpcc.go:517  [-] 18  check 3.3.2.7 took 3m43.321446218s
		  | Error: check failed: 3.3.2.7: pq: inbox communication error: rpc error: code = Canceled desc = context canceled

cockroach-teamcity · 2022-10-18T11:41:17Z

roachtest.import/tpcc/warehouses=4000/geo failed with artifacts on release-22.2 @ a3ac132c4210f3f1202adedb344df222726e8e1b:

		  | I221018 11:09:31.527249 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I221018 11:09:33.378840 1 workload/tpcc/tpcc.go:517  [-] 13  check 3.3.2.1 took 1.851497861s
		  | I221018 11:10:11.600092 1 workload/tpcc/tpcc.go:517  [-] 14  check 3.3.2.2 took 38.22119439s
		  | I221018 11:10:18.868282 1 workload/tpcc/tpcc.go:517  [-] 15  check 3.3.2.3 took 7.268137493s
		  | I221018 11:14:42.872623 1 workload/tpcc/tpcc.go:517  [-] 16  check 3.3.2.4 took 4m24.004265011s
		  | I221018 11:16:40.175145 1 workload/tpcc/tpcc.go:517  [-] 17  check 3.3.2.5 took 1m57.302418764s
		  | I221018 11:28:21.216453 1 workload/tpcc/tpcc.go:517  [-] 18  check 3.3.2.7 took 11m41.041233889s
		  | I221018 11:28:32.165760 1 workload/tpcc/tpcc.go:517  [-] 19  check 3.3.2.8 took 10.949221407s
		  | Error: check failed: 3.3.2.8: pq: inbox communication error: rpc error: code = Canceled desc = context canceled
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import tpcc --warehouses=4000 --csv-server='http://localhost:8081'
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import.go:155,import.go:182,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:155
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:182
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpcc/warehouses=4000/geo failed #90026 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-22.1 release-blocker]
roachtest: import/tpcc/warehouses=4000/geo failed #89904 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-sql-queries blocks-22.2.0-beta.5 branch-release-22.2.0 release-blocker]
roachtest: import/tpcc/warehouses=4000/geo failed #89480 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-master]

_{This test on roachdash | Improve this report!}

stevendanna · 2022-10-19T10:35:22Z

On this last failure, we see:

teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330  ‹[core]›‹[Channel #28 SubChannel #29] grpc: addrConn.createTransport failed to connect to {›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹  "Addr": "10.164.0.38:26257",›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹  "ServerName": "10.164.0.38:26257",›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹  "Attributes": null,›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹  "BalancerAttributes": null,›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹  "Type": 0,›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹  "Metadata": null›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366286 113429 google.golang.org/grpc/grpclog/component.go:41 ⋮ [-] 2330 +‹}. Err: connection error: desc = "transport: Error while dialing cannot reuse client connection"›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366352 113428 kv/kvserver/raft_transport.go:597 ⋮ [n5] 2331  creating batch client for node 3 failed: ‹rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing cannot reuse client connection"›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366352 113428 kv/kvserver/raft_transport.go:597 ⋮ [n5] 2331 +(1) ‹rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing cannot reuse client connection"›
teamcity-6994333-1666070351-53-n8cpu16-geo-0005> W221018 11:28:33.366352 113428 kv/kvserver/raft_transport.go:597 ⋮ [n5] 2331 +Error types: (1) *status.Error

in these logs.

cockroach-teamcity · 2022-10-20T11:21:35Z

roachtest.import/tpcc/warehouses=4000/geo failed with artifacts on release-22.2 @ ed1d8f5fc1ca192c32760db860656d1c9133135e:

		  | I221020 11:03:54.422479 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 274 GiB bytes in 9 tables (took 38m32.313583093s, 121.34 MiB/s)
		  | I221020 11:04:13.144540 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I221020 11:04:14.478036 1 workload/tpcc/tpcc.go:517  [-] 13  check 3.3.2.1 took 1.333435175s
		  | I221020 11:04:59.662887 1 workload/tpcc/tpcc.go:517  [-] 14  check 3.3.2.2 took 45.184783352s
		  | I221020 11:05:06.245150 1 workload/tpcc/tpcc.go:517  [-] 15  check 3.3.2.3 took 6.582208419s
		  | I221020 11:10:56.161277 1 workload/tpcc/tpcc.go:517  [-] 16  check 3.3.2.4 took 5m49.916058799s
		  | I221020 11:12:57.212817 1 workload/tpcc/tpcc.go:517  [-] 17  check 3.3.2.5 took 2m1.051449449s
		  | I221020 11:13:17.682145 1 workload/tpcc/tpcc.go:517  [-] 18  check 3.3.2.7 took 20.469236333s
		  | Error: check failed: 3.3.2.7: pq: inbox communication error: rpc error: code = Canceled desc = context canceled
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import tpcc --warehouses=4000 --csv-server='http://localhost:8081'
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import.go:155,import.go:182,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:155
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:182
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpcc/warehouses=4000/geo failed #90026 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-22.1 release-blocker]
roachtest: import/tpcc/warehouses=4000/geo failed #89904 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-sql-queries branch-release-22.2.0]
roachtest: import/tpcc/warehouses=4000/geo failed #89480 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-disaster-recovery T-sql-queries branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity added this to the 22.2 milestone Oct 9, 2022

blathers-crl bot added the T-disaster-recovery label Oct 9, 2022

cockroach-teamcity mentioned this issue Oct 9, 2022

roachtest: import/tpcc/warehouses=4000/geo failed #89480

Closed

celiala added the blocks-22.2.0-beta.4 label Oct 10, 2022

stevendanna removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. blocks-22.2.0-beta.4 labels Oct 11, 2022

exalate-issue-sync bot assigned stevendanna Oct 11, 2022

This was referenced Oct 13, 2022

roachtest: import/tpcc/warehouses=4000/geo failed #89904

Closed

roachtest: import/tpcc/warehouses=4000/geo failed #90026

Closed

stevendanna closed this as completed Mar 14, 2023

yuzefovich mentioned this issue Oct 30, 2023

roachtest: import/tpcc/warehouses=4000/geo failed #113275

Closed

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: import/tpcc/warehouses=4000/geo failed #89616

roachtest: import/tpcc/warehouses=4000/geo failed #89616

cockroach-teamcity commented Oct 9, 2022 •

edited by cockroach-jira-scripts

Loading

stevendanna commented Oct 11, 2022

stevendanna commented Oct 11, 2022

stevendanna commented Oct 11, 2022

stevendanna commented Oct 11, 2022

cockroach-teamcity commented Oct 15, 2022

irfansharif commented Oct 17, 2022

cockroach-teamcity commented Oct 18, 2022

stevendanna commented Oct 19, 2022

cockroach-teamcity commented Oct 20, 2022

roachtest: import/tpcc/warehouses=4000/geo failed #89616

roachtest: import/tpcc/warehouses=4000/geo failed #89616

Comments

cockroach-teamcity commented Oct 9, 2022 • edited by cockroach-jira-scripts Loading

stevendanna commented Oct 11, 2022

stevendanna commented Oct 11, 2022

stevendanna commented Oct 11, 2022

stevendanna commented Oct 11, 2022

cockroach-teamcity commented Oct 15, 2022

irfansharif commented Oct 17, 2022

cockroach-teamcity commented Oct 18, 2022

stevendanna commented Oct 19, 2022

cockroach-teamcity commented Oct 20, 2022

cockroach-teamcity commented Oct 9, 2022 •

edited by cockroach-jira-scripts

Loading