roachtest: import/tpcc/warehouses=4000/geo failed

[cockroach-teamcity]

roachtest.import/tpcc/warehouses=4000/geo failed with artifacts on release-22.2 @ 9b62adaceb821a96d325a8ce30f35952ec48e9e5:

		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) output in run_103452.976750600_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) ./cockroach workload fixtures import tpcc --warehouses=4000 --csv-server='http://localhost:8081' returned
		  | stderr:
		  | I220912 10:34:55.114251 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 9 tables
		  | I220912 10:35:02.163833 85 ccl/workloadccl/fixture.go:481  [-] 2  imported 7.9 MiB in item table (100000 rows, 0 index entries, took 5.329441765s, 1.48 MiB/s)
		  | I220912 10:35:02.300062 31 ccl/workloadccl/fixture.go:481  [-] 3  imported 213 KiB in warehouse table (4000 rows, 0 index entries, took 5.465835173s, 0.04 MiB/s)
		  | I220912 10:35:05.158497 32 ccl/workloadccl/fixture.go:481  [-] 4  imported 3.9 MiB in district table (40000 rows, 0 index entries, took 8.324297383s, 0.47 MiB/s)
		  | I220912 10:35:44.504185 84 ccl/workloadccl/fixture.go:481  [-] 5  imported 546 MiB in new_order table (36000000 rows, 0 index entries, took 47.669828412s, 11.46 MiB/s)
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import tpcc --warehouses=4000 --csv-server='http://localhost:8081'
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,import.go:154,import.go:181,test_runner.go:908: monitor failure: monitor task failed: read tcp 172.17.0.3:53088 -> 34.89.27.230:26257: read: connection reset by peer
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCC.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:181
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) read tcp 172.17.0.3:53088 -> 34.89.27.230:26257
		Wraps: (6) read
		Wraps: (7) connection reset by peer
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: import/tpcc/warehouses=4000/geo failed (job session ID missing) #85310 roachtest: import/tpcc/warehouses=4000/geo failed [raft sideload oom] [A-bulkio A-kv C-test-failure O-roachtest O-robot T-bulkio T-kv-replication branch-master release-blocker]
• roachtest: import/tpcc/warehouses=4000/geo failed #81430 roachtest: import/tpcc/warehouses=4000/geo failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]
• roachtest: import/tpcc/warehouses=4000/geo failed [raft sideload oom] #76824 roachtest: import/tpcc/warehouses=4000/geo failed [raft sideload oom] [C-test-failure O-roachtest O-robot S-3 T-kv-replication X-nostale branch-release-21.2 no-test-failure-activity]

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

Jira issue: CRDB-19548

[stevendanna]

> rg 'oom' 1.dmesg.txt 
577:[ 1998.529588] cockroach invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0

[stevendanna]

Node 1 was OOM killed. The most recent heap profile look similar to what we've seen in this test before and is currently being tracked by KV in #73376