roachtest: tpcc/mixed-headroom/n5cpu16 failed

[cockroach-teamcity]

roachtest.tpcc/mixed-headroom/n5cpu16 failed with artifacts on release-22.1 @ 12fac19acc2f05a1ec7c60e9a50e0c694c491657:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	monitor.go:127,versionupgrade.go:690,versionupgrade.go:206,tpcc.go:427,test_runner.go:875: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.importLargeBankStep.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:690
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*versionUpgradeTest).run
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:206
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCC.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:427
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 3: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] #74892 roachtest: tpcc/mixed-headroom/n5cpu16 failed [OOM during import while running 21.2] [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.2]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-15924

[tbg]

[ 1985.186448] Out of memory: Killed process 14684 (cockroach) total-vm:18180576kB, anon-rss:10837820kB, file-rss:54896kB, shmem-rss:0kB, UID:1000 pgtables:32736kB oom_score_adj:0

[tbg]

cockroach exited with code 137: Mon Apr 11 13:21:03 UTC 2022
from ~80s before: memprof.2022-04-11T13_19_38.785.5028720464.pprof
https://share.polarsignals.com/794bf63/

Note that this OOM happened on the predecessor, i.e. v21.2, and tsdump was unable to download the timeseries for that reason.

[blathers-crl]

cc @cockroachdb/bulk-io

[dt]

#78957 merged last week which I think knocks out the big piece of this profile, but I don't know if that really explains the OOM; that big piece, while big, was still pretty much fixed in size based on the workload (row size) and gomaxprocs, so in most of these, it was already as big as it was going to get. I suppose you could blame the oom on it, since being so big it meant we had little headroom remaining for something else (SSTs in raft?) to spike, but I don't think the workload generator mem would itself be the oom-causing spike here.

[celiala]

Hi @shermanCRL - with the T-bulkio label removed, is there another team that is still looking into this (to remove the GA-blocker)? thanks!

[shermanCRL]

Hi @celiala, I believe the GA blocking is on KV at this point. Accurate, @tbg?

[tbg]

No, as far as KV is concerned this issue can be closed (which I will do now). The heap profile points at the workload causing the OOM and David says that was fixed. We don't understand everything here but there also isn't anything else to look at.