roachtest: clearrange/checks=true failed

[cockroach-teamcity]

(roachtest).clearrange/checks=true failed on provisional_202005041945_v19.1.9@d556976a57c52e188157469ec9a64d8f388a79e9:

The test failed on branch=provisional_202005041945_v19.1.9, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=true/run_1
	cluster.go:2012,clearrange.go:47,clearrange.go:33,test_runner.go:753: output in run_223458.176_n1_cockroach_workload_fixtures_import_bank: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1920090-1588628153-28-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: exit status 30
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2087
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		  | main.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:47
		  | main.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:33
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_223458.176_n1_cockroach_workload_fixtures_import_bank
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1920090-1588628153-28-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I200504 22:34:58.963642 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		  | I200504 23:39:39.040259 38 ccl/workloadccl/fixture.go:516  imported bank (1h4m40s, 0 rows, 0 index entries, 0 B)
		  | Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: DEAD_ROACH_PROBLEM: exit status 1
		  | (1) DEAD_ROACH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cockroach (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 30
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: /clearrange/checks=true
Related:

• roachtest: clearrange/checks=true failed #44845 roachtest: clearrange/checks=true failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[petermattis]

Node 6 seems to have just disappeared. @jlinder Can you see anything in the test output that explains what happened to node 6.

[jlinder]

I don't see anything in the logs explaining why node 6 disappeared. In the past, it was at least sometimes due to OOMs (https://github.com/cockroachdb/dev-inf/issues/79) which could be the case here.

I've also seen other times when the logs don't get populated the way they are expected to be (missing the *.dmesg.txt, *.journalctl.txt and debug.zip files) and have opened a new issue to track fixing that problem (#48444).

[itsbilal]

Artifacts don't exist anymore. I'll close this issue out, as many bugs in this area were fixed recently.