roachtest: decommission/randomized failed

[cockroach-teamcity]

(roachtest).decommission/randomized failed on release-20.2@4da6dc378d563472d59166167ce760756e76f893:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/randomized/run_1
	test_runner.go:814: test timed out (10m0s)

More

Artifacts: /decommission/randomized

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[knz]

This seems like a legitimate failure, a decommission operation did not complete and caused the test to fail with a timeout.

I am a bit confused by the test artifacts: the test is meaning to decommission two nodes n4 and n5 however I only see log directories named "5" and "6" (unsure if there's a relationship to the node IDs...)

cc @irfansharif @erikgrinaker for triage

[erikgrinaker]

This is probably #56718, which was fixed by #61356 but not backported to 20.2. I'll look into it.

[tbg]

Looks like n2 spontaneously combusted and so the decommission hung because replicas didn't have anywhere to move to

teardown: 09:20:25 cluster.go:1626: ./cockroach debug zip failed: 2: pghosts: GetInternalIP: failed to execute hostname on teamcity-2757410-1615273709-37-n6cpu4:3:: exit status 255