roachtest: backup/nodeShutdown/coordinator/n4cpu4 failed

[cockroach-teamcity]

roachtest.backup/nodeShutdown/coordinator/n4cpu4 failed with artifacts on release-21.1 @ c4d0e7baee3925541eed599ae771abb95c97732b:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/backup/nodeShutdown/coordinator/n4cpu4/run_1
	jobs.go:131,backup.go:132,test_runner.go:733: unexpectedly found job 683797475003269122 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.jobSurvivesNodeShutdown.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jobs.go:79
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2666
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) unexpectedly found job 683797475003269122 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3290967-1628748601-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: dead (exit status 137)
		4: 10659
		3: 10607
		1: 11767
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 2: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh backup/nodeShutdown/coordinator/n4cpu4

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

Jira issue: CRDB-9262

[adityamaru]

The BACKUP is retrying once with:

W210812 06:14:02.318880 10994 ccl/backupccl/backup_job.go:482 ⋮ [n1,job=‹683797475003269122›] 631  BACKUP job encountered retryable error: exporting 41 ranges: inbox communication error: ‹rpc error: code = Canceled desc = context cancel

but then it appears to fail with:
I210812 06:14:02.372798 10994 jobs/registry.go:1190 ⋮ [n1] 632 BACKUP job ‹683797475003269122›: stepping through state reverting with error: failed to run backup: exporting 40 ranges: unable to dial n2: ‹breaker open›

My suspicion is that we are still planning a spec on node 2 (the node we shutdown) on retry, and when distsql tries to set up a flow it realizes it cannot dial the node. Ideally, SetupAllNodesPlanning should not be returning node 2 once it is shut down but I wonder if its view is not up to date. Either way, I think we can add this error to our set of retryable errors so that we retry again in the hope that we won't plan a flow on the shutdown node.