roachtest: acceptance/version-upgrade failed

[cockroach-teamcity]

roachtest.acceptance/version-upgrade failed with artifacts on release-21.1 @ 6859359afee920aadae63cda5fd4efd543f0b70b:

		  |   |   echo ". ${HOME}/.profile-cockroach" >> ${HOME}/.profile
		  |   | fi
		  |   |
		  |   | # We run this script (with arg "run") as a service unit. We do not use --user
		  |   | # because memory limiting doesn't work in that mode. Instead we pass the uid and
		  |   | # gid that the process will run under.
		  |   | # The "notify" service type means that systemd-run waits until cockroach
		  |   | # notifies systemd that it is ready; NotifyAccess=all is needed because this
		  |   | # notification doesn't come from the main PID (which is bash).
		  |   | sudo systemd-run --unit cockroach \
		  |   |   --same-dir --uid $(id -u) --gid $(id -g) \
		  |   |   --service-type=notify -p NotifyAccess=all \
		  |   |   -p MemoryMax=95% \
		  |   |   -p LimitCORE=infinity \
		  |   |   -p LimitNOFILE=65536 \
		  |   | 	bash $0 run
		  |   | EOF
		  |   | 
		  | Wraps: (3) exit status 255
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I210830 06:09:50.380261 1 (gostd) cluster_synced.go:1758  [-] 1  command failed
		Wraps: (2) exit status 1
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

	cluster.go:1667,context.go:89,cluster.go:1656,test_runner.go:820: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3380142-1630303664-15-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 10270
		1: 10436
		3: 10273
		2: dead (exit status 0)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 0)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1889
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 2: dead (exit status 0)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh acceptance/version-upgrade

Same failure on other branches

• roachtest: acceptance/version-upgrade failed [ambiguous result] #65701 roachtest: acceptance/version-upgrade failed [ambiguous result] [C-test-failure O-roachtest O-robot T-kv T-sql-queries branch-master]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

[tbg]

Looks odd, n2 didn't start, but it never got to the point where it was even writing anywhere. The start step exited 255.
Looks like some kind of fluke, perhaps in the ssh connection.