roachtest: restore2TB/nodes=32 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/4810e59b1d9724bfbd9e10ae6485ea45aa856fa2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=restore2TB/nodes=32 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=970138&tab=buildLog

The test failed on release-2.1:
	test.go:975: test timed out (3h0m0s)
	test.go:584,cluster.go:1115,restore.go:244,cluster.go:1437,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-970138-restore2tb-nodes-32:1 -- ./cockroach sql --insecure -e "
						RESTORE csv.bank FROM
						'gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=1/bank'
						WITH into_db = 'restore2tb'" returned:
		stderr:
		
		stdout:
		: signal: killed
	test.go:584,cluster.go:1458,restore.go:250: context canceled

[tbg]

@andreimatei could you sleuth this one as well? I recently added this timeout. Perhaps I was too optimistic, but perhaps the test just hung. If it did, expect to see similarities to #31409.

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/2cbfb514fed209e9e4192bd07af6baa8dd073bab

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=restore2TB/nodes=32 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=970864&tab=buildLog

The test failed on master:
	test.go:1002: test timed out (3h0m0s)
	test.go:606,cluster.go:1098,restore.go:244,cluster.go:1420,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-970864-restore2tb-nodes-32:1 -- ./cockroach sql --insecure -e "
						RESTORE csv.bank FROM
						'gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=1/bank'
						WITH into_db = 'restore2tb'" returned:
		stderr:
		
		stdout:
		: signal: killed
	test.go:606,cluster.go:1441,restore.go:250: context canceled

[andreimatei]

closing in the hope that #31570 fixes it