roachtest: import/tpch/nodes=8 failed

[cockroach-teamcity]

roachtest.import/tpch/nodes=8 failed with artifacts on release-21.1 @ c4d0e7baee3925541eed599ae771abb95c97732b:

The test failed on branch=release-21.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/import/tpch/nodes=8/run_1
	cluster.go:1958,import.go:167,test_runner.go:733: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-3290967-1628748601-16-n8cpu4 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach returned: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1973
		  | main.(*cluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1957
		  | main.registerImportTPCH.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/import.go:167
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:733
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) cluster.Put
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-3290967-1628748601-16-n8cpu4 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-3290967-1628748601-16-n8cpu4: putting (dist) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach
		  | .............
		  |    1: done
		  |    2: done
		  |    3: done
		  |    4: done
		  |    5: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine [email protected]:./cockroach [email protected]:./cockroach
		  | [email protected]: Permission denied (publickey).
		  | lost connection: exit status 1
		  |    6: done
		  |    7: done
		  |    8: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine [email protected]:./cockroach [email protected]:./cockroach
		  | [email protected]: Permission denied (publickey).
		  | lost connection: exit status 1
		  | I210812 06:12:26.209415 1 (gostd) cluster_synced.go:1194  [-] 1  put /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 failed
		Wraps: (4) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh import/tpch/nodes=8

Same failure on other branches

• roachtest: import/tpch/nodes=8 failed #68037 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot T-bulkio branch-master]
• roachtest: import/tpch/nodes=8 failed #66263 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker]

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

[adityamaru]

Looks like the test failed to put the cockroach binary on node 5:

[Step 2/3] 	cluster.go:1958,import.go:167,test_runner.go:733: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-3290967-1628748601-16-n8cpu4 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach returned: exit status 1

because of a Permission denied. Writing this off as an infra flake.