Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: ssh_problem failed #94819

Closed
cockroach-teamcity opened this issue Jan 6, 2023 · 26 comments
Closed

roachtest: ssh_problem failed #94819

cockroach-teamcity opened this issue Jan 6, 2023 · 26 comments
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 6, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.ssh_problem failed with artifacts on release-22.2 @ 00ed5143845ec05797d16e6ab61d179cf51775f2:

test sstable-corruption/table failed due to test artifacts and logs in: /artifacts/sstable-corruption/table/run_1
(test_impl.go:286).Fatal: parallel execution failure: unable to set cluster settings: ~ 
		if ! test -e /mnt/data1/cockroach/settings-initialized ; then
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' -e "SET CLUSTER SETTING server.remote_debugging.mode = 'any'" || true;
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' -e "
				SET CLUSTER SETTING cluster.organization = 'Cockroach Labs - Production Testing';
				SET CLUSTER SETTING enterprise.license = 'crl-0-EID/+KQHGAEiI0NvY2tyb2FjaCBMYWJzIC0gUHJvZHVjdGlvbiBUZXN0aW5n';" \
			&& touch /mnt/data1/cockroach/settings-initialized
		fi
: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-23158
@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 6, 2023
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Jan 6, 2023
@blathers-crl blathers-crl bot added the T-testeng TestEng Team label Jan 6, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 6, 2023
Open
@renatolabs renatolabs removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 12, 2023
@renatolabs
Copy link
Contributor

FYI (@smg260): this was marked a release-blocker, but the master equivalent of this issue (#90695) wasn't.

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 95b59e058c3dc01888204f46b6ff5a10962481e5:

test jepsen/register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/parts-start-kill-2/run_1
(test_impl.go:286).Fatal: output in run_135251.124309206_n6_bash: bash -e -c "\
cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
 ~/lein run test \
   --tarball file://${PWD}/cockroach.tgz \
   --username ${USER} \
   --ssh-private-key ~/.ssh/id_rsa \
   --os ubuntu \
   --time-limit 300 \
   --concurrency 30 \
   --recovery-time 25 \
   --test-count 1 \
   -n 10.142.0.111 -n 10.142.1.25 -n 10.142.0.110 -n 10.142.1.20 -n 10.142.0.66 \
   --test register --nemesis parts --nemesis2 start-kill-2 \
> invoke.log 2>&1 \
" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ f15ad13ff5b197f881f333085a2fda170857b50b:

test roachmart/partition=false failed due to test artifacts and logs in: /artifacts/roachmart/partition=false/run_1
(test_impl.go:286).Fatal: parallel execution failure: unable to set cluster settings: ~ 
		if ! test -e /mnt/data1/cockroach/settings-initialized ; then
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' -e "SET CLUSTER SETTING server.remote_debugging.mode = 'any'" || true;
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' -e "
				SET CLUSTER SETTING cluster.organization = 'Cockroach Labs - Production Testing';
				SET CLUSTER SETTING enterprise.license = 'crl-0-EID/+KQHGAEiI0NvY2tyb2FjaCBMYWJzIC0gUHJvZHVjdGlvbiBUZXN0aW5n';" \
			&& touch /mnt/data1/cockroach/settings-initialized
		fi
: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 30, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 7cdf538ae6f0ff249d397ab64dbf144a543c5d1b:

test tpcc/mixed-headroom/n5cpu16 failed due to test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1
(test_impl.go:286).Fatal: cluster.StopE: parallel execution failure: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 2, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 235a8b930aae21212e3ae0598c929ff338f3d7e6:

test autoupgrade failed due to test artifacts and logs in: /artifacts/autoupgrade/run_1
(cluster.go:1904).Start: parallel execution failure: failed to upload start script: : SSH_PROBLEM: ssh verbose log retained in ssh_070536.494771837_n1_cat-cockroachsh-chmo.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 86279b015967e35804ba03036259e77999981d74:

test tpchvec/streamer failed due to test artifacts and logs in: /artifacts/tpchvec/streamer/run_1
(tpchvec.go:462).baseTestRun: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-9262947-1679894368-26-n3cpu4:1:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_071227.494213616_n1_get-internal-ip.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 28, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 783f52ec813a67d27da4d98a9893ade76c33122b:

test tpch_concurrency failed due to test artifacts and logs in: /artifacts/tpch_concurrency/run_1
(cluster.go:1943).Stop: cluster.StopE: parallel execution failure: SSH_PROBLEM: ssh verbose log retained in ssh_063854.521031737_n1_node-stop.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ c8b87c4e0e0e77e30e14732705d88b865aac1549:

test tpcc/multiregion/survive=zone/chaos=true failed due to test artifacts and logs in: /artifacts/tpcc/multiregion/survive=zone/chaos=true/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_150139.085303046_n10_cockroach-workload-r: ./cockroach workload run tpcc --warehouses=60 --histograms=perf/workload_6.stats.json --method=simple --wait=false --tolerate-errors --survival-goal=zone --regions=us-east1,us-west1,europe-west2 --partitions=3 --ramp=5m0s --duration=1h30m0s --prometheus-port=2116 --pprofport=33339 --partition-affinity=2 {pgurl:7} returned: parallel execution failure: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-9441310-1680672012-112-n10cpu4-geo:10:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_151523.650158868_n10_get-internal-ip.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 4e32b92ff60b40ec5d9037c4fbd4844ce3428973:

test tpchvec/perf failed due to test artifacts and logs in: /artifacts/tpchvec/perf/run_1
(tpchvec.go:462).baseTestRun: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-9482223-1680844594-24-n3cpu4:3:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_071115.173657808_n3_get-internal-ip.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 14, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test zfs/ycsb/A/nodes=3/cpu=8 failed due to test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=8/run_1
(cluster.go:1980).Run: output in run_155159.447284367_n4_workload-run-ycsb-in: ./workload run ycsb --init --insert-count=1000000 --workload=A --concurrency=96 --splits=3 --histograms=perf/stats.json --select-for-update=true --ramp=2m --duration=30m {pgurl:1-3} returned: parallel execution failure: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-9761114-1682313384-120-n4cpu8:3:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_155159.453868088_n3_get-internal-ip.log: exit status 255
(monitor.go:127).Wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=zfs , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test tpcc/mixed-headroom/multiple-upgrades/n5cpu16 failed due to test artifacts and logs in: /artifacts/tpcc/mixed-headroom/multiple-upgrades/n5cpu16/run_1
(cluster.go:1980).Run: output in run_155159.241390287_n5_cockroach-workload-c: ./cockroach workload check tpcc --warehouses=909 {pgurl:1} returned: parallel execution failure: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-9761114-1682313384-113-n5cpu16:1:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_155159.247793820_n1_get-internal-ip.log: exit status 255
(mixed_version_jobs.go:74).wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test tpcc/multiregion/survive=zone/chaos=true failed due to test artifacts and logs in: /artifacts/tpcc/multiregion/survive=zone/chaos=true/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: could not stop node :7: cluster.StopE: parallel execution failure: SSH_PROBLEM: ssh verbose log retained in ssh_155500.591508902_n7_node-stop.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test tpcc/multiregion/survive=region/chaos=true failed due to test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
(monitor.go:127).Wait: monitor failure: monitor task failed: could not stop node :1-3: cluster.StopE: parallel execution failure: SSH_PROBLEM: ssh verbose log retained in ssh_155455.565385475_n3_node-stop.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test tpccbench/nodes=6/cpu=16/multi-az failed due to test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1
(cluster.go:1946).Stop: cluster.StopE: parallel execution failure: SSH_PROBLEM: ssh verbose log retained in ssh_160637.224311571_n1_node-stop.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test tpcc/headroom/n4cpu16 failed due to test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1
(cluster.go:1980).Run: output in run_165618.387022955_n4_cockroach-workload-c: ./cockroach workload check tpcc --warehouses=875 {pgurl:1} returned: parallel execution failure: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-9761114-1682313384-112-n4cpu16:4:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_165618.392868035_n4_get-internal-ip.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ ac6bd23042f11dbf7e3d444a2fd7d49b35d73b50:

test tpccbench/nodes=9/cpu=4/multi-region failed due to test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
(cluster.go:1946).Stop: cluster.StopE: parallel execution failure: SSH_PROBLEM: ssh verbose log retained in ssh_171242.460224054_n3_node-stop.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ c3145ebb4d26eed434de3abc70520a7e664c5d74:

test ruby-pg failed due to test artifacts and logs in: /artifacts/ruby-pg/run_1
(ruby_pg.go:93).func1: all attempts failed for install ruby 3.1.2: output in run_134356.703193407_n1_mkdir-p-rubyinstall-: mkdir -p ruby-install && \
        curl -fsSL https://github.com/postmodern/ruby-install/archive/v0.8.3.tar.gz | tar --strip-components=1 -C ruby-install -xz && \
        sudo make -C ruby-install install && \
        sudo ruby-install --system ruby 3.1.2 && \
        sudo gem update --system returned: error persisted after 3 attempts: SSH_PROBLEM: ssh verbose log retained in ssh_134356.713323409_n1_mkdir-p-rubyinstall-.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ f01374276f2302a2cccd3199bd53bb73e973b73a:

test ruby-pg failed due to test artifacts and logs in: /artifacts/ruby-pg/run_1
(ruby_pg.go:93).func1: all attempts failed for install ruby 3.1.2: output in run_134017.608212492_n1_mkdir-p-rubyinstall-: mkdir -p ruby-install && \
        curl -fsSL https://github.com/postmodern/ruby-install/archive/v0.8.3.tar.gz | tar --strip-components=1 -C ruby-install -xz && \
        sudo make -C ruby-install install && \
        sudo ruby-install --system ruby 3.1.2 && \
        sudo gem update --system returned: error persisted after 3 attempts: SSH_PROBLEM: ssh verbose log retained in ssh_134017.617844808_n1_mkdir-p-rubyinstall-.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 1b11ca676513ae073ea696c7cc3123d2684bc4bd:

test ruby-pg failed due to test artifacts and logs in: /artifacts/ruby-pg/run_1
(ruby_pg.go:93).func1: all attempts failed for install ruby 3.1.2: output in run_133100.353814099_n1_mkdir-p-rubyinstall-: mkdir -p ruby-install && \
        curl -fsSL https://github.com/postmodern/ruby-install/archive/v0.8.3.tar.gz | tar --strip-components=1 -C ruby-install -xz && \
        sudo make -C ruby-install install && \
        sudo ruby-install --system ruby 3.1.2 && \
        sudo gem update --system returned: error persisted after 3 attempts: SSH_PROBLEM: ssh verbose log retained in ssh_133100.360440953_n1_mkdir-p-rubyinstall-.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 9623d385bcfd03bd43e6f1f4d178c6ec25e7b47e:

test tpccbench/nodes=9/cpu=4/chaos/partition failed due to test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
(cluster.go:1907).Start: parallel execution failure: ~ ./cockroach.sh
: SSH_PROBLEM: ssh verbose log retained in ssh_094009.616179762_n9_cockroachsh.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 8a2050b00bac7e247df26c63aa0c8c1ca4fd7996:

test follower-reads/survival=zone/locality=regional/reads=bounded-staleness failed due to test artifacts and logs in: /artifacts/follower-reads/survival=zone/locality=regional/reads=bounded-staleness/run_1
(cluster.go:1908).Start: parallel execution failure: failed to upload start script: : SSH_PROBLEM: ssh verbose log retained in ssh_101758.559294329_n6_cat-cockroachsh-chmo.log: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 81b42e67c7f6ed947b29b7676054057aa5f44e49:

test decommission/randomized failed due to test artifacts and logs in: /artifacts/decommission/randomized/run_1
(cluster.go:2100).Start: parallel execution failure: failed to initialize cluster: ~ 
		if ! test -e /mnt/data1/cockroach/cluster-bootstrapped ; then
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach init --url 'postgres://root@localhost:26257?sslmode=disable' && touch /mnt/data1/cockroach/cluster-bootstrapped
		fi
: SSH_PROBLEM: ssh verbose log retained in ssh_091715.997155088_n1_init-cluster.log: exit status 255

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 0119b326fb4e58058cd51c2e660e77343b80bc08:

test schemachange/bulkingest failed due to test artifacts and logs in: /artifacts/schemachange/bulkingest/run_1
(cluster.go:2173).Run: output in run_074155.131538950_n5_cockroach-workload-f: ./cockroach workload fixtures import bulkingest {pgurl:1} --a 100000000 --b 1 --c 1 --payload-bytes 4 --index-b-c-a=false returned: parallel execution failure: parallel execution failure: pghosts: GetInternalIP: failed to execute hostname on teamcity-10858681-1689054201-21-n5cpu4:3:
(stdout) 
(stderr) : SSH_PROBLEM: ssh verbose log retained in ssh_074155.138407912_n3_get-internal-ip.log: exit status 255

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 5d70dc791027cd1b87a0d494591badde0f92f4a6:

test jepsen/bank-multitable/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/parts-start-kill-2/run_1
(jepsen.go:300).runJepsen: output in run_111232.567536778_n6_bash-e-c-cd-mntdata1: bash -e -c "\
cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
 ~/lein run test \
   --tarball file://${PWD}/cockroach.tgz \
   --username ${USER} \
   --ssh-private-key ~/.ssh/id_rsa \
   --os ubuntu \
   --time-limit 300 \
   --concurrency 30 \
   --recovery-time 25 \
   --test-count 1 \
   -n 10.142.1.90 -n 10.142.1.128 -n 10.142.1.129 -n 10.142.15.253 -n 10.142.1.130 \
   --test bank-multitable --nemesis parts --nemesis2 start-kill-2 \
> invoke.log 2>&1 \
" returned: error persisted after 3 attempts: SSH_PROBLEM: ssh verbose log retained in ssh_111232.574418553_n6_bash-e-c-cd-mntdata1.log: exit status 255

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-22.2 @ 24f5ce7a3febcd85248f98360023e9e40309ac66:

test tpccbench/nodes=6/cpu=16/multi-az failed due to test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1
(cluster.go:2100).Start: parallel execution failure: failed to set cluster settings: ~ 
		if ! test -e /mnt/data1/cockroach/settings-initialized ; then
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' -e "SET CLUSTER SETTING server.remote_debugging.mode = 'any'" || true;
			COCKROACH_CONNECT_TIMEOUT=0 ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' -e "
				SET CLUSTER SETTING cluster.organization = 'Cockroach Labs - Production Testing';
				SET CLUSTER SETTING enterprise.license = 'crl-0-EID/+KQHGAEiI0NvY2tyb2FjaCBMYWJzIC0gUHJvZHVjdGlvbiBUZXN0aW5n';" \
			&& touch /mnt/data1/cockroach/settings-initialized
		fi
: SSH_PROBLEM: ssh verbose log retained in ssh_165436.363657509_n1_set-cluster-settings.log: exit status 255

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs renatolabs removed the C-test-failure Broken test (automatically or manually discovered). label Jul 25, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 18, 2023
Open
@srosenberg srosenberg added the X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue label Aug 21, 2023
@srosenberg
Copy link
Member

superseded by #108661

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
None yet
Milestone
22.2
Development

No branches or pull requests
3 participants
@srosenberg @cockroach-teamcity @renatolabs