roachtest: disk-stalled/log=true,data=true failed #58238

cockroach-teamcity · 2020-12-23T08:19:52Z

(roachtest).disk-stalled/log=true,data=true failed on master@cd62a02833eeb02d14944668d09ee54806927a2b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2538659-1608706971-27-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-12-29T08:20:44Z

(roachtest).disk-stalled/log=true,data=true failed on master@365c5504b75c9a9260365a628a5110c48312178b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2548186-1609225512-28-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-31T08:11:22Z

(roachtest).disk-stalled/log=true,data=true failed on master@b93fd531b93cb010729cb73fe679cdff9388cf27:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2552340-1609398198-28-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-07T08:14:22Z

(roachtest).disk-stalled/log=true,data=true failed on master@15765c0fa9118885dda0bd2ad1384b8801c412c3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2564046-1610002658-27-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-09T09:22:49Z

(roachtest).disk-stalled/log=true,data=true failed on master@dbc7245c5d8c9f009072353fec261419e573032c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2569739-1610175644-37-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-11T08:12:44Z

(roachtest).disk-stalled/log=true,data=true failed on master@aa68427e69113a6b9d0aa1dac8469919c6f95e16:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2571220-1610348342-27-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-15T08:22:19Z

(roachtest).disk-stalled/log=true,data=true failed on master@a786c51627fe66e47b4a4445c67b2a9077ae2a93:

		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.13.0/thrift-0.13.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     # TODO(bilal): Change URL back to scylladb/charybdefs once https://github.com/scylladb/charybdefs/pull/21 is merged.
		  |   |     git clone --depth 1 "https://github.com/itsbilal/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-17T08:11:39Z

(roachtest).disk-stalled/log=true,data=true failed on master@7b0ccdda99b81613e70f421c9374483c3feddff3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:760: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2590485-1610867021-28-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-27T08:13:18Z

(roachtest).disk-stalled/log=true,data=true failed on master@f7c5898f3d552f7ab0751cdd9ffa95cdfd6b8a76:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:767: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2618882-1611730714-28-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-01T08:37:12Z

(roachtest).disk-stalled/log=true,data=true failed on master@e4ac407b14ed315662050f09670707cdf8ad9bfe:

		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.13.0/thrift-0.13.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     # TODO(bilal): Change URL back to scylladb/charybdefs once https://github.com/scylladb/charybdefs/pull/21 is merged.
		  |   |     git clone --depth 1 "https://github.com/itsbilal/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-04T08:25:39Z

(roachtest).disk-stalled/log=true,data=true failed on master@100c09f4f6eb3f5b18a67ec4bbfdfe989e0d6ce2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:767: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2642936-1612422004-37-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-05T08:18:13Z

(roachtest).disk-stalled/log=true,data=true failed on master@c584f62067a45aa540c26fc9081a83e460bfe37a:

		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.13.0/thrift-0.13.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     # TODO(bilal): Change URL back to scylladb/charybdefs once https://github.com/scylladb/charybdefs/pull/21 is merged.
		  |   |     git clone --depth 1 "https://github.com/itsbilal/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-24T07:55:45Z

(roachtest).disk-stalled/log=true,data=true failed on master@ec011620c7cf299fdbb898db692b36454defc4a2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:129,disk_stall.go:40,test_runner.go:767: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2712399-1614149800-28-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 Error: COMMAND_PROBLEM: exit status 7
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ```
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ```
		Wraps: (3) exit status 7
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

More

Artifacts: /disk-stalled/log=true,data=true
Related:

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-19T21:40:51Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 4dc05cceea254854317a9374a4b88df06c0946a6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	cluster.go:2312,disk_stall.go:117,disk_stall.go:40,test_runner.go:777: output in run_214025.686_n1_charybdefsnemesis: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2898497-1618858605-38-n1cpu4:1 -- charybdefs-nemesis --delay returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2390
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2310
		  | main.runDiskStalledDetection
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:117
		  | main.registerDiskStalledDetection.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:40
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_214025.686_n1_charybdefsnemesis
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2898497-1618858605-38-n1cpu4:1 -- charybdefs-nemesis --delay returned
		  | stderr:
		  | ./recipes: line 7: python: command not found
		  | Error: COMMAND_PROBLEM: exit status 127
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | charybdefs-nemesis --delay
		  |   | ``````
		  | Wraps: (3) exit status 127
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-21T21:21:52Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 66a1657247f8e1cd84ce22a32321fda6352d789f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	cluster.go:2340,disk_stall.go:117,disk_stall.go:40,test_runner.go:777: output in run_212127.581_n1_charybdefsnemesis: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2907568-1619031816-39-n1cpu4:1 -- charybdefs-nemesis --delay returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2418
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2338
		  | main.runDiskStalledDetection
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:117
		  | main.registerDiskStalledDetection.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:40
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_212127.581_n1_charybdefsnemesis
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2907568-1619031816-39-n1cpu4:1 -- charybdefs-nemesis --delay returned
		  | stderr:
		  | ./recipes: line 7: python: command not found
		  | Error: COMMAND_PROBLEM: exit status 127
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | charybdefs-nemesis --delay
		  |   | ``````
		  | Wraps: (3) exit status 127
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-23T21:30:14Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 9e821aa456bfbe8b5ec3a8ef79ae3954d52cb675:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	cluster.go:2340,disk_stall.go:117,disk_stall.go:40,test_runner.go:777: output in run_212947.688_n1_charybdefsnemesis: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2917180-1619205254-39-n1cpu4:1 -- charybdefs-nemesis --delay returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2418
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2338
		  | main.runDiskStalledDetection
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:117
		  | main.registerDiskStalledDetection.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:40
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_212947.688_n1_charybdefsnemesis
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2917180-1619205254-39-n1cpu4:1 -- charybdefs-nemesis --delay returned
		  | stderr:
		  | ./recipes: line 7: python: command not found
		  | Error: COMMAND_PROBLEM: exit status 127
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | charybdefs-nemesis --delay
		  |   | ``````
		  | Wraps: (3) exit status 127
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-24T21:54:02Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 3cd29547884feb24b150b67ca2a88073798a35fd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	cluster.go:2340,disk_stall.go:117,disk_stall.go:40,test_runner.go:777: output in run_215336.922_n1_charybdefsnemesis: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2920165-1619291008-39-n1cpu4:1 -- charybdefs-nemesis --delay returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2418
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2338
		  | main.runDiskStalledDetection
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:117
		  | main.registerDiskStalledDetection.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:40
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_215336.922_n1_charybdefsnemesis
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2920165-1619291008-39-n1cpu4:1 -- charybdefs-nemesis --delay returned
		  | stderr:
		  | ./recipes: line 7: python: command not found
		  | Error: COMMAND_PROBLEM: exit status 127
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | charybdefs-nemesis --delay
		  |   | ``````
		  | Wraps: (3) exit status 127
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-25T21:12:50Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 2d7f9a08bac49040a6d77c5465e2b4fd8fdac1f4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	cluster.go:2340,disk_stall.go:117,disk_stall.go:40,test_runner.go:777: output in run_211225.529_n1_charybdefsnemesis: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2921792-1619377326-38-n1cpu4:1 -- charybdefs-nemesis --delay returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2418
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2338
		  | main.runDiskStalledDetection
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:117
		  | main.registerDiskStalledDetection.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:40
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_211225.529_n1_charybdefsnemesis
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2921792-1619377326-38-n1cpu4:1 -- charybdefs-nemesis --delay returned
		  | stderr:
		  | ./recipes: line 7: python: command not found
		  | Error: COMMAND_PROBLEM: exit status 127
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | charybdefs-nemesis --delay
		  |   | ``````
		  | Wraps: (3) exit status 127
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-26T21:16:41Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 2dbd33ce319439bc424a52896a03e0a7ccff07f1:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	cluster.go:2340,disk_stall.go:117,disk_stall.go:40,test_runner.go:777: output in run_211615.346_n1_charybdefsnemesis: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2924709-1619463598-39-n1cpu4:1 -- charybdefs-nemesis --delay returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2418
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2338
		  | main.runDiskStalledDetection
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:117
		  | main.registerDiskStalledDetection.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/disk_stall.go:40
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_211615.346_n1_charybdefsnemesis
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2924709-1619463598-39-n1cpu4:1 -- charybdefs-nemesis --delay returned
		  | stderr:
		  | ./recipes: line 7: python: command not found
		  | Error: COMMAND_PROBLEM: exit status 127
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | charybdefs-nemesis --delay
		  |   | ``````
		  | Wraps: (3) exit status 127
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-20T08:35:28Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 7897f24246bef3cb94f9f4bfaed474ecaa9fdee6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3329754-1629441450-38-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 E210820 08:35:16.890931 1 1@cli/clierror/check.go:35  [-] 1  ERROR: fallocate: operation not supported
		ERROR: fallocate: operation not supported
		Failed running "start-single-node"
		Error: COMMAND_PROBLEM: exit status 1
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 1
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^disk-stalled/log=true,data=true$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-21T08:23:21Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 11e0a4da82124e70e772a009011ca7a4007bff85:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3333872-1629526832-39-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 E210821 08:23:05.709863 1 1@cli/clierror/check.go:35  [-] 1  ERROR: fallocate: operation not supported
		ERROR: fallocate: operation not supported
		Failed running "start-single-node"
		Error: COMMAND_PROBLEM: exit status 1
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 1
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^disk-stalled/log=true,data=true$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-22T08:29:57Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ d18da6c092bf1522e7a6478fe3973817e318c247:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3335718-1629614492-36-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 E210822 08:29:46.953873 1 1@cli/clierror/check.go:35  [-] 1  ERROR: fallocate: operation not supported
		ERROR: fallocate: operation not supported
		Failed running "start-single-node"
		Error: COMMAND_PROBLEM: exit status 1
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 1
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^disk-stalled/log=true,data=true$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-23T08:11:53Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 61bd543ba7288c8f0eed6cddded7b219c9d1fcd4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3337780-1629700327-38-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 20 E210823 08:11:41.300521 1 1@cli/clierror/check.go:35  [-] 1  ERROR: fallocate: operation not supported
		ERROR: fallocate: operation not supported
		Failed running "start-single-node"
		Error: COMMAND_PROBLEM: exit status 1
		(1) COMMAND_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 1
		Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^disk-stalled/log=true,data=true$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-24T09:18:40Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 46cef2c6f0b36ba2f7d551c8ab017832c1b9d592:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3344851-1629786541-37-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^disk-stalled/log=true,data=true$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-25T08:45:19Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 967ed00f80981ce8848a5e8144ee6fbd29bc95bb:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3352195-1629873091-37-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-26T08:46:44Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ ab1fc343c9a1140191f96353995258e609a84d02:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3359107-1629959017-38-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-27T07:55:58Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 8cae60f603ccc4d83137167b3b31cab09be9d41a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3366874-1630046181-27-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-28T08:33:43Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 44ea1fa0eba8fc78544700ef4afded62ab98a021:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3373578-1630131756-37-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-29T07:58:11Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 6700cf65e82a029a1ff76a75250c709aa9637170:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3377334-1630217394-40-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-30T08:16:44Z

roachtest.disk-stalled/log=true,data=true failed with artifacts on master @ 0b57dc40deda1206d9a1c215ffdb219bbf182a39:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/disk-stalled/log=true_data=true/run_1
	disk_stall.go:132,disk_stall.go:43,test_runner.go:777: unexpected output: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3380119-1630304219-38-n1cpu4:1 -- timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}': exit status 10 Error: SSH_PROBLEM: exit status 255
		(1) SSH_PROBLEM
		Wraps: (2) Node 1. Command with error:
		  | ``````
		  | timeout --signal 9 600s env COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT=40ms COCKROACH_LOG_MAX_SYNC_DURATION=40ms ./cockroach start-single-node --insecure --store {store-dir}/faulty --log '{sinks: {stderr: {filter: INFO}}, file-defaults: {dir: "{store-dir}/faulty/logs"}}'
		  | ``````
		Wraps: (3) exit status 255
		Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: disk-stalled/log=true,data=true failed #63950 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1]
roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

68929: opt: add AOST to ANALYZE r=mgartner a=mgartner A long running `ANALYZE` can result in a GC TTL error like: batch timestamp 1628283892.315459871,0 must be after replica GC threshold 1628283892.326056946,0 This commit prevents this error by adding `AS OF SYSTEM TIME '-0.001ms'` automatically to all `ANALYZE` statements. This triggers use of inconsistent scans. See the sql.stats.max_timestamp_age setting. Fixes #68590 Release note (bug fix): Long running `ANALYZE` statements will no longer result in GC TTL errors. Release justification: This is a low-risk fix for the `ANALYZE` statement. 69359: roachtest: clarify when to regenerate fixtures r=celiala a=celiala As per noted in #69340 (comment), this PR rewords verMap comment slightly to clarify that fixture generation only needs to happen if a new key is added (e.g. for a new release). Release justification: non-production code change Release note: None 69516: roachtest: disable ballast during disk-stall roachtest r=nicktrav a=jbowens The charybdefs FUSE filesystem used in the disk-stall roachtest slows down file writes. Creating a default 1 GiB emergency ballast via the FUSE filesystem was prohibitively slow. Fix #64240. Fix #58238. Release justification: non-production code changes Release note: none 69566: cli: make unsafe-remove-dead-replicas work with encrypted stores r=sumeerbhola a=sumeerbhola Release justification: Low-risk bug fix to existing functionality. Release note: None 69567: admission: fix overflow bug in ioLoadListener r=sumeerbhola a=sumeerbhola Fixes #69461,#69463 Release justification: High-severity bug fix to new funtionality. Release note: None Co-authored-by: Marcus Gartner <[email protected]> Co-authored-by: Celia La <[email protected]> Co-authored-by: Jackson Owens <[email protected]> Co-authored-by: sumeerbhola <[email protected]>

itsbilal removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 19, 2021

This was referenced Apr 13, 2021

roachtest: disk-stalled/log=true,data=true failed #63579

Closed

roachtest: disk-stalled/log=true,data=true failed #63697

Closed

cockroach-teamcity mentioned this issue Apr 20, 2021

roachtest: disk-stalled/log=true,data=true failed #63950

Closed

jlinder added the T-storage Storage Team label Jun 16, 2021

jbowens mentioned this issue Aug 28, 2021

roachtest: disable ballast during disk-stall roachtest #69516

Merged

craig bot closed this as completed in dcfad38 Aug 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: disk-stalled/log=true,data=true failed #58238

roachtest: disk-stalled/log=true,data=true failed #58238

cockroach-teamcity commented Dec 23, 2020

cockroach-teamcity commented Dec 29, 2020

cockroach-teamcity commented Dec 31, 2020

cockroach-teamcity commented Jan 7, 2021

cockroach-teamcity commented Jan 9, 2021

cockroach-teamcity commented Jan 11, 2021

cockroach-teamcity commented Jan 15, 2021

cockroach-teamcity commented Jan 17, 2021

cockroach-teamcity commented Jan 27, 2021

cockroach-teamcity commented Feb 1, 2021

cockroach-teamcity commented Feb 4, 2021

cockroach-teamcity commented Feb 5, 2021

cockroach-teamcity commented Feb 24, 2021

cockroach-teamcity commented Apr 19, 2021

cockroach-teamcity commented Apr 21, 2021

cockroach-teamcity commented Apr 23, 2021

cockroach-teamcity commented Apr 24, 2021

cockroach-teamcity commented Apr 25, 2021

cockroach-teamcity commented Apr 26, 2021

cockroach-teamcity commented Aug 20, 2021

cockroach-teamcity commented Aug 21, 2021

cockroach-teamcity commented Aug 22, 2021

cockroach-teamcity commented Aug 23, 2021

cockroach-teamcity commented Aug 24, 2021

cockroach-teamcity commented Aug 25, 2021

cockroach-teamcity commented Aug 26, 2021

cockroach-teamcity commented Aug 27, 2021

cockroach-teamcity commented Aug 28, 2021

cockroach-teamcity commented Aug 29, 2021

cockroach-teamcity commented Aug 30, 2021

roachtest: disk-stalled/log=true,data=true failed #58238

roachtest: disk-stalled/log=true,data=true failed #58238

Comments

cockroach-teamcity commented Dec 23, 2020

cockroach-teamcity commented Dec 29, 2020

cockroach-teamcity commented Dec 31, 2020

cockroach-teamcity commented Jan 7, 2021

cockroach-teamcity commented Jan 9, 2021

cockroach-teamcity commented Jan 11, 2021

cockroach-teamcity commented Jan 15, 2021

cockroach-teamcity commented Jan 17, 2021

cockroach-teamcity commented Jan 27, 2021

cockroach-teamcity commented Feb 1, 2021

cockroach-teamcity commented Feb 4, 2021

cockroach-teamcity commented Feb 5, 2021

cockroach-teamcity commented Feb 24, 2021

cockroach-teamcity commented Apr 19, 2021

cockroach-teamcity commented Apr 21, 2021

cockroach-teamcity commented Apr 23, 2021

cockroach-teamcity commented Apr 24, 2021

cockroach-teamcity commented Apr 25, 2021

cockroach-teamcity commented Apr 26, 2021

cockroach-teamcity commented Aug 20, 2021

cockroach-teamcity commented Aug 21, 2021

cockroach-teamcity commented Aug 22, 2021

cockroach-teamcity commented Aug 23, 2021

cockroach-teamcity commented Aug 24, 2021

cockroach-teamcity commented Aug 25, 2021

cockroach-teamcity commented Aug 26, 2021

cockroach-teamcity commented Aug 27, 2021

cockroach-teamcity commented Aug 28, 2021

cockroach-teamcity commented Aug 29, 2021

cockroach-teamcity commented Aug 30, 2021