roachtest: disk-stalled/log=true,data=true failed

[cockroach-teamcity]

roachtest.disk-stalled/log=true,data=true failed with artifacts on ricky-test-upgrade-roachprod @ 6d809e6a4377d4d9952cd0e0ed7b4bf6c9f8b3b7:

		  |   | ``````
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.13.0/thrift-0.13.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     # TODO(bilal): Change URL back to scylladb/charybdefs once https://github.com/scylladb/charybdefs/pull/21 is merged.
		  |   |     git clone --depth 1 "https://github.com/itsbilal/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ``````
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh disk-stalled/log=true,data=true

Same failure on other branches

• roachtest: disk-stalled/log=true,data=true failed #58238 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-master]
• roachtest: disk-stalled/log=true,data=true failed #57601 roachtest: disk-stalled/log=true,data=true failed [C-test-failure O-roachtest O-robot branch-release-20.2]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!