roachtest: sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed

[cockroach-teamcity]

roachtest.sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed with artifacts on master @ 771432d1099e516dbc11827c5458886c176e73e3:

		  | ed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | (last message repeated 1 times)
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | (last message repeated 1 times)
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | (last message repeated 1 times)
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | (last message repeated 1 times)
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | (last message repeated 2 times)
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | (last message repeated 2 times)
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = 'XXUUU': rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
		  | FATAL: PQexecPrepared() failed: 7 result is ambiguous: error=rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout [propagate]
		  | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:409: SQL error, errno = 0, state = '40003': result is ambiguous: error=rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout [propagate]
		  | [ 409s ] thds: 245 tps: 135.99 qps: 1829.89 (r/w/o: 1119.93/527.97/181.99) lat (ms,95%): 20137.61 err/s: 78.00 reconn/s: 0.00
		  | [ 410s ] thds: 245 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
		Wraps: (8) SSH_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | sysbench \
		  | 		--db-driver=pgsql \
		  | 		--pgsql-host=127.0.0.1 \
		  | 		--pgsql-port=26257 \
		  | 		--pgsql-user=root \
		  | 		--pgsql-password= \
		  | 		--pgsql-db=sysbench \
		  | 		--report-interval=1 \
		  | 		--time=600 \
		  | 		--threads=256 \
		  | 		--tables=10 \
		  | 		--table_size=10000000 \
		  | 		--auto_inc=false \
		  | 		oltp_read_write run
		  | ``````
		Wraps: (10) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.SSH (9) *hintdetail.withDetail (10) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-15642

[kvoli]

tracking issue #80072, caused by #76609. Resolved in #80071.