roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed

[cockroach-teamcity]

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ 12a0fdf76785787a3a7e83198f1adfd7184ea910:

(monitor.go:153).Wait: monitor failure: unexpected node event: n3: cockroach process died (exit code 137)
test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-31597

[rhu713]

Looks like an oom from MaybeInlineSideloadedRaftCommand like #73376

[erikgrinaker]

@pavelkalinnikov I thought we mitigated this by increasing the AWS disk bandwidth and memory, to correspond better to GCP nodes? Did we not cover this test?

[pav-kv]

We did cover this test in #109221 (backport #109278). However, this failure is still possible until we properly fix #73376.

[blathers-crl]

cc @cockroachdb/replication

[rhu713]

@pavelkalinnikov do you think we can close this issue as a duplicate of #73376 or does KV want to take a closer look at this failure?

[pav-kv]

@rhu713 Nah, I think it looks exactly like a duplicate. See #106496 (comment) too (and #106496 (comment) which predicted what we're seeing). Can close this, or maybe better keep open so that new failures are added here instead of creating new issues.

[pav-kv]

We are closing this, because it's a known issue #73376, and fixing it is on our mid-term roadmap. At the moment, the likelihood of this test failing with an OOM is low, but please let us @cockroachdb/replication know if it get out of control.

#111140 might help reducing the chances of these failures further, once there is a better parity in hardware provisioning for tests on AWS and GCE.