roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [CRDB-25503 replication send oom]

[cockroach-teamcity]

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ f62527e83e458f5f4521497e7bfd5c60b70f1e31:

(monitor.go:137).Wait: monitor failure: monitor command failure: unexpected node event: 8: dead (exit status 137)
(test_runner.go:1120).func1: 1 dead node(s) detected
test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #106247 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-29590

[adityamaru]

n8 OOM'ed with the last profile showing unbounded growth in maybeInlineSideloadedRaftCommand:

#73376 looks like the tracking issue for this class of OOMs.

[blathers-crl]

cc @cockroachdb/replication

[tbg]

This test fails regularly¹:

Details

(CI does not have history before April)

Looking into the investigation here² we saw that we were maxing out the EBS volumes (or got throttled in some way) and ultimately found that this test was running on a weird striped RAID0³ that ultimately gives the cluster inhomogeneous disk performance profiles, which is not something we support well. That issue is still open⁴ but @msbutler merged a targeted fix⁵ on March 30. Still this predates all of the test failures we're looking at above.

@erikgrinaker looked into a failure in April here⁶ saying that we were basically overloading the disks and entering raft overload territory in that way, but in his testing #101437 seemed to address that well enough so that the issue was closed (backport landed on April 18⁷) and we didn't see any failures until July 6, July 10 (and 16 successes since).

My read on this is similar to #106248 (comment). We know replication memory protection is insufficient and we're running these clusters with maxed-out disks, so any additional imbalance can tip them into an unhealthy territory. The temporal correlation between this issue and #106496 is interesting since both failed twice at around the same time and then not again. So there is a good chance that whatever is the issue (perhaps some PR that got merged and then reverted, or AWS infra problems, who knows) has been resolved. While it would be nice to have more confidence, not blocking the release still seems to be the right call.

Handing this back to @pavelkalinnikov to track the eventual resolution through https://cockroachlabs.atlassian.net/browse/CRDB-25503.

Footnotes

1. https://teamcity.cockroachdb.com/test/-8181195248832451825?currentProjectId=Cockroach_Nightlies&expandTestHistoryChartSection=true&orderBy=status&order=desc ↩

2. https://github.com/cockroachdb/cockroach/issues/97019#issuecomment-1441842625 ↩

3. https://github.com/cockroachdb/cockroach/issues/97019#issuecomment-1479401456 ↩

4. https://github.com/cockroachdb/cockroach/issues/98783 ↩

5. https://github.com/cockroachdb/cockroach/pull/100136/files ↩

6. https://github.com/cockroachdb/cockroach/issues/100341#issuecomment-1505496453 ↩

7. https://github.com/cockroachdb/cockroach/pull/101508 ↩

[pav-kv]

At least half of the nodes experience an extended period of being at the write throughput threshold of 125 MB/s (same behaviour as described in #99206):

We should bump the threshold as #107609 suggests.

[pav-kv]

Closing by #108513. The OOM may pop up again (less likely now), until we properly fix #73376.