roachtest: restore/tpce/400GB/aws/nodes=4/cpus=8 failed [CRDB-25503 replication send oom]

[cockroach-teamcity]

roachtest.restore/tpce/400GB/aws/nodes=4/cpus=8 failed with artifacts on release-23.1 @ fa2d7f7c9894d701ac4a393f058aa84552957087:

(test_runner.go:1073).runTest: test timed out (1h0m0s)
(monitor.go:137).Wait: monitor failure: monitor task failed: output in run_080048.232311134_n1_cockroach-sql-insecu: ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures-us-east-2/backups/tpc-e/customers=25000/v22.2.0/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2022-12-21T02:00:00Z' " returned: COMMAND_PROBLEM: exit status 137
test artifacts and logs in: /artifacts/restore/tpce/400GB/aws/nodes=4/cpus=8/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-29460

[cockroach-teamcity]

roachtest.restore/tpce/400GB/aws/nodes=4/cpus=8 failed with artifacts on release-23.1 @ 15480b8496e7928845330055d191e331b228eb66:

(monitor.go:137).Wait: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)
(test_runner.go:1120).func1: 1 dead node(s) detected
(test_runner.go:1120).func1: operation "invalid descriptors check" timed out after 1m0.027s (given timeout 1m0s): pq: query execution canceled
test artifacts and logs in: /artifacts/restore/tpce/400GB/aws/nodes=4/cpus=8/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[dt]

n3 oomed. 96% of memory usage is in raft, just over 11gb.

[dt]

cc @irfansharif

[irfansharif]

+cc @pavelkalinnikov. #106486 is the other AWS roachtest failure.

[blathers-crl]

cc @cockroachdb/replication

[irfansharif]

+cc #73376.

[tbg]

Looking at the test history, this test started running in earnest¹ in end of March '23, and promptly [failed] (edit: that's actually another, similar, test) (https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestFipsNightlyAwsBazel/9365035?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true) with an OOM, on March 31:

Details

So just from that alone we can argue that this is not a release blocker - we introduced a new test, it fails, likely highlighting an existing problem. Mechanistically, we understand why this class of test is likely to fail - we know memory bounding in the replication layer is insufficient and will cause issues when overloading disks, which is likely on AWS due to the relatively slow default gp3 provisioning (combined r+w throughput 125mb/s).

Footnotes

1. introduced a little earlier probably, https://github.com/cockroachdb/cockroach/commit/b588f1f02c4e7bd4a10a08d9b20a914e03751e40 ↩

[tbg]

I need to spend some more time here to validate that though. Over on #106496 I'm seeing that the test already exists on 27fcfd3 which is from Feb 28. Perhaps CI is not retaining history all the way, or we changed the CI config only late in March to actually run these tests. Will look a bit more later.

[tbg]

Looking across branches, the only failures we have for this particular test are three recent ones; in contrast there are 188 successful runs (dating back to late March, before which I think CI has deleted history).

The first failure is s3 object does not exist: external_storage: file doesn't exist and it was triggered on #105969, which was backported on the same day (July 3) as well¹.

We then see failures on July 6 and July 9, followed by (to date) 20 successful runs.
So perhaps changing the bucket altered the speed with which the fixtures are ingested, but then 20 subsequent successes (and a few interspersed ones) don't suggest that this is the case.

If there is a regression, it is not a common one and it will be extremely difficult to pinpoint. On top of this, since this is AWS, we don't have Grafana² which would help a lot with comparing some metrics side by side over time.

Another PR whose timing lines up is #104861. This was merged on July 5, but but not backported. It thus cannot affect the release-23.1 branch, and besides, the same argument about having had many successful runs above applies.

Realistically, we will have to live with these test failures and address them one by one through CRDB-25503.

Handing this back to @pavelkalinnikov to track the eventual resolution through https://cockroachlabs.atlassian.net/browse/CRDB-25503.

Footnotes

1. https://github.com/cockroachdb/cockroach/pull/106073 ↩

2. https://github.com/cockroachdb/cockroach/issues/97788 ↩

[tbg]

https://pprof.me/4ded4ab/ (don't forget to switch to inuse bytes) shows mem in raft transport send queue.

[pav-kv]

The first of the 2 failures is not an OOM, it just times out and gets killed after 1h. Here are some metrics:

It uses 1/2 of the memory at the peak, and stays under 125 MB/s disk throughput.

[pav-kv]

The graphs above look unusual. Most of the time it should operate close to disks throughput capacity. So the timeout is reasonable: the writes were too slow for some reason.

To compare, I ran this test manually, and see something like this T(very similar to ones in the 8TB test):

[pav-kv]

Closing, as this is tracked by #107609.