roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed #125769

cockroach-teamcity · 2024-06-17T06:33:00Z

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ efc894bfc167fe48af2ab99d1e1ecbedf8110a10:

(cluster.go:2400).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/size=64kb/conc=4096/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

/cc @cockroachdb/test-eng _{This test on roachdash | Improve this report!

Jira issue: CRDB-39600}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2024-06-19T06:33:49Z

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 67b4af76ba20ed2f6935da31eda7dfe8fc0b63e2:

(cluster.go:2400).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/size=64kb/conc=4096/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-06-21T08:39:57Z

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 065b94022a26d223b0a19809e510a6dca337f155:

(cluster.go:2400).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/size=64kb/conc=4096/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

_{This test on roachdash | Improve this report!}

renatolabs · 2024-06-25T16:50:59Z

These failures are OOMs of the (single) node in this test. Logs are composed entirely of the following error:

W240621 08:38:24.896240 1807344 kv/kvclient/kvcoord/dist_sender.go:2785 ⋮ [T1,Vsystem,n1,client=10.12.15.200:60062,hostssl,user=‹roachprod›] 335621 slow replica RPC: have been waiting 13.61s (0 attempts) for RPC Put [/Table/106/1/‹-4475947314600438026›/‹0›], EndTxn(parallel commit) [/Table/106/1/‹-4475947314600438026›/‹0›], [txn: bcdcd569], [can-forward-ts] to replica (n1,s1):1; resp: ‹(err: <nil>), *kvpb.PutResponse, *kvpb.EndTxnResponse›

Doesn't seem to reproduce easily on master or the SHA of the last failure, but we have some memory profiles in the artifacts that should help. Tagging @cockroachdb/kv.

renatolabs · 2024-06-26T17:25:46Z

Spoke with @arulajmani in person. Quick update: this test is designed to overload the cluster (see previous failure of this test that has similar discussion: #106570). Another thing that is similar from previous failures is that this seems to only be happening on AWS (and, in this case, only in ARM builds).

I'll try to reproduce this specifically using AWS + ARM64 to see if we have more success with a reproduction. The artifacts alone don't provide enough to debug this further (latest memory profile is taken on average ~10mins before OOM).

arulajmani · 2024-06-26T17:27:11Z

@renatolabs I'll remove the T-KV label for now so that it doesn't show up in our test triage queue. If this ends up needing some KV involvement, feel free to ping the issue again and add the label.

renatolabs · 2024-06-28T14:03:26Z

I'll reopen this one because there was issue being investigated with this test before the flakes.

renatolabs · 2024-06-28T20:21:54Z

Update -- I've confirmed that the OOM is reproducible by running this test on ARM64 on AWS. That said, I also saw failures on GCE, so it's not cloud specific (it didn't fail on GCE previously because it never runs in the nightlies -- #126425). The test doesn't seem to fail when running amd64 binaries.

A bisection points to a Pebble bump (#125541) as being the first commit where this test started to fail. Note that it's not deterministic -- it generally only fails ~20% of the times.

I could debug this further but I'm also juggling a few other items at the moment so I don't know when I'll have time to come back to this. I'm tagging @cockroachdb/storage for further analysis. Let me know if I can assist with anything.

For reference, this is how you run the test on ARM64:

roachtest run --cloud aws --metamorphic-arm64-probability 1.0 --count 10 'kv0/enc=false/nodes=1/size=64kb/conc=4096'

I'm using --count 10 here because, as I said, we typically only see a 20% failure rate.

cockroach-teamcity · 2024-07-03T06:29:54Z

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 7461a85f839c50edbdea502fe09794739a99632e:

(cluster.go:2417).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/size=64kb/conc=4096/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

_{This test on roachdash | Improve this report!}

sumeerbhola · 2024-07-03T15:03:39Z

#125769 (comment) doesn't seem to be an OOM.

(the latest failure) #125769 (comment) is an OOM, but the heap profile is not helpful.
Around the time of the OOM we have:

I240703 06:29:23.943507 376 2@server/status/runtime_log.go:47 ⋮ [T1,Vsystem,n1] 273  runtime stats: 29 GiB RSS, 16560 goroutines (stacks: 948 MiB), 14 GiB/16 GiB Go alloc/total (heap fragmentation: 522 MiB, heap reserved: 593 MiB, heap released: 2.6 GiB), 9.1 GiB/13 GiB CGO alloc/total (11743.2 CGO/sec), 243.8/96.9 %(u/s)time, 0.0 %gc (88x), 185 MiB/1.8 MiB (r/w)net

[Wed Jul  3 06:29:34 2024] memory: usage 30641436kB, limit 30641432kB, failcnt 86758565
...
[Wed Jul  3 06:29:34 2024] Memory cgroup out of memory: Killed process 4165 (cockroach) total-vm:42140324kB, anon-rss:30478136kB, file-rss:146688kB, shmem-rss:0kB, UID:1000 pgtables:75824kB oom_score_adj:0

There is nothing in the logs about what the Pebble block cache was configured to be, but there doesn't seem to be an override in kv.go so I'll assume the usual 25% of memory, so .25 * 29.22GiB = 7.3 GiB.

The CGO alloc of 9.1 GiB is higher: probably due to internal fragmentation caused by these 64KB writes, as discussed in https://docs.google.com/document/d/1Dbku1Z7kB-k8eCn8yEnbMEfdpbY_kVOSOSScw_TsIqc -- we are probably using the 80KiB size class for these ssblocks. This was not fixed by storage: use jemalloc size classes when intializing pebble #125793. But I think it is worth manually running this experiment and adding some logging to verify that that change is not making things worse. @aadityasondhi
The total of 13 GiB is significantly higher. We know some of the latter can be due to fragmentation. But it is probably due to not releasing memory back to the OS fast enough as discussed in https://docs.google.com/document/d/1Dbku1Z7kB-k8eCn8yEnbMEfdpbY_kVOSOSScw_TsIqc/edit#bookmark=id.m4i01kgac1nm. There is a suggestion there to set MALLOC_CONF=background_thread:true,dirty_decay_ms:2000,muzzy_decay_ms:0. We could try that.
Regarding that Pebble bump, it fiddles with readahead but I don't see how that could cause a problem. Admittedly, the concurrency of 4096 could in theory cause 4096 * fileMaxReadaheadSize = 4096 * 256KiB = 1GiB. But these are files where the readahead is done via asking the OS to readahead, which pulls things into the page cache https://man7.org/linux/man-pages/man2/readahead.2.html so it shouldn't cause process OOM. Also, this write workload does a single read for each put, so readahead should also not be active. @RaduBerinde, @jbowens

sumeerbhola · 2024-07-03T15:12:52Z

#125769 (comment) is also an OOM

I240621 08:39:27.556121 316 2@server/status/runtime_log.go:47 â‹® [T1,Vsystem,n1] 347  runtime stats: 28 GiB RSS, 16564 goroutines (stacks: 831 MiB), 6.2 GiB/16 GiB Go alloc/total (heap fragmentation: 1.0 GiB, heap reserved: 7.4 GiB, heap released: 3.6 GiB), 9.1 GiB/13 GiB CGO alloc/total (14820.0 CGO/sec), 229.3/97.1 %(u/s)time, 0.0 %gc (108x), 172 MiB/1.7 MiB (r/w)net

There is a big difference between Go alloc and total in this case.

renatolabs · 2024-07-03T15:14:00Z

#125769 (comment) doesn't seem to be an OOM.

There were a couple of flakes that affected this test. Marked them as off-topic. The remaining failures in this thread should be the OOM.

cockroach-teamcity · 2024-07-06T06:36:28Z

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ c557fb59f6aec659d364e9002fc083c59c6392b6:

(cluster.go:2417).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/size=64kb/conc=4096/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

_{This test on roachdash | Improve this report!}

sumeerbhola · 2024-07-06T19:57:47Z

#125769 (comment) is also an OOM

[Sat Jul 6 06:36:06 2024] Memory cgroup out of memory: Killed process 3857 (cockroach) total-vm:44398208kB, anon-rss:30453076kB, file-rss:147584kB, shmem-rss:0kB, UID:1000 pgtables:79868kB oom_score_adj:0

I240706 06:24:27.899766 423 2@server/status/runtime_log.go:47 â‹® [T1,Vsystem,n1] 210 runtime stats: 27 GiB RSS, 16301 goroutines (stacks: 824 MiB), 10 GiB/16 GiB Go alloc/total (heap fragmentation: 1017 MiB, heap reserved: 4.1 GiB, heap released: 2.3 GiB), 9.1 GiB/11 GiB CGO alloc/total (10103.9 CGO/sec), 286.5/89.7 %(u/s)time, 0.0 %gc (67x), 320 MiB/2.0 MiB (r/w)net

I240706 06:35:58.011119 423 2@server/status/runtime_log.go:47 â‹® [T1,Vsystem,n1] 360 runtime stats: 28 GiB RSS, 16563 goroutines (stacks: 836 MiB), 6.0 GiB/16 GiB Go alloc/total (heap fragmentation: 1.1 GiB, heap reserved: 7.6 GiB, heap released: 3.5 GiB), 9.1 GiB/13 GiB CGO alloc/total (9121.7 CGO/sec), 301.0/101.9 %(u/s)time, 0.0 %gc (107x), 182 MiB/1.7 MiB (r/w)net

… aggressively kv0/enc=false/nodes=1/size=64kb/conc=4096 is sometimes OOMing on arm64 on AWS. It is possible that this is due to higher memory usage in the cgo allocations. This change reduces the difference betweem jemalloc resident and allocated bytes by releasing more aggressively to the OS. Testing this change reduced "9.2 GiB/11 GiB CGO alloc/total" to "9.2 GiB/9.8 GiB CGO alloc/total". Informs cockroachdb#125769 Epic: none Release note: None

126520: logical: add eval logical replication options r=Jeremyyang920 a=Jeremyyang920 This commit adds the ability for the plan hook to be able to evaluate the incoming options from the sql statement. As of right now, the function will only evaluate the options, but nothing will be done with them. If there is an error in the format of an option, the associated error will be returned. See individual commits for details. Epic: None Release note: None 126930: tests: for kv0 overload test, make jemalloc release memory to OS more… r=srosenberg,renatolabs a=sumeerbhola … aggressively kv0/enc=false/nodes=1/size=64kb/conc=4096 is sometimes OOMing on arm64 on AWS. It is possible that this is due to higher memory usage in the cgo allocations. This change reduces the difference betweem jemalloc resident and allocated bytes by releasing more aggressively to the OS. Testing this change reduced "9.2 GiB/11 GiB CGO alloc/total" to "9.2 GiB/9.8 GiB CGO alloc/total". Informs #125769 Epic: none Release note: None 127041: roachtest: fix multi-store-remove r=itsbilal a=nicktrav The intention of the test is to compare the number of ranges (multiplied by the replication factor) to the _sum_ of replicas across all stores. The current implementation is incorrect, as it compares range count to store count. Fix the test by using a `sum` of replicas across each store, rather than a `count`, which will return the number of stores. Fix #123989. Release note: None. Co-authored-by: Jeremy Yang <[email protected]> Co-authored-by: sumeerbhola <[email protected]> Co-authored-by: Nick Travers <[email protected]>

sumeerbhola · 2024-07-12T17:43:43Z

fixed by #126930

… aggressively kv0/enc=false/nodes=1/size=64kb/conc=4096 is sometimes OOMing on arm64 on AWS. It is possible that this is due to higher memory usage in the cgo allocations. This change reduces the difference betweem jemalloc resident and allocated bytes by releasing more aggressively to the OS. Testing this change reduced "9.2 GiB/11 GiB CGO alloc/total" to "9.2 GiB/9.8 GiB CGO alloc/total". Informs #125769 Epic: none Release note: None

renatolabs added the T-kv KV Team label Jun 25, 2024

arulajmani removed the T-kv KV Team label Jun 26, 2024

This comment was marked as off-topic.

Sign in to view

DarrylWong marked this as a duplicate of #126378 Jun 28, 2024

DarrylWong closed this as completed Jun 28, 2024

renatolabs reopened this Jun 28, 2024

renatolabs added the T-storage Storage Team label Jun 28, 2024

blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Jun 28, 2024

github-project-automation bot added this to [Deprecated] Storage Jun 28, 2024

github-project-automation bot moved this to Incoming in [Deprecated] Storage Jun 28, 2024

nicktrav moved this from Incoming to Tests (failures, skipped, flakes) in [Deprecated] Storage Jul 2, 2024

nicktrav assigned sumeerbhola Jul 2, 2024

nicktrav added the P-2 Issues/test failures with a fix SLA of 3 months label Jul 2, 2024

sumeerbhola removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jul 3, 2024

sumeerbhola mentioned this issue Jul 10, 2024

tests: for kv0 overload test, make jemalloc release memory to OS more… #126930

Merged

sumeerbhola closed this as completed Jul 12, 2024

github-project-automation bot moved this from Tests (failures, skipped, flakes) to Done in [Deprecated] Storage Jul 12, 2024

blathers-crl bot mentioned this issue Jul 29, 2024

release-24.2: tests: for kv0 overload test, make jemalloc release memory to OS more… #127828

Merged

github-project-automation bot added this to KV Aug 28, 2024

github-project-automation bot moved this to Incoming in KV Aug 28, 2024

github-project-automation bot added this to Test Engineering Aug 28, 2024

github-project-automation bot moved this to Done in Test Engineering Aug 28, 2024

DarrylWong mentioned this issue Sep 23, 2024

roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed #130223

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed #125769

roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed #125769

cockroach-teamcity commented Jun 17, 2024 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jun 19, 2024

cockroach-teamcity commented Jun 21, 2024

renatolabs commented Jun 25, 2024

renatolabs commented Jun 26, 2024

arulajmani commented Jun 26, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

renatolabs commented Jun 28, 2024

renatolabs commented Jun 28, 2024 •

edited

Loading

cockroach-teamcity commented Jul 3, 2024

sumeerbhola commented Jul 3, 2024

sumeerbhola commented Jul 3, 2024

renatolabs commented Jul 3, 2024

cockroach-teamcity commented Jul 6, 2024

sumeerbhola commented Jul 6, 2024

sumeerbhola commented Jul 12, 2024

roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed #125769

roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed #125769

Comments

cockroach-teamcity commented Jun 17, 2024 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jun 19, 2024

cockroach-teamcity commented Jun 21, 2024

renatolabs commented Jun 25, 2024

renatolabs commented Jun 26, 2024

arulajmani commented Jun 26, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

renatolabs commented Jun 28, 2024

renatolabs commented Jun 28, 2024 • edited Loading

cockroach-teamcity commented Jul 3, 2024

sumeerbhola commented Jul 3, 2024

sumeerbhola commented Jul 3, 2024

renatolabs commented Jul 3, 2024

cockroach-teamcity commented Jul 6, 2024

sumeerbhola commented Jul 6, 2024

sumeerbhola commented Jul 12, 2024

cockroach-teamcity commented Jun 17, 2024 •

edited by cockroach-jira-scripts

Loading

renatolabs commented Jun 28, 2024 •

edited

Loading