Skip to content

Commit

Permalink
eth2util/keystore: lower amount of loadStoreWorkers (#2584)
Browse files Browse the repository at this point in the history
When running `create cluster` with the `--split-existing-keys` flag, I ran into the charon service getting OOM killed. A bit of trial and error helped me figure this was related to the amount of keys to be split. A low amount (<30) would work fine.

I found out splitting 25 existing keys resulted in a memory consumption of over 6GB, 10 keys used about 2.5GB.

Curious to see what would lead to such high memory consumption, I built the binary locally and extracted a heap dump. The heap dump showed that pretty much all the memory was consumed by `golang.org/x/crypto/scrypt.Key objects` , 256MB per key (temporarily even 512MB is allocated if I'm looking at the data correctly).

64 workers (current value) * 256MB per key results in an upper limit of more than 16GB of memory consumed, with an even higher temporary spike.

This PR lowers the amount of workers to 10, resulting in an upper limit of 10 * 512MB = 5GB of memory consumption.

I didn't investigate why the scrypt.Key object itself needs as much memory as it seems to be using, I'm not that well versed at Go. But it does seem a bit excessive. Fixing that would likely be the more proper fix.

Let me know if this deserves an issue, but it seemed trivial enough to me.

category: bug
ticket: none
  • Loading branch information
eth2353 authored Oct 16, 2023
1 parent 466866f commit 41dd725
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion eth2util/keystore/keystore.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ const (
insecureCost = 4

// loadStoreWorkers is the amount of workers to use when loading/storing keys concurrently.
loadStoreWorkers = 64
loadStoreWorkers = 10
)

type confirmInsecure struct{}
Expand Down

0 comments on commit 41dd725

Please sign in to comment.