Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix memory trouble shoot #145

Merged
merged 2 commits into from
Nov 1, 2021
Merged

fix memory trouble shoot #145

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
454bd18
fix memory trouble shoot
yun-yeo Nov 1, 2021
e362424
specify versioning
yun-yeo Nov 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 8 additions & 36 deletions docs/How-to/Manage-a-Terra-validator/Troubleshoot-validator-problems.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,43 +109,15 @@ If you run a [local LCD](../Start-LCD.md) (for example, localhost:1317), ensure

## Terrad crashes because of memory fragmentation

As described in [this issue](https://github.com/terra-money/core/issues/592), the default memory allocator causes memory fragmentation on the `Columbus-5` network. To fix this problem, complete the following steps to install `jemalloc` and preload the `jemalloc` shared library:
As described in [this issue](https://github.com/terra-money/core/issues/592), huge memory allocation can cause memory fragmentation issue. Temporal solution is just using small wasm cache size like 50~100MB.

1. Install `jemalloc`.
If you use v0.5.10+,

```bash
JEMALLOC_VERSION=5.2.1
wget https://github.com/jemalloc/jemalloc/releases/download/$JEMALLOC_VERSION/jemalloc-$JEMALLOC_VERSION.tar.bz2
tar -xf ./jemalloc-$JEMALLOC_VERSION.tar.bz2
cd jemalloc-$JEMALLOC_VERSION

# for query node, we recommend to use below configuration
# ./configure --with-malloc-conf=background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000
./configure --with-malloc-conf=background_thread:true,metadata_thp:auto,dirty_decay_ms:30000,muzzy_decay_ms:30000
make
sudo make install
```toml
contract-memory-cache-size = 100
```

2. Restart the process by running `LD_PRELOAD=/usr/local/lib/libjemalloc.so terrad start`.

If you are using `systemd` or another process manager to launch `terrad`, you might need to configure them. The following sample `systemd` file fixes the problem:

```systemd
# /etc/systemd/system/terrad.service
[Unit]
Description=Terra Columbus Node
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu
ExecStart=/home/ubuntu/go/bin/terrad start
Restart=on-failure
RestartSec=3
LimitNOFILE=4096
Environment="LD_PRELOAD=/usr/local/lib/libjemalloc.so"

[Install]
WantedBy=multi-user.target
```
If you use v0.5.7~v0.5.9,
```toml
write-vm-memory-cache-size = 100
```