Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(docs): Pruning and Snapshots recovery basic docs #2265

Merged
merged 12 commits into from
Jun 20, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/bin/external_node/src/config/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -541,7 +541,7 @@ impl OptionalENConfig {
}

fn default_pruning_data_retention_sec() -> u64 {
3_600 // 1 hour
3_600 * 24 * 7 // 7 days
}

fn from_env() -> anyhow::Result<Self> {
Expand Down
26 changes: 26 additions & 0 deletions docs/guides/external-node/07_snapshots_recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Snapshots Recovery

Instead of starting node using DB snapshots, it's possible to configure them to start from a protocol-level snapshots.
This process is much faster and requires way less storage. Postgres database of a mainnet node recovered from a snapshot
is only about 300GB.
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

> [!NOTE]
>
> Nodes recovered from snapshot don't have any historical data from before the recovery!

## Configuration

To enable snapshots-recovery on mainnet, you need to set:
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

EN_SNAPSHOTS_RECOVERY_ENABLED: "true"
EN_SNAPSHOTS_OBJECT_STORE_BUCKET_BASE_URL: "zksync-era-mainnet-external-node-snapshots"
EN_SNAPSHOTS_OBJECT_STORE_MODE: "GCSAnonymousReadOnly"
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

For sepolia testnet, use:

EN_SNAPSHOTS_RECOVERY_ENABLED: "true"
EN_SNAPSHOTS_OBJECT_STORE_BUCKET_BASE_URL: "zksync-era-boojnet-external-node-snapshots"
EN_SNAPSHOTS_OBJECT_STORE_MODE: "GCSAnonymousReadOnly"

For a working examples of a fully configured Nodes recovering from snapshots, see docker-compose-examples directory and
00_quick_start.md
tomg10 marked this conversation as resolved.
Show resolved Hide resolved
38 changes: 38 additions & 0 deletions docs/guides/external-node/08_pruning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Pruning

It is possible to configure Node to periodically remove all data from batches older than a threshold, both from Postgres
and from tree.
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

> [!NOTE]
>
> If you need a node with data retention period of up to a few days, please set up a node from a snapshot (see previous
> chapter) and wait for it to have enough data. Pruning an archival node can take unpractical amount of time. In the
tomg10 marked this conversation as resolved.
Show resolved Hide resolved
> future we will be offering pre-pruned DB snapshots with a few months of data.

You can enable pruning by setting
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

```
EN_PRUNING_ENABLED=true
```

By default, it will keep history for 7 days. You can configure retention period using:

```
EN_PRUNING_DATA_RETENTION_SEC: '259200' // 3 days
tomg10 marked this conversation as resolved.
Show resolved Hide resolved
```

The data retention can be set to any value, but values under 21h will as the batch can only be pruned as soon as it has
been executed on Ethereum.
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

## Storage requirements for pruned nodes

The storage requirements depend on how long you configure to retain the data, but are roughly:

40GB + ~5GB/data-retention-day space needed on machine that runs the node
300GB + ~15GB/day-retention-day for Postgres
tomg10 marked this conversation as resolved.
Show resolved Hide resolved

> [!NOTE]
>
> When pruning an existing archival node, Postgre will be unable to reclaim disk space automatically, to reclaim disk
> space, you need to manually run VACUUM FULL, which requires an ACCESS EXCLUSIVE lock, you can read more about it in
> https://www.postgresql.org/docs/current/sql-vacuum.html
Loading