Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add user guide for compaction offload #158

Merged
merged 5 commits into from
Nov 22, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion content/en/docs/user-guide/cluster_deployment/with_meta.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,9 +187,28 @@ timeout = "5s"
server_addrs = ['http://{HoraeMetaAddr}:2379']
```

### Compaction Offload

Compaction offload is also supported. To enable compaction offload, the corresponding compaction mode with node picker and endpoint should be configured.

- `node_picker`: There are two types of node picker -- `Local` and `Remote`(WIP).
- When the `Local` is setted, the local compaction task would be offloaded to the specific remote compaction server, which decided by `endpoint`.
- `endpoint`: The endpoint, in the form `addr:port`, indicating the _grpc port_ of the remote compaction server.

Here is an example for it:

```toml
[analytic.compaction_mode]
compaction_mode = "Offload"
node_picker = "Local"
endpoint = "{RemoteCompactionServerAddr}:{RemoteCompactionServerGrpcPort}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remote node picker is not supported without horaemeta. Currently, we need to use local node picker and specify the endpoint.

```

A Compaction Server, responsible for executing the compaction task, is also needed. Currently `horaedb-server` will act as this role, in the future we can move it to an independent service.

### Complete Config of HoraeDB

With all the parts of the configurations mentioned above, a runnable complete config for HoraeDB can be made. In order to make the HoraeDB cluster runnable, we can decide to adopt RocksDB-based WAL and local-disk-based Object Storage:
With all the parts of the configurations mentioned above, a runnable complete config for HoraeDB can be made. In order to make the HoraeDB cluster runnable, we can decide to adopt RocksDB-based WAL and local-disk-based Object Storage without compaction offload:

```toml
[server]
Expand Down