Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for segment replication GA release #3461

Merged
merged 18 commits into from
Apr 18, 2023
Merged

Add documentation for segment replication GA release #3461

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9da6776
Add documentation for segment replication GA release
ariamarble Mar 15, 2023
0d594e8
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 5, 2023
4567d50
SegRep doc updates GA
ariamarble Apr 5, 2023
5796571
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 10, 2023
52f11b0
content updates
ariamarble Apr 10, 2023
5446b9e
Apply suggestions from doc review
ariamarble Apr 10, 2023
9d8809e
additional doc review changes
ariamarble Apr 10, 2023
813ed17
small changes
ariamarble Apr 10, 2023
4523881
Apply suggestions from editorial review
ariamarble Apr 11, 2023
1289018
Update _tuning-your-cluster/availability-and-recovery/segment-replica…
ariamarble Apr 11, 2023
2f551ca
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 11, 2023
dbc27dd
Add primary balance information
kolchfa-aws Apr 14, 2023
a6c2438
Merge branch 'issue12255' of https://github.com/opensearch-project/do…
ariamarble Apr 15, 2023
966585f
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 15, 2023
d79ab3f
adding backpressure page
ariamarble Apr 17, 2023
8d51eeb
Implemented tech review comments
kolchfa-aws Apr 17, 2023
b697049
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Apr 18, 2023
343a954
final changes
ariamarble Apr 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 0 additions & 84 deletions ...ing-your-cluster/availability-and-recovery/segment-replication/configuration.md
View file
Open in desktop

This file was deleted.

188 changes: 181 additions & 7 deletions _tuning-your-cluster/availability-and-recovery/segment-replication/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,191 @@ redirect_from:

# Segment replication

Segment replication is an experimental feature with OpenSearch 2.3. Therefore, we do not recommend the use of segment replication in a production environment. For updates on the progress of segment replication or if you want leave feedback that could help improve the feature, see the [Segment replication git issue](https://github.com/opensearch-project/OpenSearch/issues/2194).
{: .warning}

With segment replication, segment files are copied across shards instead of documents being indexed on each shard copy. This improves indexing throughput and lowers resource utilization at the expense of increased network utilization.

As an experimental feature, segment replication will be behind a feature flag and must be enabled on **each node** of a cluster and pass a new setting during index creation.
{: .note }
This is the first step in a series of features designed to decouple reads and writes in order to lower compute costs.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

### Potential use cases
## Use cases

- Users who have high write loads but do not have high search requirements and are comfortable with longer refresh times.
- Users with very high loads who want to add new nodes, as you do not need to index all nodes when adding a new node to the cluster.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
- Compared to document replication, segment replication performs better in OpenSearch cluster deployments with low replica counts, such as those used for log analytics.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

## Segment replication configuration

To set the replication strategy to segment replication, create an index with replication.type set to `SEGMENT`:
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

````json
PUT /my-index1
{
"settings": {
"index": {
"replication.type": "SEGMENT"
}
}
}
````

## Benchmarking
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

During experimental testing, our experimental release users reported up to 40% higher throughput with segment replication than with document replication for the same cluster setup.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

The following benchmarks were collected with [OpenSearch-benchmark](https://github.com/opensearch-project/opensearch-benchmark) using the [`stackoverflow`](https://www.kaggle.com/datasets/stackoverflow/stackoverflow) and [`nyc_taxi`](https://github.com/topics/nyc-taxi-dataset) datasets.

Both test runs were performed on a 10-node (m5.xlarge) cluster with 10 shards and 5 replicas. Each shard was about <!-- TODO: insert size --> xxMBs in size. The benchmarking results are listed in the following table.

<table>
<tr>
<td></td>
<td></td>
<td>Document Replication</td>
<td>Segment Replication</td>
<td>Percent difference</td>
</tr>
<tr>
<td>Test execution time (minutes)</td>
<td></td>
<td>40.00</td>
<td>22.00</td>
<td></td>
</tr>
<tr>
<td rowspan="3">Throughput (number of requests per second)</td>
<td>p0</td>
<td>17553.90</td>
<td>28584.30</td>
<td>63%</td>
</tr>
<tr>
<td>p50</td>
<td>20647.20</td>
<td>32790.20</td>
<td>59%</td>
</tr>
<tr>
<td>p100</td>
<td>23209.00</td>
<td>34286.00</td>
<td>48%</td>
</tr>
<tr>
<td rowspan="4">CPU (%)</td>
<td>p50</td>
<td>65.00</td>
<td>30.00</td>
<td>-54%</td>
</tr>
<tr>
<td>p90</td>
<td>79.00</td>
<td>35.00</td>
<td>-56%</td>
</tr>
<tr>
<td>p99</td>
<td>98.00</td>
<td>45.08</td>
<td>-54%</td>
</tr>
<tr>
<td>p100</td>
<td>98.00</td>
<td>59.00</td>
<td>-40%</td>
</tr>
<tr>
<td rowspan="4">Memory (%)</td>
<td>p50</td>
<td>48.20</td>
<td>39.00</td>
<td>-19%</td>
</tr>
<tr>
<td>p90</td>
<td>62.00</td>
<td>61.00</td>
<td>-2%</td>
</tr>
<tr>
<td>p99</td>
<td>66.21</td>
<td>68.00</td>
<td>3%</td>
</tr>
<tr>
<td>p100</td>
<td>71.00</td>
<td>69.00</td>
<td>-3%</td>
</tr>
<tr>
<td rowspan="4">IOPS</td>
<td>p50</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p90</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p99</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p100</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td rowspan="4">Latency</td>
<td>p50</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p90</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p99</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>p100</td>
<td></td>
<td></td>
<td></td>
</tr>
</table>

**Note** : Your results may vary based on the cluster topology, hardware used, shard count, and merge settings.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

Your results may vary based on cluster topology, hardware used, shard count, and merge settings.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
{: .note }

## Further resources regarding segment replication
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

1. Track [future enhancements on segment replication](https://github.com/orgs/opensearch-project/projects/99).
2. [Segment replication blog post](https://github.com/opensearch-project/project-website/pull/1504/files).

## Other considerations
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

This is the first step in a series of features designed to decouple reads and writes in order to lower compute costs.
1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685).
1. Rolling upgrades are currently not supported. Full cluster restarts are required when upgrading indexes using segment replication. [Issue 3881](https://github.com/opensearch-project/OpenSearch/issues/3881).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
1. Increased network congestion on primary shards. [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
1. Integration with remote-backed storage as the source of replication is [currently unsupported](https://github.com/opensearch-project/OpenSearch/issues/4448).
1. Read-after-write guarantees: The `wait_until` refresh policy is not compatible with segment replication. If you use the `wait_until` refresh policy while ingesting documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/6046).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
1. System indexes will continue to use document replication internally until read-after-write guarantees are available. In this case, document replication does not hinder the overall performance because there are few system indexes.