| layout | title | nav_order | has_children | nav_exclude | has_toc |
|---|---|---|---|---|---|
default |
Migration Assistant for OpenSearch |
1 |
false |
true |
false |
Migrations Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios:
- Metadata migration: Migrating cluster metadata, such as index settings, aliases, and templates.
- Backfill migration: Migrating existing or historical data from a source to a target cluster.
- Live traffic migration: Replicating live ongoing traffic from source to target cluster.
- Comparative tooling: Comparing the performance and behaviors of an existing cluster with a prospective new one.
This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration.
It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration. {: .tip }
The following are the key components of Migration Assistant.
Your source cluster in this solution operates on Elasticsearch or OpenSearch, hosted on EC2 instances or similar computing environments. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster.
A console that provides a migration-specific CLI and offers a variety of tools to streamline the migration process. Everything necessary for completing a migration, other than cleaning up the migration resources, can be done via this Console.
This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream-processing service for later playback.
Acting as a traffic simulation tool, the Traffic Replayer replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis.
The Metadata migration tool integrated into the Migration CLI can be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases.
reindex-from-snapshot reindexes data from an existing snapshot on Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster.
The destination cluster for migration or comparison in an A/B test.
The Migration assistant architecture is based on the use of an AWS cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available.
The design deployed in AWS is as follows:
- Client traffic is directed to the existing cluster.
- An Application Load Balancer (ALB) with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apace Kafka (AWS MSK).
- Using the migration console, you can initiate metadata migration to establish indexes, templates, component templates, and aliases on the target cluster.
- With continuous traffic capture in place, you can use a
reindex-from-snapshotprocess to capture data from your current index. - Once
reindex-from-snapshotis complete, captured traffic is replayed from AWS MSK to the target cluster by the traffic replayer. - Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics.
- After confirming the target cluster’s functionality meets expectations, clients are redirected to the new target.