Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migration: Add information about Vitess to TiDB migration #5783

Merged
merged 17 commits into from
Nov 18, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@
- [Migrate Large Datasets from MySQL](/migrate-large-mysql-to-tidb.md)
- [Migrate and Merge MySQL Shards of Small Datasets](/migrate-small-mysql-shards-to-tidb.md)
- [Migrate and Merge MySQL Shards of Large Datasets](/migrate-large-mysql-shards-to-tidb.md)
- [Migrate from Vitess](/migrate-from-vitess.md)
- [Migrate from MariaDB](/migrate-from-mariadb.md)
- [Migrate from CSV Files](/migrate-from-csv-files-to-tidb.md)
- [Migrate from SQL Files](/migrate-from-sql-files-to-tidb.md)
Expand Down
Binary file added media/vitess_to_tidb.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/vitess_to_tidb_dm.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/vitess_to_tidb_dumpling_local.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
34 changes: 34 additions & 0 deletions migrate-from-vitess.md
dveeden marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

@qiancai qiancai Nov 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this document, there are no detailed steps. Are these instructions sufficient for our users?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is ok. @mjonss what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should add something like 'high level migration' to clarify it is not an step-by-step instruction?

Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: Migrate Data from Vitess to TiDB
summary: Learn about the tools to migrate data from Vitess to TiDB.
---

# Migrate Data from Vitess to TiDB

dveeden marked this conversation as resolved.
Show resolved Hide resolved
This document describes the tools that you can use to migrate data from [Vitess](https://vitess.io/) to TiDB.

As the backend of Vitess is based on MySQL, when migrating data from Vitess to TiDB, you can use the same migration tools that are applicable to MySQL, such as [Dumpling](/dumpling-overview.md), [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md), and [TiDB Data Migration (DM)](/dm/dm-overview.md). Note that these tools should be set up for each shard in Vitess for data migration.
dveeden marked this conversation as resolved.
Show resolved Hide resolved

In addition to these tools, you can also use [Debezium connector for Vitess](https://debezium.io/documentation/reference/connectors/vitess.html). This connector enables you to use [Kafka Connect](https://kafka.apache.org/documentation/#connect) or [Apache Flink](https://nightlies.apache.org/flink/flink-docs-stable/) to stream changes from Vitess to TiDB.

Because both Vitess and TiDB support the MySQL protocol and SQL dialect, changes at the application level are expected to be small. For tasks directly managing sharding or other implementation-specific aspects, however, the changes might be larger. To facilitate the data migration from Vitess to TiDB, TiDB introduces the [`VITESS_HASH()`](/functions-and-operators/tidb-functions.md) function, which returns the hash of a string that is compatible with Vitess' HASH function.

## Examples

### Dumpling and TiDB Lightning

The following two examples show how Dumpling and TiDB Lightning work together to migrate data from Vitess to TiDB.

- In this example, TiDB Lightning uses the [logical import mode](/tidb-lightning/tidb-lightning-logical-import-mode.md), which first encodes data into SQL statements and then runs the SQL statements to import data.

![Vitess to TiDB Migration with TiDB backend](/media/vitess_to_tidb.png)

- In this example, TiDB Lightning uses the [physical import mode](/tidb-lightning/tidb-lightning-physical-import-mode.md) to directly ingest data into TiKV.

![Vitess to TiDB Migration with local backend](/media/vitess_to_tidb_dumpling_local.png)

### DM

dveeden marked this conversation as resolved.
Show resolved Hide resolved
The following example shows how DM migrates data from Vitess to TiDB.
dveeden marked this conversation as resolved.
Show resolved Hide resolved
dveeden marked this conversation as resolved.
Show resolved Hide resolved

![Vitess to TiDB with DM](/media/vitess_to_tidb_dm.png)
8 changes: 7 additions & 1 deletion migration-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ If the data size of the sharded tables is large (for example, larger than 1 TiB)

- [Migrate and Merge MySQL Shards of Large Datasets to TiDB](/migrate-large-mysql-shards-to-tidb.md)

## Migrate data from Vitess to TiDB

To migrate data from Vitess to TiDB, the following guide is available:

- [Migrate Data from Vitess to TiDB](/migrate-from-vitess.md)

## Migrate data from files to TiDB

- [Migrate data from CSV files to TiDB](/migrate-from-csv-files-to-tidb.md)
Expand All @@ -61,4 +67,4 @@ The following features can improve the migration process and might meet more nee
- [Continuous Replication from Databases that Use gh-ost or pt-osc](/migrate-with-pt-ghost.md)
- [Migrate Data to a Downstream TiDB Table with More Columns](/migrate-with-more-columns-downstream.md)
- [Filter Binlog Events](/filter-binlog-event.md)
- [Filter DML Events Using SQL Expressions](/filter-dml-event.md)
- [Filter DML Events Using SQL Expressions](/filter-dml-event.md)
Loading