diff --git a/TOC.md b/TOC.md index 136f84aa5f829..e207bcc4781ac 100644 --- a/TOC.md +++ b/TOC.md @@ -74,11 +74,9 @@ - [Overview](/how-to/monitor/overview.md) - [Monitor a TiDB Cluster](/how-to/monitor/monitor-a-cluster.md) + Migrate - - [Overview](/how-to/migrate/overview.md) + - [Migration Tools User Guide](/reference/tools/user-guide.md) + Migrate from MySQL - - [Migrate the Full Data](/how-to/migrate/from-mysql.md) - - [Migrate the Incremental Data](/how-to/migrate/incrementally-from-mysql.md) - - [Migrate from Aurora](/how-to/migrate/from-aurora.md) + - [Migration Case of Amazon Aurora](/how-to/migrate/from-mysql-aurora.md) - [Migrate from CSV](/reference/tools/tidb-lightning/csv.md) + Maintain - [Common Ansible Operations](/how-to/deploy/orchestrated/ansible-operations.md) @@ -308,6 +306,7 @@ - [Error Handling](/reference/tidb-binlog/troubleshoot/error-handling.md) - [FAQ](/reference/tidb-binlog/faq.md) + Tools + + - [Tools User Guide](/reference/tools/user-guide.md) - [Mydumper](/reference/tools/mydumper.md) - [Syncer](/reference/tools/syncer.md) - [Loader](/reference/tools/loader.md) @@ -346,7 +345,7 @@ - [Skip or Replace Abnormal SQL Statements](/reference/tools/data-migration/skip-replace-sqls.md) - [Monitor](/reference/tools/data-migration/monitor.md) + Migrate from MySQL compatible database - - [Migrate from Aurora](/how-to/migrate/from-aurora.md) + - [Migrate from Amazon Aurora](/how-to/migrate/from-mysql-aurora.md) + Troubleshoot - [DM Troubleshooting](/reference/tools/data-migration/troubleshoot/dm.md) - [Error Description](/reference/tools/data-migration/troubleshoot/error-system.md) diff --git a/how-to/migrate/from-aurora.md b/how-to/migrate/from-mysql-aurora.md similarity index 97% rename from how-to/migrate/from-aurora.md rename to how-to/migrate/from-mysql-aurora.md index 901983ae235c3..bc8c10007ee7e 100644 --- a/how-to/migrate/from-aurora.md +++ b/how-to/migrate/from-mysql-aurora.md @@ -1,10 +1,11 @@ --- -title: Migrate from Amazon Aurora MySQL to TiDB -summary: Learn how to migrate from Amazon Aurora MySQL to TiDB by using TiDB Data Migration (DM). +title: Migrate from MySQL (Amazon Aurora) +summary: Learn how to migrate from MySQL (using a case of Amazon Aurora) to TiDB by using TiDB Data Migration (DM). category: how-to +aliases: ['/docs/v3.0/how-to/migrate/from-aurora/','/docs/op-guide/migration/'] --- -# Migrate from Amazon Aurora MySQL to TiDB +# Migrate from MySQL (Amazon Aurora) This document describes how to migrate from [Amazon Aurora MySQL](https://aws.amazon.com/rds/aurora/details/mysql-details/?nc1=h_ls) to TiDB by using TiDB Data Migration (DM). diff --git a/how-to/migrate/from-mysql.md b/how-to/migrate/from-mysql.md deleted file mode 100644 index 70dbf33d1f55c..0000000000000 --- a/how-to/migrate/from-mysql.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: Migrate Data from MySQL to TiDB -summary: Use `mydumper` and `loader` to migrate data from MySQL to TiDB. -category: how-to -aliases: ['/docs/op-guide/migration/'] ---- - -# Migrate Data from MySQL to TiDB - -## Use the `mydumper`/`loader` tool to export and import all the data - -You can use `mydumper` to export data from MySQL and `loader` to import the data into TiDB. - -> **Note:** -> -> Although TiDB also supports the official `mysqldump` tool from MySQL for data migration, it is not recommended to use it. Its performance is much lower than `mydumper` / `loader` and it takes much time to migrate large amounts of data. It is important to use the `mydumper` provided by TiDB and not the upstream `mydumper` version. See [Mydumper](/reference/tools/mydumper.md) for more information. - -`Mydumper` and `loader` can be [downloaded as part of Enterprise Tools](/reference/tools/download.md). - -### Export data from MySQL - -Use the `mydumper` tool to export data from MySQL by using the following command: - -{{< copyable "shell-regular" >}} - -```bash -./bin/mydumper -h 127.0.0.1 -P 3306 -u root -t 16 -F 64 -B test -T t1,t2 --skip-tz-utc -o ./var/test -``` - -In this command, - -- `-B test`: means the data is exported from the `test` database. -- `-T t1,t2`: means only the `t1` and `t2` tables are exported. -- `-t 16`: means 16 threads are used to export the data. -- `-F 64`: means a table is partitioned into chunks and one chunk is 64MB. -- `--skip-tz-utc`: the purpose of adding this parameter is to ignore the inconsistency of time zone setting between MySQL and the data exporting machine and to disable automatic conversion. - -> **Note:** -> -> On the Cloud platforms which require the `super privilege`, such as on the Amazon RDS, add the `--no-locks` parameter to the command. If not, you might get the error message that you don't have the privilege. - -### Import data to TiDB - -Use `loader` to import the data from MySQL to TiDB. See [Loader instructions](/reference/tools/loader.md) for more information. - -{{< copyable "shell-regular" >}} - -```bash -./bin/loader -h 127.0.0.1 -u root -P 4000 -t 32 -d ./var/test -``` - -After the data is imported, you can view the data in TiDB using the MySQL client: - -{{< copyable "shell-regular" >}} - -```bash -mysql -h 127.0.0.1 -P 4000 -u root -``` - -{{< copyable "sql" >}} - -```sql -show tables; -``` - -``` -+----------------+ -| Tables_in_test | -+----------------+ -| t1 | -| t2 | -+----------------+ -``` - -{{< copyable "sql" >}} - -```sql -select * from t1; -``` - -``` -+----+------+ -| id | age | -+----+------+ -| 1 | 1 | -| 2 | 2 | -| 3 | 3 | -+----+------+ -``` - -{{< copyable "sql" >}} - -```sql -select * from t2; -``` - -``` -+----+------+ -| id | name | -+----+------+ -| 1 | a | -| 2 | b | -| 3 | c | -+----+------+ -``` - -### Best practice - -To migrate data quickly, especially for huge amount of data, you can refer to the following recommendations. - -- Keep the exported data file as small as possible and it is recommended keep it within 64M. You can use the `-F` parameter to set the value. -- You can adjust the `-t` parameter of `loader` based on the number and the load of TiKV instances. For example, if there are three TiKV instances, `-t` can be set to 3 * (1 ~ n). If the load of TiKV is too high and the log `backoffer.maxSleep 15000ms is exceeded` is displayed many times, decrease the value of `-t`; otherwise, increase it. - -### A sample and the configuration - -- The total size of the exported files is 214G. A single table has 8 columns and 2 billion rows. -- The cluster topology: - - 12 TiKV instances: 4 nodes, 3 TiKV instances per node - - 4 TiDB instances - - 3 PD instances -- The configuration of each node: - - CPU: Intel Xeon E5-2670 v3 @ 2.30GHz - - 48 vCPU [2 x 12 physical cores] - - Memory: 128G - - Disk: sda [raid 10, 300G] sdb[RAID 5, 2T] - - Operating System: CentOS 7.3 -- The `-F` parameter of `mydumper` is set to 16 and the `-t` parameter of `loader` is set to 64. - -**Results**: It takes 11 hours to import all the data, which is 19.4G/hour. diff --git a/how-to/migrate/incrementally-from-mysql.md b/how-to/migrate/incrementally-from-mysql.md deleted file mode 100644 index 93991a6415741..0000000000000 --- a/how-to/migrate/incrementally-from-mysql.md +++ /dev/null @@ -1,215 +0,0 @@ ---- -title: Migrate Incrementally Using Syncer -summary: Use `mydumper`, `loader` and `syncer` tools to migrate data from MySQL to TiDB. -category: how-to -aliases: ['/docs/op-guide/migration-incremental/'] ---- - -# Migrate Incrementally Using Syncer - -The [previous guide](/how-to/migrate/from-mysql.md) introduces how to import a full database from MySQL to TiDB using `mydumper`/`loader`. This methodology is not recommended for large databases with frequent updates, since it can lead to a larger downtime window during migration. It is instead recommended to use syncer. - -Syncer can be [downloaded as part of Enterprise Tools](/reference/tools/download.md). - -Assuming the data from `t1` and `t2` is already imported to TiDB using `mydumper`/`loader`. Now we hope that any updates to these two tables are replicated to TiDB in real time. - -## Obtain the position to replicate - -The data exported from MySQL contains a metadata file which includes the position information. Take the following metadata information as an example: - -``` -Started dump at: 2017-04-28 10:48:10 -SHOW MASTER STATUS: - Log: mysql-bin.000003 - Pos: 930143241 - GTID: - -Finished dump at: 2017-04-28 10:48:11 -``` - -The position information (`Pos: 930143241`) needs to be stored in the `syncer.meta` file for `syncer` to replicate: - -{{< copyable "shell-regular" >}} - -```bash -cat syncer.meta -``` - -``` -binlog-name = "mysql-bin.000003" -binlog-pos = 930143241 -``` - -> **Note:** -> -> The `syncer.meta` file only needs to be configured once when it is first used. The position will be automatically updated when binlog is replicated. - -## Start `syncer` - -The `config.toml` file for `syncer`: - -```toml -log-level = "info" -log-file = "syncer.log" -log-rotate = "day" - -server-id = 101 - -# The file path for meta: -meta = "./syncer.meta" -worker-count = 16 -batch = 1000 -flavor = "mysql" - -# The testing address for pprof. It can also be used by Prometheus to pull Syncer metrics. -status-addr = ":8271" - -# If you set its value to true, Syncer stops and exits when it encounters the DDL operation. -stop-on-ddl = false - -# max-retry is used for retry during network interruption. -max-retry = 100 - -# Specify the database name to be replicated. Support regular expressions. Start with '~' to use regular expressions. -# replicate-do-db = ["~^b.*","s1"] - -# Specify the database you want to ignore in replication. Support regular expressions. Start with '~' to use regular expressions. -# replicate-ignore-db = ["~^b.*","s1"] - -# skip-ddls skips the ddl statements. -# skip-ddls = ["^OPTIMIZE\\s+TABLE"] - -# skip-dmls skips the DML statements. The type value can be 'insert', 'update' and 'delete'. -# The 'delete' statements that skip-dmls skips in the foo.bar table: -# [[skip-dmls]] -# db-name = "foo" -# tbl-name = "bar" -# type = "delete" -# -# The 'delete' statements that skip-dmls skips in all tables: -# [[skip-dmls]] -# type = "delete" -# -# The 'delete' statements that skip-dmls skips in all foo.* tables: -# [[skip-dmls]] -# db-name = "foo" -# type = "delete" - -# Specify the db.table to be replicated. -# db-name and tbl-name do not support the `db-name ="dbname, dbname2"` format. -# [[replicate-do-table]] -# db-name ="dbname" -# tbl-name = "table-name" - -# [[replicate-do-table]] -# db-name ="dbname1" -# tbl-name = "table-name1" - -# Specify the db.table to be replicated. Support regular expressions. Start with '~' to use regular expressions. -# [[replicate-do-table]] -# db-name ="test" -# tbl-name = "~^a.*" - -# Specify the database table you want to ignore in replication. -# db-name and tbl-name do not support the `db-name ="dbname, dbname2"` format. -# [[replicate-ignore-table]] -# db-name = "your_db" -# tbl-name = "your_table" - -# Specify the database table you want to ignore in replication. Support regular expressions. Start with '~' to use regular expressions. -# [[replicate-ignore-table]] -# db-name ="test" -# tbl-name = "~^a.*" - -# The sharding replicating rules support wildcharacter. -# 1. The asterisk character ("*", also called "star") matches zero or more characters, -# For example, "doc*" matches "doc" and "document" but not "dodo"; -# The asterisk character must be in the end of the wildcard word, -# and there is only one asterisk in one wildcard word. -# 2. The question mark ("?") matches any single character. -# [[route-rules]] -# pattern-schema = "route_*" -# pattern-table = "abc_*" -# target-schema = "route" -# target-table = "abc" - -# [[route-rules]] -# pattern-schema = "route_*" -# pattern-table = "xyz_*" -# target-schema = "route" -# target-table = "xyz" - -[from] -host = "127.0.0.1" -user = "root" -password = "" -port = 3306 - -[to] -host = "127.0.0.1" -user = "root" -password = "" -port = 4000 -``` - -Start `syncer`: - -{{< copyable "shell-regular" >}} - -```bash -./bin/syncer -config config.toml -``` - -``` -2016/10/27 15:22:01 binlogsyncer.go:226: [info] begin to sync binlog from position (mysql-bin.000003, 1280) -2016/10/27 15:22:01 binlogsyncer.go:130: [info] register slave for master server 127.0.0.1:3306 -2016/10/27 15:22:01 binlogsyncer.go:552: [info] rotate to (mysql-bin.000003, 1280) -2016/10/27 15:22:01 syncer.go:549: [info] rotate binlog to (mysql-bin.000003, 1280) -``` - -## Insert data into MySQL - -{{< copyable "sql" >}} - -```sql -INSERT INTO t1 VALUES (4, 4), (5, 5); -``` - -## Log in TiDB and view the data - -{{< copyable "shell-regular" >}} - -```bash -mysql -h 127.0.0.1 -P 4000 -u root -p -``` - -{{< copyable "sql" >}} - -```sql -select * from t1; -``` - -``` -+----+------+ -| id | age | -+----+------+ -| 1 | 1 | -| 2 | 2 | -| 3 | 3 | -| 4 | 4 | -| 5 | 5 | -+----+------+ -``` - -`syncer` outputs the current replicated data statistics every 30 seconds: - -``` -2017/06/08 01:18:51 syncer.go:934: [info] [syncer]total events = 15, total tps = 130, recent tps = 4, -master-binlog = (ON.000001, 11992), master-binlog-gtid=53ea0ed1-9bf8-11e6-8bea-64006a897c73:1-74, -syncer-binlog = (ON.000001, 2504), syncer-binlog-gtid = 53ea0ed1-9bf8-11e6-8bea-64006a897c73:1-17 -2017/06/08 01:19:21 syncer.go:934: [info] [syncer]total events = 15, total tps = 191, recent tps = 2, -master-binlog = (ON.000001, 11992), master-binlog-gtid=53ea0ed1-9bf8-11e6-8bea-64006a897c73:1-74, -syncer-binlog = (ON.000001, 2504), syncer-binlog-gtid = 53ea0ed1-9bf8-11e6-8bea-64006a897c73:1-35 -``` - -You can see that by using `syncer`, the updates in MySQL are automatically replicated in TiDB. diff --git a/how-to/migrate/overview.md b/how-to/migrate/overview.md deleted file mode 100644 index 3da7101d84ef3..0000000000000 --- a/how-to/migrate/overview.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Migration Overview -summary: Learn how to migrate data into TiDB. -category: how-to -aliases: ['/docs/op-guide/migration-overview/'] ---- - -# Migration Overview - -This document describes scenarios for migrating data into TiDB from either MySQL or another data source via CSV format. - -## Tools overview - -Migrations will often make use of the following tools. The following is a brief overview of their usage: - -- [`mydumper`](/reference/tools/mydumper.md) exports data from MySQL. It is recommended over using mysqldump. -- [`loader`](/reference/tools/loader.md) imports data in Mydumper format into TiDB. -- [`syncer`](/reference/tools/syncer.md) acts like a MySQL replication slave and pushes data from MySQL into TiDB. -- [DM](/reference/tools/data-migration/overview.md) (Data Migration) integrates the functions of Mydumper, Loader and syncer to support the export and import of full-size data, as well as incremental replication of MySQL Binlog data, and supports data replication of a more complete pooled table scenario. -- [TiDB Lightning](/reference/tools/tidb-lightning/overview.md) imports data to TiDB in an optimized way. For example, a 1TiB backup could take 24+ hours to import with loader, while it will typically complete at least 3 times faster in TiDB Lightning. - -## Scenarios - -The following example scenarios show how you can put to use the tools mentioned above. - -### Full data migration from MySQL - -To migrate the full data, you can use one of the following three solutions: - -- Mydumper + Loader: use Mydumper to export data from MySQL and use Loader to import the data into TiDB. -- Mydumper + TiDB Lightning: use Mydumper to export data from MySQL and use TiDB Lightning to import the data into TiDB. -- DM: use DM to export data from MySQL and import the data into TiDB. - -For detailed operations, follow the steps in [Migrate Data from MySQL to TiDB](/how-to/migrate/from-mysql.md). - -### Full data migration and incremental replication - -To migrate the full data and then replicate data incrementally, you can use one of the following three solutions: - -- Mydumper + Loader + Syncer: use Mydumper to export data from MySQL, use Loader to import the data into TiDB, and then use Syncer to replicate the incremental binlog data from MySQL into TiDB. -- Mydumper + TiDB Lightning + Syncer: use Mydumper to export data from MySQL, use TiDB Lightning to import the data into TiDB, and then use Syncer to replicate the incremental binlog data from MySQL into TiDB. -- DM: use DM to migrate the full data from MySQL to TiDB and then replicate the incremental data from MySQL into TiDB. - -For detailed operations, follow the steps in [Incremental Migration](/how-to/migrate/incrementally-from-mysql.md). - - > **Note:** - > - > To replicate data from MySQL to TiDB, binary logging [must be enabled](http://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) with the [row format](https://dev.mysql.com/doc/refman/5.7/en/binary-log-formats.html) enabled. - -### Dump and restore from database other than MySQL - -To import data from another database, it is recommended to: - -- Export the data as CSV format. -- Import the data using TiDB Lightning. - -For more information, refer to [CSV support for TiDB Lightning](/reference/tools/tidb-lightning/csv.md). diff --git a/reference/sql/statements/recover-table.md b/reference/sql/statements/recover-table.md index 9a831e5ec7a32..4289fe9a6c8ac 100644 --- a/reference/sql/statements/recover-table.md +++ b/reference/sql/statements/recover-table.md @@ -44,7 +44,7 @@ When you use `RECOVER TABLE` in the upstream TiDB during TiDB Binlog replication + Latency occurs during replication between upstream and downstream databases. An error instance: `snapshot is older than GC safe point 2019-07-10 13:45:57 +0800 CST`. -For the above three situations, you can resume data replication from TiDB Binlog with a [full import of the deleted table](/how-to/migrate/overview.md#full-data-migration-from-mysql). +For the above three situations, you can resume data replication from TiDB Binlog with a [full import of the deleted table](/reference/tools/user-guide.md#full-backup-and-restore-of-tidb-cluster-data-1). ## Examples diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md new file mode 100644 index 0000000000000..cca8417923466 --- /dev/null +++ b/reference/tools/user-guide.md @@ -0,0 +1,189 @@ +--- +title: TiDB Ecosystem Tools User Guide +category: reference +aliases: ['/docs/v3.0/how-to/migrate/from-mysql/','/docs/v3.0/how-to/migrate/incrementally-from-mysql/','/docs/v3.0/how-to/migrate/overview/'] +--- + +# TiDB Ecosystem Tools User Guide + +Currently, TiDB has multiple ecosystem tools. Some of them have overlapping functionality, and some are different versions of the same tool. This document introduces each of these tools, illustrates their relationship, and describes when to use which tool for each TiDB version. + +## TiDB ecosystem tools overview + +TiDB ecosystem tools can be divided into: + +- Data import tools, including full import tools, backup and restore tools, incremental import tools, and so forth. +- Data export tools, including full export tools. incremental export tools, and so forth. + +The two types of tools are discussed in detail below. + +### Data import tools + +#### Full import tool TiDB Lightning + +[TiDB Lightning](/reference/tools/tidb-lightning/overview.md) is a tool used for fast full import of data into a TiDB cluster. + +> **Note:** +> +> When you import data into TiDB using TiDB Lightning, there are two modes: +> +> - The default mode: Use `tikv-importer` as the backend. In this mode, the cluster can not provide normal services during the data import process. It is used when you import large amounts (TBs) of data. +> - The second mode: Use `TiDB` as the backend (similar to Loader). The import speed is slower than that in the default mode. However, the second mode supports online import. + +The following are the basics of TiDB Lightning: + +- Input: + - Files output by Mydumper; + - CSV files. +- Compatibility: Compatible with TiDB v2.1 and later versions. +- Kubernetes: Supported. See [Quickly restore data into a TiDB cluster in Kubernetes using TiDB Lightning](/tidb-in-kubernetes/maintain/lightning.md). + +#### Backup and restore tool BR + +[BR](https://pingcap.com/docs/dev/how-to/maintain/backup-and-restore/br/) is a command-line tool used for distributed data backup and restoration for a TiDB cluster. Compared with Mydumper and Loader, BR allows you to finish backup and restore tasks with greater efficiency in scenarios of huge data volume. + +The following are the basics of BR: + +- [Types of backup files](https://pingcap.com/docs/dev/reference/tools/br/br/#types-of-backup-files): The SST file and the `backupmeta` file. +- Compatibility: Compatible with TiDB v3.1 and v4.0 versions. +- Kubernetes: Supported. Relevant documents are on the way. + +#### Incremental and full import tool TiDB Data Migration + +[TiDB Data Migration (DM)](/reference/tools/data-migration/overview.md) is an tool used for data migration from MySQL/MariaDB into TiDB. It supports both the full and incremental data replication. + +The following are the basics of DM: + +- Input: Full data and binlog data of MySQL/MariaDB. +- Output: SQL statements written to TiDB. +- Compatibility: Compatible with all TiDB versions. +- Kubernetes: In development. + +#### Full import tool Loader (Stop maintenance, not recommended) + +[Loader](/reference/tools/loader.md) is a lightweight full data import tool. Data is imported into TiDB in the form of SQL statements. Currently, this tool is gradually replaced by [TiDB Lightning](#full-import-tool-tidb-lightning), see [TiDB Lightning TiDB-backend Document](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend). + +The following are the basics of Loader: + +- Input: Files output by Mydumper. +- Output: SQL statements written to TiDB. +- Compatibility: Compatible with all TiDB versions. +- Kubernetes: Supported. See [Backup and restore](/tidb-in-kubernetes/maintain/backup-and-restore.md). + +#### Incremental import tool Syncer (Stop maintenance, not recommended) + +[Syncer](/reference/tools/syncer.md) is a tool used for incremental import of real-time binlog data from MySQL/MariaDB into TiDB. It is recommended to use [TiDB Data Migration](#Incremental-import-tool-tidb-data-migration) to replace Syncer. + +The following are the basics of Syncer: + +- Input: Binlog data of MySQL/MariaDB. +- Output: SQL statements written to TiDB. +- Compatibility: Compatible with all TiDB versions. +- Kubernetes: Not supported. + +### Data export tools + +#### Full export tool Mydumper + +[Mydumper](/reference/tools/mydumper.md) is a MySQL community tool used for full logical backups of MySQL that also works with TiDB. + +The following are the basics of Mydumper: + +- Input: MySQL/TiDB clusters. +- Output: SQL files. +- Compatibility: Compatible with all TiDB versions. +- Kubernetes: Supported. See [Backup and Restore](/tidb-in-kubernetes/maintain/backup-and-restore.md). + +#### Full export tool TiDB Binlog + +[TiDB Binlog](/reference/tidb-binlog/overview.md) is a tool used to collect binlog data from TiDB. It provides near real-time backup and replication to downstream platforms. + +The following are the basics of TiDB Binlog: + +- Input: TiDB clusters. +- Output: MySQL, TiDB, Kafka or incremental backup files. +- Compatibility: Compatible with TiDB v2.1 and later versions. +- Kubernetes: Supported. See [TiDB Binlog Cluster Operations](/tidb-in-kubernetes/maintain/tidb-binlog.md) and [TiDB Binlog Drainer Configurations in Kubernetes](/tidb-in-kubernetes/reference/configuration/tidb-drainer.md). + +## Tools development roadmap + +To help you understand the relationships between the above tools, here is a brief introduction to TiDB ecosystem tools development roadmap. + +### TiDB backup and restore + +Mydumper and Loader -> BR: + +Mydumper and Loader are inefficient since they back up and restore data on the logical level. BR is much more efficient because it takes advantage of TiDB features for backup and restore tasks. BR can be applied in huge data volume scenarios. + +### TiDB full data restore + +Loader -> TiDB Lightning: + +Loader is inefficient since it performs full data restoration using SQL. TiDB Lightning imports data into TiKV directly, so it is much more efficient and can be used for fast full import of large amounts (more than TBs) of data into a new TiDB cluster. + +TiDB Lightning also integrates the logical data import function of Loader and supports online data import. For details, see [TiDB Lightning TiDB-backend Document](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend). + +### MySQL data migration + +- Mydumper, Loader and Syncer -> DM: + + It is tedious to migrate MySQL data to TiDB using Mydumper, Loader, and Syncer. DM provides an integrated data migration approach that improves usability. DM can be also used to merge the sharded schemas and tables. + +- Loader -> TiDB Lightning: + + TiDB Lightning integrates the logical data import function of Loader. See [TiDB Lightning TiDB-backend document](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend) for details. It is used to perform full data restoration. + +## Data migration solutions + +For TiDB 2.1, 3.0, and 3.1 versions, this section introduces data migration solutions in typical application scenarios. + +### Full link data migration solutions for v3.0 + +#### Migrating MySQL data to TiDB + +If the volume is more than TBs of data, the recommended migration steps are: + +1. Export full MySQL data using Mydumper; +2. Import full backup data from MySQL into a TiDB cluster using TiDB Lightning; +3. Replicate the incremental data of MySQL into TiDB. + +If the volume is less than TBs of data, it is recommended to migrate MySQL data to TiDB using DM (the migrating process includes full data import and incremental data replication). + +#### Replication of TiDB cluster data + +It is recommended that you use TiDB Binlog to replicate TiDB data to downstream TiDB/MySQL. + +#### Full backup and restore of TiDB cluster data + +The recommended steps are: + +1. Back up full data using Mydumper; +2. Restore full data into TiDB/MySQL using TiDB Lightning. + +### Full link data migration solutions for v3.1 + +#### Migrating MySQL data to TiDB + +If the volume is more than TBs of data, the recommended migration steps are: + +1. Export full MySQL data using Mydumper; +2. Import full backup data from MySQL into a TiDB cluster using TiDB Lightning; +3. Replicate the incremental data of MySQL into TiDB. + +If the volume is less than TBs of data, it is recommended to migrate MySQL data to TiDB using DM (the migrating process includes full data import and incremental data replication). + +#### Replication of TiDB cluster data + +It is recommended that you use TiDB Binlog to replicate TiDB data to downstream TiDB/MySQL. + +#### Full backup and restore of TiDB cluster data + +- Restore to TiDB + + - Back up full data using BR; + - Restore full data using BR. + +- Restore to MySQL + + - Back up full data using Mydumper; + - Restore full data using TiDB Lightning.