From 8515abde90ba3c69532bf001bb4526b3838d5775 Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 10 Jun 2020 17:14:55 +0800 Subject: [PATCH 1/4] migrate: migrate from mysql files --- migrate-from-mysql-mydumper-files.md | 62 ++++++++++++++++++++++++++++ mydumper-overview.md | 2 +- 2 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 migrate-from-mysql-mydumper-files.md diff --git a/migrate-from-mysql-mydumper-files.md b/migrate-from-mysql-mydumper-files.md new file mode 100644 index 0000000000000..cba514797028d --- /dev/null +++ b/migrate-from-mysql-mydumper-files.md @@ -0,0 +1,62 @@ +--- +title: Migrate Data from MySQL SQL File +summary: Learn how to migrate data from MySQL SQL files to TiDB using TiDB Lightning. +category: how-to +--- + +# Migrate Data from MySQL SQL File + +This document describes how to migrate data from MySQL SQL files to TiDB using TiDB Lightning. For details on how to generate MySQL SQL files, see [Mydumper](/mydumper-overview.md) or [Dumpling](/export-or-backup-using-dumpling.md). + +## Deploy TiDB Lightning + +The data migration process described in this document uses TiDB Lightning. Before you start the migration, [deploy TiDB Lightning](/tidb-lightning/deploy-tidb-lightning.md). + +> **Note:** +> +> - If you choose the importer backend, you need to deploy tikv-importer along with TiDB Lightning. During the import process, the TiDB cluster cannot provide services. This mode imports data quickly, which is suitable for importing a large amount of data (above the TB level). +> - If you choose the TiDB backend, only TiDB Lightning is required. The cluster can provide services normally during the import. +> - For detailed differences between the two backend mode, see [TiDB Lightning Backend](/tidb-lightning/tidb-lightning-tidb-backend.md). + +## Configure data source of TiDB Lightning + +This document takes the TiDB backend as an example. Add the `tidb-lightning.toml` configuration file and add the following major configurations in the file: + +1. Set the `data-source-dir` under `[mydumper]` to the path of the MySQL SQL file. + + ``` + [mydumper] + # data source directory + data-source-dir = "/data/export" + ``` + + > **Note:** + > + > If a corresponding schema already exists in the downstream, set `no-schema=true` to skip the creation of schema. + +2. Add the TiDB configurations of the target cluster. + + ``` + [tidb] + # the target cluster information. Fill in one address of tidb-server + host = "172.16.31.1" + port = 4000 + user = "root" + password = "" + ``` + +For other configurations, see [TiDB Lightning Configuration](/tidb-lightning/tidb-lightning-configuration.md). + +## Run TiDB Lightning to import data + +Run TiDB Lightning to start the import operation. If you start TiDB Lightning by using `nohup` directly in the command line, the program might exit because of the `SIGHUP` signal. Therefore, it is recommended to write `nohup` in a script. For example: + +``` +# !/bin/bash +nohup ./tidb-lightning -config tidb-lightning.toml > nohup.out & +``` + +When the import operation is started, you can view the progress by the following two ways: + +- `grep` the keyword `progress` in logs, which is updated every 5 minutes by default. +- Access the monitoring dashboard. See [Monitor TiDB Lightning](/tidb-lightning/monitor-tidb-lightning.md) for details. diff --git a/mydumper-overview.md b/mydumper-overview.md index 87b17e56def6b..62b20bc9c8b1e 100644 --- a/mydumper-overview.md +++ b/mydumper-overview.md @@ -9,7 +9,7 @@ aliases: ['/docs/dev/reference/tools/mydumper/'] ## What is Mydumper? -[Mydumper](https://github.com/pingcap/mydumper) is a fork project optimized for TiDB. You can use this tool for logical backups of TiDB. +[Mydumper](https://github.com/pingcap/mydumper) is a fork project optimized for TiDB. You can use this tool for logical backups of **MySQL** or **TiDB**. It can be [downloaded](/download-ecosystem-tools.md) as part of the Enterprise Tools package. From 08bae4d2ffbafb40599de5199ff87b6e6ec08d40 Mon Sep 17 00:00:00 2001 From: yikeke Date: Thu, 11 Jun 2020 15:41:28 +0800 Subject: [PATCH 2/4] refine format and terms --- migrate-from-mysql-mydumper-files.md | 32 +++++++++++++++------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/migrate-from-mysql-mydumper-files.md b/migrate-from-mysql-mydumper-files.md index cba514797028d..c474c2ea022db 100644 --- a/migrate-from-mysql-mydumper-files.md +++ b/migrate-from-mysql-mydumper-files.md @@ -1,26 +1,28 @@ --- -title: Migrate Data from MySQL SQL File +title: Migrate Data from MySQL SQL Files summary: Learn how to migrate data from MySQL SQL files to TiDB using TiDB Lightning. category: how-to --- -# Migrate Data from MySQL SQL File +# Migrate Data from MySQL SQL Files -This document describes how to migrate data from MySQL SQL files to TiDB using TiDB Lightning. For details on how to generate MySQL SQL files, see [Mydumper](/mydumper-overview.md) or [Dumpling](/export-or-backup-using-dumpling.md). +This document describes how to migrate data from MySQL SQL files to TiDB using TiDB Lightning. For details on how to generate MySQL SQL files, refer to [Mydumper](/mydumper-overview.md) or [Dumpling](/export-or-backup-using-dumpling.md). -## Deploy TiDB Lightning +The data migration process described in this document uses TiDB Lightning. The steps are as follows. -The data migration process described in this document uses TiDB Lightning. Before you start the migration, [deploy TiDB Lightning](/tidb-lightning/deploy-tidb-lightning.md). +## Step 1: Deploy TiDB Lightning + +Before you start the migration, [deploy TiDB Lightning](/tidb-lightning/deploy-tidb-lightning.md). > **Note:** > -> - If you choose the importer backend, you need to deploy tikv-importer along with TiDB Lightning. During the import process, the TiDB cluster cannot provide services. This mode imports data quickly, which is suitable for importing a large amount of data (above the TB level). -> - If you choose the TiDB backend, only TiDB Lightning is required. The cluster can provide services normally during the import. +> - If you choose the Importer-backend to import data, you need to deploy `tikv-importer` along with TiDB Lightning. During the import process, the TiDB cluster cannot provide services. This mode imports data quickly, which is suitable for importing a large amount of data (above the TB level). +> - If you choose the TiDB-backend, deploy TiDB Lightning only. The cluster can provide services normally during the import. > - For detailed differences between the two backend mode, see [TiDB Lightning Backend](/tidb-lightning/tidb-lightning-tidb-backend.md). -## Configure data source of TiDB Lightning +## Step 2: Configure data source of TiDB Lightning -This document takes the TiDB backend as an example. Add the `tidb-lightning.toml` configuration file and add the following major configurations in the file: +This document takes the TiDB-backend as an example. Create the `tidb-lightning.toml` configuration file and add the following major configurations in the file: 1. Set the `data-source-dir` under `[mydumper]` to the path of the MySQL SQL file. @@ -32,13 +34,13 @@ This document takes the TiDB backend as an example. Add the `tidb-lightning.toml > **Note:** > - > If a corresponding schema already exists in the downstream, set `no-schema=true` to skip the creation of schema. + > If a corresponding schema already exists in the downstream, set `no-schema=true` to skip the creation of the schema. -2. Add the TiDB configurations of the target cluster. +2. Add the configuration of the target TiDB cluster. ``` [tidb] - # the target cluster information. Fill in one address of tidb-server + # The target cluster information. Fill in one address of tidb-server. host = "172.16.31.1" port = 4000 user = "root" @@ -47,16 +49,16 @@ This document takes the TiDB backend as an example. Add the `tidb-lightning.toml For other configurations, see [TiDB Lightning Configuration](/tidb-lightning/tidb-lightning-configuration.md). -## Run TiDB Lightning to import data +## Step 3: Run TiDB Lightning to import data Run TiDB Lightning to start the import operation. If you start TiDB Lightning by using `nohup` directly in the command line, the program might exit because of the `SIGHUP` signal. Therefore, it is recommended to write `nohup` in a script. For example: -``` +```bash # !/bin/bash nohup ./tidb-lightning -config tidb-lightning.toml > nohup.out & ``` -When the import operation is started, you can view the progress by the following two ways: +When the import operation is started, view the progress by the following two ways: - `grep` the keyword `progress` in logs, which is updated every 5 minutes by default. - Access the monitoring dashboard. See [Monitor TiDB Lightning](/tidb-lightning/monitor-tidb-lightning.md) for details. From 26e490cbe9e889d28110a32b5097146fe8bdaf05 Mon Sep 17 00:00:00 2001 From: yikeke Date: Thu, 11 Jun 2020 15:44:36 +0800 Subject: [PATCH 3/4] Update TOC.md --- TOC.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/TOC.md b/TOC.md index e386ef62acb81..376ee73ad7c70 100644 --- a/TOC.md +++ b/TOC.md @@ -43,10 +43,12 @@ + Migrate + [Migration Tools Guide](/ecosystem-tool-user-guide.md) + Migrate from MySQL + + [Migrate from Mydumper Files](/migrate-from-mysql-mydumper-files.md) + [Migrate from Amazon Aurora MySQL Using DM](/migrate-from-aurora-mysql-database.md) + Migrate from CSV Files + [Use TiDB Lightning](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + [Use `LOAD DATA` Statement](/sql-statements/sql-statement-load-data.md) + + [Migrate from SQL Files](/migrate-from-mysql-mydumper-files.md) + Maintain + Upgrade + [Use TiUP (Recommended)](/upgrade-tidb-using-tiup.md) From 02bd523d100b5972d01a2498761534739333c5b5 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Thu, 11 Jun 2020 15:45:58 +0800 Subject: [PATCH 4/4] Update migrate-from-mysql-mydumper-files.md --- migrate-from-mysql-mydumper-files.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migrate-from-mysql-mydumper-files.md b/migrate-from-mysql-mydumper-files.md index c474c2ea022db..eb2b61e2a6444 100644 --- a/migrate-from-mysql-mydumper-files.md +++ b/migrate-from-mysql-mydumper-files.md @@ -28,7 +28,7 @@ This document takes the TiDB-backend as an example. Create the `tidb-lightning.t ``` [mydumper] - # data source directory + # Data source directory data-source-dir = "/data/export" ```