From 31e0b10e6d71798338b3885a75051f6113dc6018 Mon Sep 17 00:00:00 2001 From: yikeke Date: Wed, 25 Mar 2020 17:35:07 +0800 Subject: [PATCH 1/8] tools: update user guide --- reference/tools/user-guide.md | 293 ++++++++++++++++++++++------------ 1 file changed, 190 insertions(+), 103 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index e28f0efe0deb1..792c88b8dbe15 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -6,184 +6,271 @@ aliases: ['/docs/dev/how-to/migrate/from-mysql/','/docs/dev/how-to/migrate/incre # TiDB Ecosystem Tools User Guide -Currently, TiDB has multiple ecosystem tools. Some of them have overlapping functionality, and some are different versions of the same tool. This document introduces each of these tools, illustrates their relationship, and describes when to use which tool for each TiDB version. +The TiDB ecosystem has a wealth of tools for data migration, backup & restore for users with different use cases to choose from. -## TiDB ecosystem tools overview +- Some of the functionalities of these tools may overlap. For example, TiDB Loader, TiDB Lightning and TiDB DM can all do full data loading. +- Some of the tools may have evolved. For example, TiDB Binlog will be evolved to CDC (Change Data Capture). +- Some of the tools are designed to support specific TiDB versions and the others might be deprecated as user requirement change. -TiDB ecosystem tools can be divided into: +This guide is specifically designed to help you better understand these tools and therefore make an informed decision while choosing these tools to support your business. -- Data import tools, including full import tools, backup and restore tools, incremental import tools, and so forth. -- Data export tools, including full export tools. incremental export tools, and so forth. +## Data import (restore or data sync) -The two types of tools are discussed in detail below. +### Full data import tools -### Data import tools +#### TiDB Lightning -#### Full import tool TiDB Lightning +[TiDB Lightning](/reference/tools/tidb-lightning/overview.md) (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source. -[TiDB Lightning](/reference/tools/tidb-lightning/overview.md) is a tool used for fast full import of data into a TiDB cluster. +You can use it in the following two scenarios: -> **Note:** -> -> When you import data into TiDB using TiDB Lightning, there are two modes: -> -> - The default mode: Use `tikv-importer` as the backend. In this mode, the cluster can not provide normal services during the data import process. It is used when you import large amounts (TBs) of data. -> - The second mode: Use `TiDB` as the backend (similar to Loader). The import speed is slower than that in the default mode. However, the second mode supports online import. +- Import large amounts of new data quickly +- Backup and restore all the data + +TiDB Lightning supports two back ends: “Importer” and “TiDB”. It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows: + +1. The default one is [`Importer` back end](/reference/tools/tidb-lightning/overview.md). When using `Importer` as the back end, the cluster cannot provide normal services during the import process. It is used for a large amount of data importing (TB). +2. The second one is [`TiDB` back end](/reference/tools/tidb-lightning/tidb-backend.md) (just work as [Loader](#tidb-loader-to-be-deprecated)). It is much slower than `Importer` back end model. But the cluster could serve the application during the import process. It is used to handle tens/hundreds of GB data. The following are the basics of TiDB Lightning: -- Input: - - Files output by Mydumper; - - CSV files. -- Compatibility: Compatible with TiDB v2.1 and later versions. -- Kubernetes: Supported. See [Quickly restore data into a TiDB cluster in Kubernetes using TiDB Lightning](/tidb-in-kubernetes/maintain/lightning.md). +- Input data source: + - The output file of Mydumper + - CSV file +- Supported TiDB versions: v2.1 or later +- Kubernetes support: Yes. See [Quickly restore data into a TiDB cluster in Kubernetes using TiDB Lightning](/tidb-in-kubernetes/maintain/lightning.md) for details. -#### Backup and restore tool BR +#### BR (beta) -[BR](/reference/tools/br/br.md) is a command-line tool used for distributed data backup and restoration for a TiDB cluster. Compared with Mydumper and Loader, BR allows you to finish backup and restore tasks with greater efficiency in scenarios of huge data volume. +[BR](/reference/tools/br/br.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume. The following are the basics of BR: -- [Types of backup files](/reference/tools/br/br.md#types-of-backup-files): The SST file and the `backupmeta` file. -- Compatibility: Compatible with TiDB v3.1 and v4.0 versions. -- Kubernetes: Supported. Relevant documents are on the way. +- Input data source: The output file of BR +- Supported TiDB versions: v3.1 or later +- Kubernetes support: Yes. The document is WIP. -#### Incremental and full import tool TiDB Data Migration +#### TiDB Loader (to be deprecated) -[TiDB Data Migration (DM)](/reference/tools/data-migration/overview.md) is an tool used for data migration from MySQL/MariaDB into TiDB. It supports both the full and incremental data replication. +> **Note:** +> +> TiDB Loader is to be deprecated and replaced with [Lightning](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-back-end). -The following are the basics of DM: +[TiDB Loader](/reference/tools/loader.md) is a lightweight full-data importing tool for TiDB. It reads the output file of Mydumper and loads the data into TiDB. -- Input: Full data and binlog data of MySQL/MariaDB. -- Output: SQL statements written to TiDB. -- Compatibility: Compatible with all TiDB versions. -- Kubernetes: In development. +The following are the basics of Loader: -#### Full import tool Loader (Stop maintenance, not recommended) +- Input data source: Mydumper’s output file +- Supported TiDB versions: all versions +- Kubernetes support: Yes. See [Backup and restore](/tidb-in-kubernetes/maintain/backup-and-restore.md) for details. -[Loader](/reference/tools/loader.md) is a lightweight full data import tool. Data is imported into TiDB in the form of SQL statements. Currently, this tool is gradually replaced by [TiDB Lightning](#full-import-tool-tidb-lightning), see [TiDB Lightning TiDB-backend Document](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend). +### Incremental data import tools -The following are the basics of Loader: +#### TiDB Data Migration + +[TiDB Data Migration](/reference/tools/data-migration/overview.md) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. + +For the full data migration, it uses an embedded Loader. For the incremental data migration, it uses Syncer as its kernel. + +The following are the basics of DM: -- Input: Files output by Mydumper. -- Output: SQL statements written to TiDB. -- Compatibility: Compatible with all TiDB versions. -- Kubernetes: Supported. See [Backup and restore](/tidb-in-kubernetes/maintain/backup-and-restore.md). +- Input data source: MySQL/MariaDB master host/port +- Supported TiDB versions: all versions +- Kubernetes support: No, under development (the estimated time is 2020 Q2) -#### Incremental import tool Syncer (Stop maintenance, not recommended) +#### Syncer (deprecated) -[Syncer](/reference/tools/syncer.md) is a tool used for incremental import of real-time binlog data from MySQL/MariaDB into TiDB. It is recommended to use [TiDB Data Migration](#Incremental-import-tool-tidb-data-migration) to replace Syncer. +[Syncer](/reference/tools/syncer.md) is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use [TiDB Data Migration](#tidb-data-migration) to replace Syncer. The following are the basics of Syncer: -- Input: Binlog data of MySQL/MariaDB. -- Output: SQL statements written to TiDB. -- Compatibility: Compatible with all TiDB versions. -- Kubernetes: Not supported. +- Input data source: MySQL/MariaDB binlog service +- Supported TiDB versions: all versions +- Kubernetes support: No + +## Data export (backup) -### Data export tools +### Full data export tools -#### Full export tool Mydumper +#### Mydumper -[Mydumper](/reference/tools/mydumper.md) is a MySQL community tool used for full logical backups of MySQL that also works with TiDB. +[Mydumper](/reference/tools/mydumper.md) is a tool to create a logical full backup for TiDB. The following are the basics of Mydumper: -- Input: MySQL/TiDB clusters. -- Output: SQL files. -- Compatibility: Compatible with all TiDB versions. -- Kubernetes: Supported. See [Backup and Restore](/tidb-in-kubernetes/maintain/backup-and-restore.md). +- Input/Output + - Input: TiDB/MySQL host:port + - Output: schema and insert statements file +- Supported TiDB versions: all versions +- Kubernetes support: Yes. See [Backup and Restore](/tidb-in-kubernetes/maintain/backup-and-restore.md) for details. -#### Full export tool TiDB Binlog +#### BR (beta) -[TiDB Binlog](/reference/tidb-binlog/overview.md) is a tool used to collect binlog data from TiDB. It provides near real-time backup and replication to downstream platforms. +[BR](/reference/tools/br/br.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume. -The following are the basics of TiDB Binlog: +The following are the basics of BR: + +- Input/Output + - Input: TiDB cluster + - Output: Full backup file +- Supported TiDB versions: v3.1 or v4.0 +- Kubernetes support: Yes. The document is WIP. -- Input: TiDB clusters. -- Output: MySQL, TiDB, Kafka or incremental backup files. -- Compatibility: Compatible with TiDB v2.1 and later versions. -- Kubernetes: Supported. See [TiDB Binlog Cluster Operations](/tidb-in-kubernetes/maintain/tidb-binlog.md) and [TiDB Binlog Drainer Configurations in Kubernetes](/tidb-in-kubernetes/reference/configuration/tidb-drainer.md). +### Incremental data export tools -## Tools development roadmap +#### TiDB Binlog -To help you understand the relationships between the above tools, here is a brief introduction to TiDB ecosystem tools development roadmap. +[TiDB Binlog](/reference/tidb-binlog/overview.md) is a system that collects binlog for TiDB clusters and provides tools for near real-time sync and backup. -### TiDB backup and restore +The following are the basics of TiDB Binlog: -Mydumper and Loader -> BR: +- Input/Output: + - Input: TiDB Cluster + - Output: MySQL, TiDB, Kafka or incremental backup files +- Supported TiDB versions: v2.1 or later +- Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](/tidb-in-kubernetes/maintain/tidb-binlog.md) and [TiDB Binlog Drainer Configurations in Kubernetes](/tidb-in-kubernetes/reference/configuration/tidb-drainer.md) for details. + +#### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0) + +CDC (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in transactional order. + +- Input/Output: + - Input: TiDB Cluster + - Output: MySQL, TiDB, Kafka or incremental backup files +- Supported TiDB versions: v4.0 +- Document: WIP +- Kubernetes support: On the development road map, ETA Q2 2020 + +## Recommended tools for TiDB versions + +### Recommended tools for TiDB 3.0 or earlier + +- MySQL full data backup: use Mydumper +- MySQL full data import to TiDB: + - TB scale: use TiDB Lightning + - Sub-TB scale: Use DM +- MySQL incremental data sync to TiDB: use DM +- TiDB full data backup: Use Mydumper +- TiDB full data restore: + - TB scale: use TiDB Lightning + - Sub-TB scale: Use DM +- TiDB incremental backup & restore: use TiDB-Binlog + +### Recommended tools for TiDB 3.1 + +- MySQL full data backup: use Mydumper +- MySQL full data import to TiDB: + - TB scale: use TiDB Lightning + - Sub-TB scale: use DM +- MySQL incremental data sync to TiDB: use DM +- TiDB full data backup: use BR +- TiDB full data restore: use BR +- TiDB incremental backup & restore: use TiDB-Binlog + +### Recommended tools for TiDB 4.0 + +- MySQL full data backup: use Mydumper +- MySQL full data import to TiDB: + - TB scale: use TiDB Lightning + - Sub-TB scale: use DM +- MySQL incremental data sync to TiDB: use DM +- TiDB full data backup: use BR +- TiDB full data restore: use BR +- TiDB incremental backup & restore: use CDC + +## Tools evolution roadmap + +- TiDB Full Data Backup: + - Mydumper -> BR + - Mydumper -> [dumpling](https://github.com/pingcap/dumpling) (under development, replace Lighting in lightweight scenarios) +- TiDB Full Data Restore: + - Loader -> Lightning -> BR +- MySQL Data Migration: + - Mydumper/Loader + Syncer -> Loader + DM -> Lightning + DM -> DM (in the next step, we will integrate Lightning into DM) +- TiDB Incremental Data Migration: + - TiDB Binlog -> CDC + +## Full-path data migration solution for TiDB 3.0, 3.1 and 4.0 + +TiDB 3.0 is the recommended version and is also the most widely adopted version. In addition, TiDB 3.1 and 4.0 will be released this year. The following sections will cover how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data. + +### For TiDB 3.0 -Mydumper and Loader are inefficient since they back up and restore data on the logical level. BR is much more efficient because it takes advantage of TiDB features for backup and restore tasks. BR can be applied in huge data volume scenarios. +#### Migrating MySQL data to TiDB -### TiDB full data restore +If the MySQL data volume is in TBs: -Loader -> TiDB Lightning: +- Use Mydumper to export MySQL full data as a backup +- Use Lightning to import the full MySQL backup data into TiDB cluster +- Use DM to replicate incremental MySQL data to TiDB -Loader is inefficient since it performs full data restoration using SQL. TiDB Lightning imports data into TiKV directly, so it is much more efficient and can be used for fast full import of large amounts (more than TBs) of data into a new TiDB cluster. +If the MySQL data volume is in GBs: -TiDB Lightning also integrates the logical data import function of Loader and supports online data import. For details, see [TiDB Lightning TiDB-backend Document](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend). +- Use DM to migrate MySQL data to TiDB for both full and incremental data import -### MySQL data migration +#### Data replication between TiDB/MySQL clusters -- Mydumper, Loader and Syncer -> DM: +You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. - It is tedious to migrate MySQL data to TiDB using Mydumper, Loader, and Syncer. DM provides an integrated data migration approach that improves usability. DM can be also used to merge the sharded schemas and tables. +#### Full backup and restore of the data in TiDB/MySQL cluster -- Loader -> TiDB Lightning: +- Use the Mydumper tool for full data backup +- Use the Lightning tool for full data restore - TiDB Lightning integrates the logical data import function of Loader. See [TiDB Lightning TiDB-backend document](/reference/tools/tidb-lightning/tidb-backend.md#migrating-from-loader-to-tidb-lightning-tidb-backend) for details. It is used to perform full data restoration. +### For TiDB 3.1 -## Data migration solutions +#### Migrating MySQL data to TiDB -For TiDB 2.1, 3.0, and 3.1 versions, this section introduces data migration solutions in typical application scenarios. +If the MySQL data volume is in TBs: -### Full link data migration solutions for v3.0 +- Use Mydumper to export MySQL full data as a backup +- Use Lightning to import the full MySQL backup data into TiDB cluster +- Use DM to replicate incremental MySQL data to TiDB -#### Migrating MySQL data to TiDB +If the MySQL data volume is in GBs: -If the volume is more than TBs of data, the recommended migration steps are: +- Use DM to migrate MySQL data to TiDB for both full and incremental data import -1. Export full MySQL data using Mydumper; -2. Import full backup data from MySQL into a TiDB cluster using TiDB Lightning; -3. Replicate the incremental data of MySQL into TiDB. +#### Data replication between TiDB/MySQL clusters -If the volume is less than TBs of data, it is recommended to migrate MySQL data to TiDB using DM (the migrating process includes full data import and incremental data replication). +You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. -#### Replication of TiDB cluster data +#### Full backup and restore of the data in TiDB/MySQL cluster -It is recommended that you use TiDB Binlog to replicate TiDB data to downstream TiDB/MySQL. +To restore data to TiDB cluster: -#### Full backup and restore of TiDB cluster data +- Use the BR tool for both full data backup and full data restore -The recommended steps are: +To restore data to MySQL cluster: -1. Back up full data using Mydumper; -2. Restore full data into TiDB/MySQL using TiDB Lightning. +- Use the Mydumper tool for full data backup +- Use the Lightning tool for full data restore -### Full link data migration solutions for v3.1 +### For TiDB 4.0 #### Migrating MySQL data to TiDB -If the volume is more than TBs of data, the recommended migration steps are: +If the MySQL data volume is in TBs: + +- Use Mydumper to export MySQL full data as a backup +- Use Lightning to import full MySQL backup data into TiDB cluster +- Use DM to replicate incremental MySQL data to TiDB -1. Export full MySQL data using Mydumper; -2. Import full backup data from MySQL into a TiDB cluster using TiDB Lightning; -3. Replicate the incremental data of MySQL into TiDB. +If the MySQL data volume is in GBs: -If the volume is less than TBs of data, it is recommended to migrate MySQL data to TiDB using DM (the migrating process includes full data import and incremental data replication). +- Use DM to migrate MySQL data to TiDB for both full and incremental data import -#### Replication of TiDB cluster data +#### Data replication between TiDB/MySQL clusters -It is recommended that you use TiDB Binlog to replicate TiDB data to downstream TiDB/MySQL. +You can use the TiDB CDC tool to replicate data between TiDB clusters. You can also use the CDC tool to replicate data to the downstream MySQL cluster. -#### Full backup and restore of TiDB cluster data +#### Full backup and restore of the data in TiDB/MySQL cluster -- Restore to TiDB +To restore data to TiDB cluster: - - Back up full data using BR; - - Restore full data using BR. +- Use the BR tool for both full data backup and full data restore -- Restore to MySQL +To restore data to MySQL cluster: - - Back up full data using Mydumper; - - Restore full data using TiDB Lightning. +- Use the Mydumper tool for full data backup +- Use the Lightning tool for full data restore From a95dff5c40f514d3fbe19fd24dec27ddc712e552 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Wed, 25 Mar 2020 18:23:28 +0800 Subject: [PATCH 2/8] Apply suggestions from code review Co-Authored-By: Lilian Lee --- reference/tools/user-guide.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index 792c88b8dbe15..b094216b7ecc5 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -8,13 +8,13 @@ aliases: ['/docs/dev/how-to/migrate/from-mysql/','/docs/dev/how-to/migrate/incre The TiDB ecosystem has a wealth of tools for data migration, backup & restore for users with different use cases to choose from. -- Some of the functionalities of these tools may overlap. For example, TiDB Loader, TiDB Lightning and TiDB DM can all do full data loading. -- Some of the tools may have evolved. For example, TiDB Binlog will be evolved to CDC (Change Data Capture). -- Some of the tools are designed to support specific TiDB versions and the others might be deprecated as user requirement change. +- Some of the functionalities of these tools might overlap. For example, TiDB Loader, TiDB Lightning and TiDB DM can all do full data loading. +- Some of the tools might have evolved. For example, TiDB Binlog will be evolved to CDC (Change Data Capture). +- Some of the tools are designed to support specific TiDB versions and the others might be deprecated as user requirements change. This guide is specifically designed to help you better understand these tools and therefore make an informed decision while choosing these tools to support your business. -## Data import (restore or data sync) +## Data import (restore or data replication) ### Full data import tools @@ -192,7 +192,7 @@ CDC (Change Data Capture) is a system that collects changelog for key value pair ## Full-path data migration solution for TiDB 3.0, 3.1 and 4.0 -TiDB 3.0 is the recommended version and is also the most widely adopted version. In addition, TiDB 3.1 and 4.0 will be released this year. The following sections will cover how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data. +TiDB 3.0 is the recommended version and is also the most widely adopted version. In addition, TiDB 3.1 GA and 4.0 GA will be released this year. The following sections will cover how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data. ### For TiDB 3.0 @@ -212,7 +212,7 @@ If the MySQL data volume is in GBs: You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. -#### Full backup and restore of the data in TiDB/MySQL cluster +#### Full backup and restore of the data in TiDB/MySQL clusters - Use the Mydumper tool for full data backup - Use the Lightning tool for full data restore @@ -235,13 +235,13 @@ If the MySQL data volume is in GBs: You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster. -#### Full backup and restore of the data in TiDB/MySQL cluster +#### Full backup and restore of the data in TiDB/MySQL clusters -To restore data to TiDB cluster: +To restore data to a TiDB cluster: - Use the BR tool for both full data backup and full data restore -To restore data to MySQL cluster: +To restore data to a MySQL cluster: - Use the Mydumper tool for full data backup - Use the Lightning tool for full data restore @@ -264,13 +264,13 @@ If the MySQL data volume is in GBs: You can use the TiDB CDC tool to replicate data between TiDB clusters. You can also use the CDC tool to replicate data to the downstream MySQL cluster. -#### Full backup and restore of the data in TiDB/MySQL cluster +#### Full backup and restore of the data in TiDB/MySQL clusters -To restore data to TiDB cluster: +To restore data to a TiDB cluster: - Use the BR tool for both full data backup and full data restore -To restore data to MySQL cluster: +To restore data to a MySQL cluster: - Use the Mydumper tool for full data backup - Use the Lightning tool for full data restore From 1ff929472e442af54793a1d4b0071a969b2cdff2 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Thu, 26 Mar 2020 11:57:05 +0800 Subject: [PATCH 3/8] Apply suggestions from code review Co-Authored-By: WangXiangUSTC --- reference/tools/user-guide.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index b094216b7ecc5..e0d4da5706346 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -27,7 +27,7 @@ You can use it in the following two scenarios: - Import large amounts of new data quickly - Backup and restore all the data -TiDB Lightning supports two back ends: “Importer” and “TiDB”. It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows: +TiDB Lightning supports two back ends: "Importer" and "TiDB". It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows: 1. The default one is [`Importer` back end](/reference/tools/tidb-lightning/overview.md). When using `Importer` as the back end, the cluster cannot provide normal services during the import process. It is used for a large amount of data importing (TB). 2. The second one is [`TiDB` back end](/reference/tools/tidb-lightning/tidb-backend.md) (just work as [Loader](#tidb-loader-to-be-deprecated)). It is much slower than `Importer` back end model. But the cluster could serve the application during the import process. It is used to handle tens/hundreds of GB data. @@ -132,13 +132,12 @@ The following are the basics of TiDB Binlog: #### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0) -CDC (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in transactional order. +[CDC](/reference/tools/ticdc/overview.md) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in transactional order. - Input/Output: - Input: TiDB Cluster - Output: MySQL, TiDB, Kafka or incremental backup files - Supported TiDB versions: v4.0 -- Document: WIP - Kubernetes support: On the development road map, ETA Q2 2020 ## Recommended tools for TiDB versions From 5b8197cc65a025cebc423c1071029865c12d6119 Mon Sep 17 00:00:00 2001 From: yikeke Date: Mon, 30 Mar 2020 19:57:02 +0800 Subject: [PATCH 4/8] address comments from ian and wangxiang --- reference/tools/user-guide.md | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index e0d4da5706346..f3d078938d837 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -20,12 +20,7 @@ This guide is specifically designed to help you better understand these tools an #### TiDB Lightning -[TiDB Lightning](/reference/tools/tidb-lightning/overview.md) (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source. - -You can use it in the following two scenarios: - -- Import large amounts of new data quickly -- Backup and restore all the data +[TiDB Lightning](/reference/tools/tidb-lightning/overview.md) (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source. TiDB Lightning supports two back ends: "Importer" and "TiDB". It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows: @@ -70,7 +65,7 @@ The following are the basics of Loader: [TiDB Data Migration](/reference/tools/data-migration/overview.md) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. -For the full data migration, it uses an embedded Loader. For the incremental data migration, it uses Syncer as its kernel. +For the full data migration, it uses an embedded Loader and an embedded Mydumper. For the incremental data migration, it uses Syncer as its kernel. The following are the basics of DM: @@ -132,7 +127,7 @@ The following are the basics of TiDB Binlog: #### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0) -[CDC](/reference/tools/ticdc/overview.md) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in transactional order. +[CDC](/reference/tools/ticdc/overview.md) (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in row changed order. - Input/Output: - Input: TiDB Cluster @@ -147,12 +142,12 @@ The following are the basics of TiDB Binlog: - MySQL full data backup: use Mydumper - MySQL full data import to TiDB: - TB scale: use TiDB Lightning - - Sub-TB scale: Use DM + - Sub-TB scale: use DM - MySQL incremental data sync to TiDB: use DM -- TiDB full data backup: Use Mydumper +- TiDB full data backup: use Mydumper - TiDB full data restore: - TB scale: use TiDB Lightning - - Sub-TB scale: Use DM + - Sub-TB scale: use TiDB Lightning - TiDB incremental backup & restore: use TiDB-Binlog ### Recommended tools for TiDB 3.1 @@ -214,7 +209,7 @@ You can use TiDB Binlog to replicate data between TiDB clusters. You can also us #### Full backup and restore of the data in TiDB/MySQL clusters - Use the Mydumper tool for full data backup -- Use the Lightning tool for full data restore +- Use the Lightning tool with `tidb` backend for full data restore ### For TiDB 3.1 @@ -243,7 +238,7 @@ To restore data to a TiDB cluster: To restore data to a MySQL cluster: - Use the Mydumper tool for full data backup -- Use the Lightning tool for full data restore +- Use the Lightning tool with `tidb` backend for full data restore ### For TiDB 4.0 @@ -272,4 +267,4 @@ To restore data to a TiDB cluster: To restore data to a MySQL cluster: - Use the Mydumper tool for full data backup -- Use the Lightning tool for full data restore +- Use the Lightning tool with `tidb` backend for full data restore From 654b731f990e9e0c1a216dbe66ed87020f0cfc31 Mon Sep 17 00:00:00 2001 From: Keke Yi <40977455+yikeke@users.noreply.github.com> Date: Mon, 30 Mar 2020 19:58:45 +0800 Subject: [PATCH 5/8] Apply suggestions from code review Co-Authored-By: Ian --- reference/tools/user-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index f3d078938d837..3768aec214a3e 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -37,7 +37,7 @@ The following are the basics of TiDB Lightning: #### BR (beta) -[BR](/reference/tools/br/br.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume. +[BR](/reference/tools/br/br.md) (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/Loader/Lightning, BR is more suitable for scenarios of huge data volume. The following are the basics of BR: @@ -180,7 +180,7 @@ The following are the basics of TiDB Binlog: - TiDB Full Data Restore: - Loader -> Lightning -> BR - MySQL Data Migration: - - Mydumper/Loader + Syncer -> Loader + DM -> Lightning + DM -> DM (in the next step, we will integrate Lightning into DM) + - Mydumper/Loader + Syncer -> DM (in the next step, we will integrate Lightning into DM) - TiDB Incremental Data Migration: - TiDB Binlog -> CDC From 109c1cb6105161968d9f33c0e934590adafd45d7 Mon Sep 17 00:00:00 2001 From: yikeke Date: Wed, 1 Apr 2020 15:19:06 +0800 Subject: [PATCH 6/8] fix a dead link --- reference/tools/user-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index 3c9374f6d85a9..507f08868519c 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -63,7 +63,7 @@ The following are the basics of Loader: #### TiDB Data Migration -[TiDB Data Migration](/reference/tools/data-migration/overview.md) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. +[TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. For the full data migration, it uses an embedded Loader and an embedded Mydumper. For the incremental data migration, it uses Syncer as its kernel. From a93645f8559c5bb635767c6058f2daa810e25546 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Wed, 1 Apr 2020 15:26:33 +0800 Subject: [PATCH 7/8] add new section for dm --- reference/tools/user-guide.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index 507f08868519c..f59a0459e3c32 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -61,6 +61,18 @@ The following are the basics of Loader: ### Incremental data import tools +#### Syncer (deprecated) + +[Syncer](/reference/tools/syncer.md) is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use [TiDB Data Migration](#tidb-data-migration) to replace Syncer. + +The following are the basics of Syncer: + +- Input data source: MySQL/MariaDB binlog service +- Supported TiDB versions: all versions +- Kubernetes support: No + +### Full and incremental data import tools + #### TiDB Data Migration [TiDB Data Migration](https://pingcap.com/docs/tidb-data-migration/stable/) (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process. @@ -73,16 +85,6 @@ The following are the basics of DM: - Supported TiDB versions: all versions - Kubernetes support: No, under development (the estimated time is 2020 Q2) -#### Syncer (deprecated) - -[Syncer](/reference/tools/syncer.md) is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use [TiDB Data Migration](#tidb-data-migration) to replace Syncer. - -The following are the basics of Syncer: - -- Input data source: MySQL/MariaDB binlog service -- Supported TiDB versions: all versions -- Kubernetes support: No - ## Data export (backup) ### Full data export tools From fb46f7fe4572f203cd0804a1144b48d3567a141c Mon Sep 17 00:00:00 2001 From: yikeke Date: Wed, 1 Apr 2020 16:29:20 +0800 Subject: [PATCH 8/8] fix 3 links --- reference/tools/user-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/tools/user-guide.md b/reference/tools/user-guide.md index f59a0459e3c32..22c9404eb064a 100644 --- a/reference/tools/user-guide.md +++ b/reference/tools/user-guide.md @@ -57,7 +57,7 @@ The following are the basics of Loader: - Input data source: Mydumper’s output file - Supported TiDB versions: all versions -- Kubernetes support: Yes. See [Backup and restore](/tidb-in-kubernetes/maintain/backup-and-restore.md) for details. +- Kubernetes support: Yes. See [Backup and restore](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-and-restore-using-helm-charts/) for details. ### Incremental data import tools @@ -125,7 +125,7 @@ The following are the basics of TiDB Binlog: - Input: TiDB Cluster - Output: MySQL, TiDB, Kafka or incremental backup files - Supported TiDB versions: v2.1 or later -- Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](/tidb-in-kubernetes/maintain/tidb-binlog.md) and [TiDB Binlog Drainer Configurations in Kubernetes](/tidb-in-kubernetes/reference/configuration/tidb-drainer.md) for details. +- Kubernetes support: Yes. See [TiDB Binlog Cluster Operations](https://pingcap.com/docs/tidb-in-kubernetes/stable/deploy-tidb-binlog/) and [TiDB Binlog Drainer Configurations in Kubernetes](https://pingcap.com/docs/tidb-in-kubernetes/stable/configure-tidb-binlog-drainer/) for details. #### CDC (Beta, under development, ETA May/June 2020 with TiDB 4.0)