From b6ec5b17b434df914bade69ef52b34948561e590 Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 11:39:57 +0800 Subject: [PATCH 1/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index ebcedd41acf53..7e196022232c7 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -15,6 +15,10 @@ TiDB Binlog has the following features: * **Data replication:** replicate the data in the TiDB cluster to other databases * **Real-time backup and restoration:** back up the data in the TiDB cluster and restore the TiDB cluster when the cluster fails +> **Note:** +> +> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and cannot be used together. For details, see [Notes](#Notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. + ## TiDB Binlog architecture The TiDB Binlog architecture is as follows: @@ -50,15 +54,12 @@ The TiDB Binlog cluster is composed of Pump and Drainer. ## Notes -* You need to use TiDB v2.0.8-binlog, v2.1.0-rc.5 or a later version. Older versions of TiDB cluster are not compatible with the cluster version of TiDB Binlog. - -* Drainer supports replicating binlogs to MySQL, TiDB, Kafka or local files. If you need to replicate binlogs to other Drainer unsuppored destinations, you can set Drainer to replicate the binlog to Kafka and read the data in Kafka for customized processing according to binlog consumer protocol. See [Binlog Consumer Client User Guide](/tidb-binlog/binlog-consumer-client.md). - -* To use TiDB Binlog for recovering incremental data, set the config `db-type` to `file` (local files in the proto buffer format). Drainer converts the binlog to data in the specified [proto buffer format](https://github.com/pingcap/tidb-binlog/blob/master/proto/binlog.proto) and writes the data to local files. In this way, you can use [Reparo](/tidb-binlog/tidb-binlog-reparo.md) to recover data incrementally. +* TiDB Binlog is not compatible with the following features introduced in TiDB v5.0.0-rc and cannot be used together: - Pay attention to the value of `db-type`: + - [TiDB Clustered Index](/clustered-indexes.md#limitations): After TiDB Binlog is enabled, TiDB does not allow creating clustered indexes with non-single integer columns as primary keys; data insertion, deletion, and update for created clustered index tables will not be replicated downstream via TiDB Binlog. To replicate clustered index tables, you can use [TiCDC](/ticdc/ticdc-overview.md). + - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. + - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. - - If your TiDB version is earlier than 2.1.9, set `db-type="pb"`. - - If your TiDB version is 2.1.9 or later, set `db-type="file"` or `db-type="pb"`. +* TiDB Binlog is incompatible with the following features introduced in TiDB v4.0.7 and cannot be used together: -* If the downstream is MySQL, MariaDB, or another TiDB cluster, you can use [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) to verify the data after data replication. + - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together will cause correctness issues with inconsistent TiDB Binlog replication data. From 86db98dd262a0328c531e720e1cbdb87ace5302e Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 14:35:26 +0800 Subject: [PATCH 2/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index 7e196022232c7..b5b120f696318 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -17,7 +17,7 @@ TiDB Binlog has the following features: > **Note:** > -> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and cannot be used together. For details, see [Notes](#Notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. +> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and cannot be used together. For details, see [Notes](#notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. ## TiDB Binlog architecture @@ -60,6 +60,6 @@ The TiDB Binlog cluster is composed of Pump and Drainer. - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. -* TiDB Binlog is incompatible with the following features introduced in TiDB v4.0.7 and cannot be used together: +* TiDB Binlog is incompatible with the following feature introduced in TiDB v4.0.7 and cannot be used together: - - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together will cause correctness issues with inconsistent TiDB Binlog replication data. + - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together will cause correctness issues with inconsistent replication data of TiDB Binlog. From 710693a270bbf27b1af21a06be47b836ebdb3478 Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 16:28:29 +0800 Subject: [PATCH 3/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index b5b120f696318..146919cc421b0 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -17,7 +17,7 @@ TiDB Binlog has the following features: > **Note:** > -> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and cannot be used together. For details, see [Notes](#notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. +> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and they cannot be used together. For details, see [Notes](#notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. ## TiDB Binlog architecture @@ -54,9 +54,9 @@ The TiDB Binlog cluster is composed of Pump and Drainer. ## Notes -* TiDB Binlog is not compatible with the following features introduced in TiDB v5.0.0-rc and cannot be used together: +* TiDB Binlog is not compatible with the following features introduced in TiDB v5.0.0-rc and they cannot be used together: - - [TiDB Clustered Index](/clustered-indexes.md#limitations): After TiDB Binlog is enabled, TiDB does not allow creating clustered indexes with non-single integer columns as primary keys; data insertion, deletion, and update for created clustered index tables will not be replicated downstream via TiDB Binlog. To replicate clustered index tables, you can use [TiCDC](/ticdc/ticdc-overview.md). + - [TiDB Clustered Index](/clustered-indexes.md#limitations): After TiDB Binlog is enabled, TiDB does not allow creating clustered indexes with non-single integer columns as primary keys; data insertion, deletion, and update of the created clustered index tables will not be replicated downstream via TiDB Binlog. To replicate clustered index tables, you can use [TiCDC](/ticdc/ticdc-overview.md). - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. From d0d3e888484480d9dc0e3b2bdb53458e365b9fd3 Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 16:41:18 +0800 Subject: [PATCH 4/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index 146919cc421b0..776b2f7c6e557 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -62,4 +62,4 @@ The TiDB Binlog cluster is composed of Pump and Drainer. * TiDB Binlog is incompatible with the following feature introduced in TiDB v4.0.7 and cannot be used together: - - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together will cause correctness issues with inconsistent replication data of TiDB Binlog. + - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together might cause the issue that TiDB Binlog replicates data inconsistently. \ No newline at end of file From b1a6a51448724e29b70e1ecbf0a8e0c085ef0dd0 Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 16:51:40 +0800 Subject: [PATCH 5/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index 776b2f7c6e557..e56b51157a9bd 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -60,6 +60,6 @@ The TiDB Binlog cluster is composed of Pump and Drainer. - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. -* TiDB Binlog is incompatible with the following feature introduced in TiDB v4.0.7 and cannot be used together: +* TiDB Binlog is incompatible with the following feature introduced in TiDB v4.0.7 and they cannot be used together: - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together might cause the issue that TiDB Binlog replicates data inconsistently. \ No newline at end of file From 67a76ecb1f97830f11093e9d6369f1a11549cca3 Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 17:42:15 +0800 Subject: [PATCH 6/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index e56b51157a9bd..63f76c0d9953f 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -17,7 +17,7 @@ TiDB Binlog has the following features: > **Note:** > -> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and they cannot be used together. For details, see [Notes](#notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. +> TiDB Binlog is not compatible with some features introduced in TiDB v5.0.0-rc and they cannot be used together. For details, see [Notes](#notes). It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead of TiDB Binlog. ## TiDB Binlog architecture @@ -57,9 +57,21 @@ The TiDB Binlog cluster is composed of Pump and Drainer. * TiDB Binlog is not compatible with the following features introduced in TiDB v5.0.0-rc and they cannot be used together: - [TiDB Clustered Index](/clustered-indexes.md#limitations): After TiDB Binlog is enabled, TiDB does not allow creating clustered indexes with non-single integer columns as primary keys; data insertion, deletion, and update of the created clustered index tables will not be replicated downstream via TiDB Binlog. To replicate clustered index tables, you can use [TiCDC](/ticdc/ticdc-overview.md). - - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. - - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. + - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead of TiDB Binlog. + - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead of TiDB Binlog. * TiDB Binlog is incompatible with the following feature introduced in TiDB v4.0.7 and they cannot be used together: - - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together might cause the issue that TiDB Binlog replicates data inconsistently. \ No newline at end of file + - TiDB system variable [tidb_enable_amend_pessimistic_txn](/system-variables.md#tidb_enable_amend_pessimistic_txn-new-in-v407): The two features have compatibility issues. Using them together might cause the issue that TiDB Binlog replicates data inconsistently. + +* Drainer supports replicating binlogs to MySQL, TiDB, Kafka or local files. If you need to replicate binlogs to other Drainer unsuppored destinations, you can set Drainer to replicate the binlog to Kafka and read the data in Kafka for customized processing according to binlog consumer protocol. See [Binlog Consumer Client User Guide](/tidb-binlog/binlog-consumer-client.md). + +* To use TiDB Binlog for recovering incremental data, set the config `db-type` to `file` (local files in the proto buffer format). Drainer converts the binlog to data in the specified [proto buffer format](https://github.com/pingcap/tidb-binlog/blob/master/proto/binlog.proto) and writes the data to local files. In this way, you can use [Reparo](/tidb-binlog/tidb-binlog-reparo.md) to recover data incrementally. + + Pay attention to the value of `db-type`: + + - If your TiDB version is earlier than 2.1.9, set `db-type="pb"`. + - If your TiDB version is 2.1.9 or later, set `db-type="file"` or `db-type="pb"`. + +* If your TiDB version is earlier than 2.1.9, set `db-type="pb"`. + - If your TiDB version is 2.1.9 or later, set `db-type="file"` or `db-type="pb"`. \ No newline at end of file From 620e0c9b4342562d89c50cafcc8545376414870a Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 17:43:53 +0800 Subject: [PATCH 7/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index 63f76c0d9953f..52c8a84c01887 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -73,5 +73,4 @@ The TiDB Binlog cluster is composed of Pump and Drainer. - If your TiDB version is earlier than 2.1.9, set `db-type="pb"`. - If your TiDB version is 2.1.9 or later, set `db-type="file"` or `db-type="pb"`. -* If your TiDB version is earlier than 2.1.9, set `db-type="pb"`. - - If your TiDB version is 2.1.9 or later, set `db-type="file"` or `db-type="pb"`. \ No newline at end of file +* If the downstream is MySQL, MariaDB, or another TiDB cluster, you can use [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) to verify the data after data replication. \ No newline at end of file From adcc2834e7c0915ee510a5d8e56563c921894f71 Mon Sep 17 00:00:00 2001 From: Joyinqin Date: Fri, 26 Mar 2021 18:18:14 +0800 Subject: [PATCH 8/8] Update tidb-binlog-overview.md --- tidb-binlog/tidb-binlog-overview.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tidb-binlog/tidb-binlog-overview.md b/tidb-binlog/tidb-binlog-overview.md index 52c8a84c01887..518deb569ad5a 100644 --- a/tidb-binlog/tidb-binlog-overview.md +++ b/tidb-binlog/tidb-binlog-overview.md @@ -54,11 +54,11 @@ The TiDB Binlog cluster is composed of Pump and Drainer. ## Notes -* TiDB Binlog is not compatible with the following features introduced in TiDB v5.0.0-rc and they cannot be used together: +* TiDB Binlog is not compatible with the following features introduced in TiDB v5.0.0-rc and they cannot be used together. It is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead of TiDB Binlog: - - [TiDB Clustered Index](/clustered-indexes.md#limitations): After TiDB Binlog is enabled, TiDB does not allow creating clustered indexes with non-single integer columns as primary keys; data insertion, deletion, and update of the created clustered index tables will not be replicated downstream via TiDB Binlog. To replicate clustered index tables, you can use [TiCDC](/ticdc/ticdc-overview.md). - - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead of TiDB Binlog. - - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v500-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. To improve performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead of TiDB Binlog. + - [TiDB Clustered Index](/clustered-indexes.md#limitations): After TiDB Binlog is enabled, TiDB does not allow creating clustered indexes with non-single integer columns as primary keys; data insertion, deletion, and update of the created clustered index tables will not be replicated downstream via TiDB Binlog. + - TiDB system variable [tidb_enable_async_commit](/system-variables.md#tidb_enable_async_commit-new-in-v50-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. + - TiDB system variable [tidb_enable_1pc](/system-variables.md#tidb_enable_1pc-new-in-v50-rc): After TiDB Binlog is enabled, performance cannot be improved by enabling this option. * TiDB Binlog is incompatible with the following feature introduced in TiDB v4.0.7 and they cannot be used together: @@ -73,4 +73,4 @@ The TiDB Binlog cluster is composed of Pump and Drainer. - If your TiDB version is earlier than 2.1.9, set `db-type="pb"`. - If your TiDB version is 2.1.9 or later, set `db-type="file"` or `db-type="pb"`. -* If the downstream is MySQL, MariaDB, or another TiDB cluster, you can use [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) to verify the data after data replication. \ No newline at end of file +* If the downstream is MySQL, MariaDB, or another TiDB cluster, you can use [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) to verify the data after data replication.