From 1991ebc91c592e48b3a8c990eff18f78bb2efde5 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 12 Apr 2021 17:52:48 +0800 Subject: [PATCH 1/6] br:tidy up the incompatible issues in using BR tools --- br/backup-and-restore-tool.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index edac616c28e10..5eb383c50b60a 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -104,7 +104,7 @@ After the restoration operation is completed, BR performs a checksum calculation ## Deploy and use BR -## Recommended deployment configuration +### Recommended deployment configuration - It is recommended that you deploy BR on the PD node. - It is recommended that you mount a high-performance SSD to BR nodes and all TiKV nodes. A 10-gigabit network card is recommended. Otherwise, bandwidth is likely to be the performance bottleneck during the backup and restore process. @@ -122,6 +122,36 @@ The following are the limitations of using BR for backup and restoration: - When BR restores data to the upstream cluster of TiCDC/Drainer, TiCDC/Drainer cannot replicate the restored data to the downstream. - BR supports operations only between clusters with the same [`new_collations_enabled_on_first_bootstrap`](/character-set-and-collation.md#collation-support-framework) value because BR only backs up KV data. If the cluster to be backed up and the cluster to be restored use different collations, the data validation fails. Therefore, before restoring a cluster, make sure that the switch value from the query result of the `select VARIABLE_VALUE from mysql.tidb where VARIABLE_NAME='new_collation_enabled';` statement is consistent with that during the backup process. +### Compatibility + +The compatibility issues of BR and the TiDB cluster are divided into the following categories: + ++ The KV format might change when some features are enabled or disabled. If these features are not consistently enabled or disabled during backup and restore, compatibility issues might occur. ++ Some versions of BR are not compatible with the interface of the TiDB cluster. + +These features are as follows: + +| Features | Related issues | Solutions | +| ---- | ---- | ----- | +| Clustered index | [#565](https://github.com/pingcap/br/issues/565) | Make sure that the value of the `tidb_enable_clustered_index` global variable during restore is consistent with that during backup. Otherwise, data inconsistency might occur, such as `default not found` and inconsistent data index. | +| New collation | [#352](https://github.com/pingcap/br/issues/352) | Make sure that the value of the `new_collations_enabled_on_first_bootstrap` variable is consistent with that during backup. Otherwise, inconsistent data index might occur and checksum might not pass. | +| TiCDC enabled on the restore cluster | [#364](https://github.com/pingcap/br/issues/364#issuecomment-646813965) | Currently, TiKV cannot push down the BR-ingested SST files to TiCDC. Therefore, you need to disable TiCDC when using BR to restore data. | + +However, even after you have ensured that the above features are consistently enabled or disabled during backup and restore, compatibility issues might still occur due to the inconsistent internal versions or inconsistent interfaces between BR and TiKV/TiDB/PD. To avoid such cases, BR have the built-in version check. + +#### Version check + +Before performing backup and restore, BR compares and checks the TiDB cluster version and the BR version. If there is a major-version inconsistency (for example, BR v4.x and TiDB v5.x), BR prompts a reminder to exit. To forcibly skip the version check, you can set `--check-requirements=false`. + +Note that skipping the version check might introduce incompatibility. The version compatibility information between BR and TiDB versions are as follows: + +| Backup version (vertical) \ Restore version (horizontal) | BR nightly or TiDB nightly | BR v5.0 or TiDB v5.0| BR v4.0 or TiDB v4.0 | +| ---- | ---- | ---- | ---- | +| **BR nightly or TiDB nightly** | ✅ | ✅ | ✅ | +| **BR v5.0 or TiDB v5.0** | ✅ | ✅ | ✅ +| **BR v4.0 or TiDB v4.0** | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ❌(If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | +| **BR nightly / v5.0 or TiDB v4.0** | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | + ### Minimum machine configuration required for running BR The minimum machine configuration required for running BR is as follows: @@ -130,7 +160,7 @@ The minimum machine configuration required for running BR is as follows: | --- | --- | --- | --- | | 1 core | 4 GB | HDD | Gigabit network card | -In general scenarios (less than 1000 tables for backup and restore), the CPU consumption of BR at runtime does not exceed 200%, and the memory consumption does not exceed 1 GB. However, when backing up and restoring a large number of tables, BR might consume more than 3 GB of memory. In a test of backing up 24000 tables, BR consumes about 2.7 GB of memory, and the CPU consumption remains below 100%. +In general scenarios (less than 1000 tables for backup and restore), the CPU consumption of BR at runtime does not exceed 200%, and the memory consumption does not exceed 4 GB. However, when backing up and restoring a large number of tables, BR might consume more than 4 GB of memory. In a test of backing up 24000 tables, BR consumes about 2.7 GB of memory, and the CPU consumption remains below 100%. ### Best practices From 5b97642640e6cbd35ed7d49b60753a68d2af0ec9 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 12 Apr 2021 18:10:37 +0800 Subject: [PATCH 2/6] reword --- br/backup-and-restore-tool.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 5eb383c50b60a..135c30f2b42fd 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -145,12 +145,12 @@ Before performing backup and restore, BR compares and checks the TiDB cluster ve Note that skipping the version check might introduce incompatibility. The version compatibility information between BR and TiDB versions are as follows: -| Backup version (vertical) \ Restore version (horizontal) | BR nightly or TiDB nightly | BR v5.0 or TiDB v5.0| BR v4.0 or TiDB v4.0 | +| Backup version (vertical) \ Restore version (horizontal) | Use BR nightly to restore TiDB nightly | Use BR v5.0 to restore TiDB v5.0| Use BR v4.0 to restore TiDB v4.0 | | ---- | ---- | ---- | ---- | -| **BR nightly or TiDB nightly** | ✅ | ✅ | ✅ | -| **BR v5.0 or TiDB v5.0** | ✅ | ✅ | ✅ -| **BR v4.0 or TiDB v4.0** | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ❌(If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | -| **BR nightly / v5.0 or TiDB v4.0** | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | +| Use BR nightly to back up TiDB nightly | ✅ | ✅ | ✅ | +| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ✅ +| Use BR v4.0 to back up TiDB v4.0 | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ❌(If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | +| Use BR nightly or v5.0 to back up TiDB v4.0 | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ### Minimum machine configuration required for running BR From f9a92ccb1f12250f6006147bc3c353ebbaec1f16 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 13 Apr 2021 11:19:17 +0800 Subject: [PATCH 3/6] correct br compatibility table --- br/backup-and-restore-tool.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 135c30f2b42fd..6d3a8bbf75974 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -147,9 +147,9 @@ Note that skipping the version check might introduce incompatibility. The versio | Backup version (vertical) \ Restore version (horizontal) | Use BR nightly to restore TiDB nightly | Use BR v5.0 to restore TiDB v5.0| Use BR v4.0 to restore TiDB v4.0 | | ---- | ---- | ---- | ---- | -| Use BR nightly to back up TiDB nightly | ✅ | ✅ | ✅ | -| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ✅ -| Use BR v4.0 to back up TiDB v4.0 | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ❌(If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | +| Use BR nightly to back up TiDB nightly | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | +| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) +| Use BR v4.0 to back up TiDB v4.0 | ✅ | ✅ | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | | Use BR nightly or v5.0 to back up TiDB v4.0 | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ### Minimum machine configuration required for running BR From 371a9e0270048553da65bdb8ded5ebc9f24a8021 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 13 Apr 2021 18:07:34 +0800 Subject: [PATCH 4/6] Update br/backup-and-restore-tool.md --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 6d3a8bbf75974..85c62b5b2298c 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -141,7 +141,7 @@ However, even after you have ensured that the above features are consistently en #### Version check -Before performing backup and restore, BR compares and checks the TiDB cluster version and the BR version. If there is a major-version inconsistency (for example, BR v4.x and TiDB v5.x), BR prompts a reminder to exit. To forcibly skip the version check, you can set `--check-requirements=false`. +Before performing backup and restore, BR compares and checks the TiDB cluster version and the BR version. If there is a major-version mismatch (for example, BR v4.x and TiDB v5.x), BR prompts a reminder to exit. To forcibly skip the version check, you can set `--check-requirements=false`. Note that skipping the version check might introduce incompatibility. The version compatibility information between BR and TiDB versions are as follows: From 4e9d357c068ea343fbf6342e475795392b757085 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 16 Apr 2021 10:54:15 +0800 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Lilian Lee --- br/backup-and-restore-tool.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 85c62b5b2298c..04677fc0c8059 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -134,10 +134,10 @@ These features are as follows: | Features | Related issues | Solutions | | ---- | ---- | ----- | | Clustered index | [#565](https://github.com/pingcap/br/issues/565) | Make sure that the value of the `tidb_enable_clustered_index` global variable during restore is consistent with that during backup. Otherwise, data inconsistency might occur, such as `default not found` and inconsistent data index. | -| New collation | [#352](https://github.com/pingcap/br/issues/352) | Make sure that the value of the `new_collations_enabled_on_first_bootstrap` variable is consistent with that during backup. Otherwise, inconsistent data index might occur and checksum might not pass. | +| New collation | [#352](https://github.com/pingcap/br/issues/352) | Make sure that the value of the `new_collations_enabled_on_first_bootstrap` variable is consistent with that during backup. Otherwise, inconsistent data index might occur and checksum might fail to pass. | | TiCDC enabled on the restore cluster | [#364](https://github.com/pingcap/br/issues/364#issuecomment-646813965) | Currently, TiKV cannot push down the BR-ingested SST files to TiCDC. Therefore, you need to disable TiCDC when using BR to restore data. | -However, even after you have ensured that the above features are consistently enabled or disabled during backup and restore, compatibility issues might still occur due to the inconsistent internal versions or inconsistent interfaces between BR and TiKV/TiDB/PD. To avoid such cases, BR have the built-in version check. +However, even after you have ensured that the above features are consistently enabled or disabled during backup and restore, compatibility issues might still occur due to the inconsistent internal versions or inconsistent interfaces between BR and TiKV/TiDB/PD. To avoid such cases, BR has the built-in version check. #### Version check From 25cc992e27721ab550adec35c5fa6b4489e331ee Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 16 Apr 2021 10:55:13 +0800 Subject: [PATCH 6/6] Update br/backup-and-restore-tool.md Co-authored-by: Lilian Lee --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 04677fc0c8059..2f716e146a2cd 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -126,8 +126,8 @@ The following are the limitations of using BR for backup and restoration: The compatibility issues of BR and the TiDB cluster are divided into the following categories: -+ The KV format might change when some features are enabled or disabled. If these features are not consistently enabled or disabled during backup and restore, compatibility issues might occur. + Some versions of BR are not compatible with the interface of the TiDB cluster. ++ The KV format might change when some features are enabled or disabled. If these features are not consistently enabled or disabled during backup and restore, compatibility issues might occur. These features are as follows: