From c9a279cf247a1afa555e50acd8b6c2c7174f5df7 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Thu, 18 Nov 2021 10:53:18 +0800 Subject: [PATCH 1/5] Update tidb-lightning-faq.md --- tidb-lightning/tidb-lightning-faq.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index 165b881b0d85f..b2e7298b6890f 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -222,22 +222,21 @@ Try the latest version! Maybe there is new speed improvement. ## `checksum failed: checksum mismatched remote vs local` -**Cause**: The checksum of a table in the local data source and the remote imported database differ. This error has several deeper reasons: +**Cause**: The checksum of a table in the local data source and the remote imported database differ. This error has several deeper reasons. You can further locate the reason by checking the log that contains `checksum mismatched`. -1. The table might already have data before. These old data can affect the final checksum. +The lines that contain `checksum mismatched` provides the information `total_kvs: x vs y`, where `x` indicates the number of key-value pairs (KV pairs) calculated by the import cluster after the import is completed, and `y` indicates the number of key-value pairs generated by the local data source. -2. If the remote checksum is 0, which means nothing is imported, it is possible that the cluster is too hot and fails to take in any data. - -3. If the data is mechanically generated, ensure it respects the constrains of the table: - - * `AUTO_INCREMENT` columns need to be positive, and do not contain the value "0". - * The UNIQUE and PRIMARY KEYs must have no duplicated entries. - -4. If TiDB Lightning has failed before and was not properly restarted, a checksum mismatch may happen due to data being out-of-sync. +- If `x` is greater, it means that there are more KV pairs in the import cluster. + - It is possible that this table is populated before the import and therefore affects the data checksum. It is also possible that TiDB Lightning has previously failed and shut down, but did not restart correctly. +- If `y` is greater, it means that there are more KV pairs in the local data source. + - If the checksum of the target database is all 0, it means that no import has occurred. It is possible that the cluster is too busy to receive any data. + - It is possible that the exported data contains duplicate data, such as unique and primary keys (UNIQUE and PRIMARY KEYs) with duplicate values, or that the downstream table structure is case-insensitive while the data is case-sensitive. +- Other possible reasons + - If the data source is machine-generated and not backed up by Dumpling, make sure the data conforms to the table limits, e.g., the AUTO_INCREMENT column needs to be positive and not 0. **Solutions**: -1. Delete the corrupted data using `tidb-lightning-ctl`, and restart TiDB Lightning to import the affected tables again. +1. Delete the corrupted data using `tidb-lightning-ctl`, check the table structure and the data, and restart TiDB Lightning to import the affected tables again. {{< copyable "shell-regular" >}} From 82ae484c5cc35c9a390215b16bbf06337469f74e Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 18 Nov 2021 13:55:01 +0800 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: glorv --- tidb-lightning/tidb-lightning-faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index b2e7298b6890f..c981e81d0a9b6 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -226,8 +226,8 @@ Try the latest version! Maybe there is new speed improvement. The lines that contain `checksum mismatched` provides the information `total_kvs: x vs y`, where `x` indicates the number of key-value pairs (KV pairs) calculated by the import cluster after the import is completed, and `y` indicates the number of key-value pairs generated by the local data source. -- If `x` is greater, it means that there are more KV pairs in the import cluster. - - It is possible that this table is populated before the import and therefore affects the data checksum. It is also possible that TiDB Lightning has previously failed and shut down, but did not restart correctly. +- If `x` is greater, it means that there are more KV pairs in the target cluster. + - It is possible that this table is not empty before the import and therefore affects the data checksum. It is also possible that TiDB Lightning has previously failed and shut down, but did not restart correctly. - If `y` is greater, it means that there are more KV pairs in the local data source. - If the checksum of the target database is all 0, it means that no import has occurred. It is possible that the cluster is too busy to receive any data. - It is possible that the exported data contains duplicate data, such as unique and primary keys (UNIQUE and PRIMARY KEYs) with duplicate values, or that the downstream table structure is case-insensitive while the data is case-sensitive. From c9dbcb3d4fe934f1bbf74b312b34c19c6b039597 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Nov 2021 20:18:43 +0800 Subject: [PATCH 3/5] Update tidb-lightning/tidb-lightning-faq.md Co-authored-by: lance6716 --- tidb-lightning/tidb-lightning-faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index c981e81d0a9b6..aa4b2bab22746 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -224,7 +224,7 @@ Try the latest version! Maybe there is new speed improvement. **Cause**: The checksum of a table in the local data source and the remote imported database differ. This error has several deeper reasons. You can further locate the reason by checking the log that contains `checksum mismatched`. -The lines that contain `checksum mismatched` provides the information `total_kvs: x vs y`, where `x` indicates the number of key-value pairs (KV pairs) calculated by the import cluster after the import is completed, and `y` indicates the number of key-value pairs generated by the local data source. +The lines that contain `checksum mismatched` provides the information `total_kvs: x vs y`, where `x` indicates the number of key-value pairs (KV pairs) calculated by the target cluster after the import is completed, and `y` indicates the number of key-value pairs generated by the local data source. - If `x` is greater, it means that there are more KV pairs in the target cluster. - It is possible that this table is not empty before the import and therefore affects the data checksum. It is also possible that TiDB Lightning has previously failed and shut down, but did not restart correctly. From f808329cd8adafac343a88af672d0e32c975aa79 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Nov 2021 20:18:54 +0800 Subject: [PATCH 4/5] Update tidb-lightning/tidb-lightning-faq.md Co-authored-by: lance6716 --- tidb-lightning/tidb-lightning-faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index aa4b2bab22746..9b5437654fdc2 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -230,7 +230,7 @@ The lines that contain `checksum mismatched` provides the information `total_kvs - It is possible that this table is not empty before the import and therefore affects the data checksum. It is also possible that TiDB Lightning has previously failed and shut down, but did not restart correctly. - If `y` is greater, it means that there are more KV pairs in the local data source. - If the checksum of the target database is all 0, it means that no import has occurred. It is possible that the cluster is too busy to receive any data. - - It is possible that the exported data contains duplicate data, such as unique and primary keys (UNIQUE and PRIMARY KEYs) with duplicate values, or that the downstream table structure is case-insensitive while the data is case-sensitive. + - It is possible that the exported data contains duplicate data, such as UNIQUE and PRIMARY KEYs with duplicate values, or that the downstream table structure is case-insensitive while the data is case-sensitive. - Other possible reasons - If the data source is machine-generated and not backed up by Dumpling, make sure the data conforms to the table limits, e.g., the AUTO_INCREMENT column needs to be positive and not 0. From c6db9b4e69666b8444801d763291f9a8ca0474b1 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Nov 2021 21:03:01 +0800 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Liuxiaozhen12 <82579298+Liuxiaozhen12@users.noreply.github.com> --- tidb-lightning/tidb-lightning-faq.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index 9b5437654fdc2..556d438fe66ea 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -224,15 +224,15 @@ Try the latest version! Maybe there is new speed improvement. **Cause**: The checksum of a table in the local data source and the remote imported database differ. This error has several deeper reasons. You can further locate the reason by checking the log that contains `checksum mismatched`. -The lines that contain `checksum mismatched` provides the information `total_kvs: x vs y`, where `x` indicates the number of key-value pairs (KV pairs) calculated by the target cluster after the import is completed, and `y` indicates the number of key-value pairs generated by the local data source. +The lines that contain `checksum mismatched` provide the information `total_kvs: x vs y`, where `x` indicates the number of key-value pairs (KV pairs) calculated by the target cluster after the import is completed, and `y` indicates the number of key-value pairs generated by the local data source. - If `x` is greater, it means that there are more KV pairs in the target cluster. - It is possible that this table is not empty before the import and therefore affects the data checksum. It is also possible that TiDB Lightning has previously failed and shut down, but did not restart correctly. - If `y` is greater, it means that there are more KV pairs in the local data source. - If the checksum of the target database is all 0, it means that no import has occurred. It is possible that the cluster is too busy to receive any data. - - It is possible that the exported data contains duplicate data, such as UNIQUE and PRIMARY KEYs with duplicate values, or that the downstream table structure is case-insensitive while the data is case-sensitive. + - It is possible that the exported data contains duplicate data, such as the UNIQUE and PRIMARY KEYs with duplicate values, or that the downstream table structure is case-insensitive while the data is case-sensitive. - Other possible reasons - - If the data source is machine-generated and not backed up by Dumpling, make sure the data conforms to the table limits, e.g., the AUTO_INCREMENT column needs to be positive and not 0. + - If the data source is machine-generated and not backed up by Dumpling, make sure the data conforms to the table limits. For example, the AUTO_INCREMENT column needs to be positive and not 0. **Solutions**: