Lightning ResolveDuplicateRows casue checksum mismatch

[sleepymole]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Table schema:

CREATE TABLE `t` (
     `id` int(11) NOT NULL,
     `name` varchar(255) DEFAULT NULL,
     PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
     UNIQUE KEY `uni_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;

Enable duplicate-resolution

[tikv-importer]
backend = "local"
duplicate-resolution = "remove"

Use tidb-lightning to import data twice.

For the first time, import data with unique key conflicts.

id,name
1,"aaa01"
2,"aaa02"
3,"aaa03"
4,"aaa04"
5,"aaa04"

Conflict errors are detected as expected.

Import Data Error Summary: 
+---+---------------------+-------------+------------------------------------- ~
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                     ~
+---+---------------------+-------------+------------------------------------- ~
| 1 | Unique Key Conflict |          2 | `lightning_task_info`.`conflict_erro ~
+---+---------------------+-------------+------------------------------------- ~

For the second time, truncate the table and import data with no unique key conflicts.

id,name
1,"aaa01"
2,"aaa02"
3,"aaa03"
4,"aaa04"
5,"aaa05"

No conflict error is detected. But checksum mismatch is reported.

2. What did you expect to see? (Required)

After the table is truncated, import should complete without any error.

3. What did you see instead (Required)

checksum mismatch error is reported.

4. What is your TiDB version? (Required)

master

[sleepymole]

In processProcess, when duplicate-resolution is enabled, lightning will always call ResolveDuplicateRows even if there is no duplicate data detected.

tidb/br/pkg/lightning/restore/table_restore.go

Line 815 in 5cb64fd

if err = rc.backend.ResolveDuplicateRows(ctx, tr.encTable, tr.tableName, rc.cfg.TikvImporter.DuplicateResolution); err != nil {

In normal cases, it is ok. However, if we import data with duplicates to the table, then truncate the table and import data with no duplicates to the same table, we may encounter checksum mismatch error. This is because ResolveDuplicateRows will accidentally delete duplicate data that is detected during the first time we import.