Lightning: Table Empty Check Cannot Find Imported Dirty Data on Previous Failed Imports

[dsdashun]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Import the data using Lightning, and halt it half-way during importing the data records, but before importing any indices .
2. Import the data a second time.

2. What did you expect to see? (Required)

The second import should not be allowed (if check-requirements is turned on ), because the table has actually imported some record data in the previous import.

3. What did you see instead (Required)

• The second import can be executed successfully
• If checking the record count, we can find inconsistencies on these two statements:

SELECT COUNT(*) FROM t;
SELECT COUNT(*) FROM t USE INDEX();

Seems the dirty data is still inside the table, and it cannot be cleared unless dropping the table and re-import again.

4. What is your TiDB version? (Required)

Latest master version

[dsdashun]

/component lightning

[dsdashun]

/severity moderate

[lance6716]

When lightning failed the first time, it should modify the checkpoint to an invalid state so the second import will report error at the beginning. Does this behaviour still work?

[dsdashun]

When lightning failed the first time, it should modify the checkpoint to an invalid state so the second import will report error at the beginning. Does this behaviour still work?

It should have no problem. Because it is not related to the the checkpoint implementations. It is just a pre-check refinement. The current mechanism on checkpoints still applies.