Cluster restore stuck after completing DDL jobs

[rachitmanit]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   Motive is to restore full (entire cluster) backup within same cluster after dropping all user databases.

• Completed a cluster backup to S3:// based store (br backup full).
• drop all tables except INFORMATION_SCHEMA, METRICS_SCHEMA, PERFORMANCE_SCHEMA and mysql
• All the components (TiDB, TiKV, PD) are still running. No change in state.

2. What did you expect to see?
   Expected cluster restore successfully.

3. What did you see instead?

Stuck with error in logs: "splitting regions failed retry, rpc error"

• Triggered br restore full, and restore got stuck with log:

• Restore progress is also stuck at 0.00%:

4. What version of BR and TiDB/TiKV/PD are you using?

[kennytm]

Hi @rachitmanit.

The error message is "Canceled", which suggests either there is another error before 02:23:53 or BR has received a SIGHUP signal. Are there any other WARN and ERROR logs around "splitting regions failed retry, rpc error"?

Thanks!

[rachitmanit]

No @kennytm. There was no other error logs as i looked upon every component's log during the same time.

This is happening each time i apply restore on cluster.
After dropping all added tables and waiting for GC to complete delete ranges tasks (if any), I change tikv gc lifetime to 24h and then i trigger cluster restore.

[rachitmanit]

Restore successful on freshly created cluster but not old one.
Looks like the recommendation to restore Full backup on new cluster still holds.

You can add that

"Full/cluster restore can only be done on new clusters"

As Table/Database restore is working on existing cluster too.

[3pointer]

The error is strange, could you provide complete logs of restore full cluster, this may help us to find the root cause why the restore failed. Thanks.

[rachitmanit]

@3pointer above observations were from the test setup which is destroyed now.

[3pointer]

@rachitmanit this issue has been solved in #266, v4.0.2 or later version of br will solve this problem. I'm going to close this issue, Thanks for the bug report.