Please answer these questions before submitting your issue. Thanks!
we want restore all databases data from cluster A(v4.0.0) to new cluster B(v4.0.1)
[2020/06/23 19:13:56.195 +08:00] [WARN] [split.go:132] ["splitting regions failed, retry it"] [error="rpc error: code = Canceled desc = Cancelled"] [keys="[\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b6\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b9\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b<\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b?\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bB\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bE\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bH\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bK\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bN\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bQ\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bT\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bW\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bZ\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b]\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b`\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bc\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bf\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bi\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bl\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bo\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001br\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bu\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bx\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b{\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b~\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0002\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0005\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0008\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u000b\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u000e\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0011\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0014\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0017\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u001a\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u001d\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c \",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c#\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c&\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c)\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c,\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c/\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c2\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c5\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c8\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c;\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c>\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cA\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cD\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cG\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cJ\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cM\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cP\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cS\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cV\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cY\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\\\\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c_\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cb\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ce\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ch\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ck\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cn\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cq\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ct\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cw\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cz\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c}\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000
[2020/06/23 19:13:56.211 +08:00] [ERROR] [restore.go:283] ["split regions failed"] [error="rpc error: code = Canceled desc = Cancelled"] [errorVerbose="rpc error: code = Canceled desc = Cancelled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/br/pkg/restore.(*RegionSplitter).Split\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/restore/split.go:143\ngithub.com/pingcap/br/pkg/restore.SplitRanges\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/restore/util.go:313\ngithub.com/pingcap/br/pkg/task.RunRestore\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/task/restore.go:281\ngithub.com/pingcap/br/cmd.runRestoreCommand\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:21\ngithub.com/pingcap/br/cmd.newFullRestoreCommand.func1\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/main.go:54\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/log.Error\n\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\ngithub.com/pingcap/br/pkg/task.RunRestore\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/task/restore.go:283\ngithub.com/pingcap/br/cmd.runRestoreCommand\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:21\ngithub.com/pingcap/br/cmd.newFullRestoreCommand.func1\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/main.go:54\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
[2020/06/23 19:13:56.963 +08:00] [INFO] [client.go:134] ["Restore client closed"]
[2020/06/23 19:13:57.043 +08:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"] [error="context canceled"]
[2020/06/23 19:13:57.043 +08:00] [INFO] [manager.go:239] ["etcd session is done, creates a new one"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"]
[2020/06/23 19:13:57.043 +08:00] [INFO] [manager.go:243] ["break campaign loop, NewSession failed"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/tidb/owner.contextDone\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200512142211-0623e4d44563/owner/manager.go:371\ngithub.com/pingcap/tidb/owner.NewSession\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200512142211-0623e4d44563/owner/manager.go:142\ngithub.com/pingcap/tidb/owner.(*ownerManager).campaignLoop\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200512142211-0623e4d44563/owner/manager.go:241\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/06/23 19:13:57.072 +08:00] [INFO] [manager.go:292] ["revoke session"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"] []
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 1, tp general"] ["take time"=9.687µs]
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 2, tp add index"] ["take time"=17.123µs]
[2020/06/23 19:13:57.085 +08:00] [INFO] [delete_range.go:123] ["[ddl] closing delRange"]
[2020/06/23 19:13:57.085 +08:00] [INFO] [session_pool.go:85] ["[ddl] closing sessionPool"]
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=0f7a8b15-6842-4f48-b6db-3a9dad89b2f5] ["take time"=121.337306ms]
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=0f7a8b15-6842-4f48-b6db-3a9dad89b2f5]
[2020/06/23 19:13:57.196 +08:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=232.939018ms]
[2020/06/23 19:13:57.197 +08:00] [INFO] [collector.go:172] ["Full restore Failed summary : total restore files: 0, total success: 0, total failed: 0"] ["split region"=33m38.360532887s] ["restore ranges"=2387]
```
Full restore <................................................................................................................................> 0.00% ` there are so many this.
```
br restore all databases froe cluster A to cluster B.
1.split region failed .
2.cluster B databases are emtpy.
3.full restore progress bar always be 0.00%
Please answer these questions before submitting your issue. Thanks!
If possible, provide a recipe for reproducing the error.
we want restore all databases data from cluster A(v4.0.0) to new cluster B(v4.0.1)
backup all databases data from cluster A.
scp all the sst files from tikvA's nodes to tikvB's nodes
restore all data region to another cluster B ( all of databases in the B ,has been dropped before restoring)
after waiting for half an hour
4.cluster B just create many tables but none data
br restore all databases froe cluster A to cluster B.
1.split region failed .
2.cluster B databases are emtpy.
3.full restore progress bar always be 0.00%
br : Release Version: v4.0.0
tidb-server:Release Version: v4.0.1
tikv-server:Release Version: 4.0.1
pd-server:Release Version: v4.0.1