Skip to content
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
This repository was archived by the owner on Jul 24, 2024. It is now read-only.

br don't work well when restoring region file to a new cluster #380

@2983777417

Description

@2983777417

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
    If possible, provide a recipe for reproducing the error.

we want restore all databases data from cluster A(v4.0.0) to new cluster B(v4.0.1)

  1. backup all databases data from cluster A.

    ./br backup full --pd "192.168.15.101:2379" -s "local:///home/tidb/backup"
  2. scp all the sst files from tikvA's nodes to tikvB's nodes

  3. restore all data region to another cluster B ( all of databases in the B ,has been dropped before restoring)

    ./br restore full --pd "192.168.15.213:2379" --storage "local:///home/tidb/backup" --ratelimit 100 --log-file restorefull.log
  4. after waiting for half an hour

[2020/06/23 19:13:56.195 +08:00] [WARN] [split.go:132] ["splitting regions failed, retry it"] [error="rpc error: code = Canceled desc = Cancelled"] [keys="[\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b6\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b9\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b<\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b?\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bB\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bE\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bH\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bK\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bN\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bQ\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bT\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bW\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bZ\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b]\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b`\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bc\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bf\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bi\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bl\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bo\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001br\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bu\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001bx\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b{\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b~\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001b\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0002\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0005\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0008\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u000b\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u000e\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0011\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0014\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u0017\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u001a\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\u001d\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c \",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c#\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c&\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c)\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c,\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c/\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c2\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c5\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c8\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c;\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c>\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cA\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cD\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cG\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cJ\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cM\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cP\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cS\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cV\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cY\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\\\\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c_\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cb\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ce\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ch\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ck\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cn\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cq\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001ct\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cw\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001cz\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c}\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u001c\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000
[2020/06/23 19:13:56.211 +08:00] [ERROR] [restore.go:283] ["split regions failed"] [error="rpc error: code = Canceled desc = Cancelled"] [errorVerbose="rpc error: code = Canceled desc = Cancelled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/br/pkg/restore.(*RegionSplitter).Split\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/restore/split.go:143\ngithub.com/pingcap/br/pkg/restore.SplitRanges\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/restore/util.go:313\ngithub.com/pingcap/br/pkg/task.RunRestore\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/task/restore.go:281\ngithub.com/pingcap/br/cmd.runRestoreCommand\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:21\ngithub.com/pingcap/br/cmd.newFullRestoreCommand.func1\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/main.go:54\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/log.Error\n\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\ngithub.com/pingcap/br/pkg/task.RunRestore\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/pkg/task/restore.go:283\ngithub.com/pingcap/br/cmd.runRestoreCommand\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:21\ngithub.com/pingcap/br/cmd.newFullRestoreCommand.func1\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/cmd/restore.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0/go/src/github.com/pingcap/br/main.go:54\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
[2020/06/23 19:13:56.963 +08:00] [INFO] [client.go:134] ["Restore client closed"]
[2020/06/23 19:13:57.043 +08:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"] [error="context canceled"]
[2020/06/23 19:13:57.043 +08:00] [INFO] [manager.go:239] ["etcd session is done, creates a new one"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"]
[2020/06/23 19:13:57.043 +08:00] [INFO] [manager.go:243] ["break campaign loop, NewSession failed"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ngithub.com/pingcap/tidb/owner.contextDone\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200512142211-0623e4d44563/owner/manager.go:371\ngithub.com/pingcap/tidb/owner.NewSession\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200512142211-0623e4d44563/owner/manager.go:142\ngithub.com/pingcap/tidb/owner.(*ownerManager).campaignLoop\n\t/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200512142211-0623e4d44563/owner/manager.go:241\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/06/23 19:13:57.072 +08:00] [INFO] [manager.go:292] ["revoke session"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager 0f7a8b15-6842-4f48-b6db-3a9dad89b2f5"] []
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 1, tp general"] ["take time"=9.687µs]
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 2, tp add index"] ["take time"=17.123µs]
[2020/06/23 19:13:57.085 +08:00] [INFO] [delete_range.go:123] ["[ddl] closing delRange"]
[2020/06/23 19:13:57.085 +08:00] [INFO] [session_pool.go:85] ["[ddl] closing sessionPool"]
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=0f7a8b15-6842-4f48-b6db-3a9dad89b2f5] ["take time"=121.337306ms]
[2020/06/23 19:13:57.085 +08:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=0f7a8b15-6842-4f48-b6db-3a9dad89b2f5]
[2020/06/23 19:13:57.196 +08:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=232.939018ms]
[2020/06/23 19:13:57.197 +08:00] [INFO] [collector.go:172] ["Full restore Failed summary : total restore files: 0, total success: 0, total failed: 0"] ["split region"=33m38.360532887s] ["restore ranges"=2387]

4.cluster B just create many tables but none data

```
Full restore <................................................................................................................................> 0.00%  ` there are so many this.
```
  1. What did you expect to see?

br restore all databases froe cluster A to cluster B.

  1. What did you see instead?

1.split region failed .
2.cluster B databases are emtpy.
3.full restore progress bar always be 0.00%

  1. What version of BR and TiDB/TiKV/PD are you using?

br : Release Version: v4.0.0
tidb-server:Release Version: v4.0.1
tikv-server:Release Version: 4.0.1
pd-server:Release Version: v4.0.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions