[Rebalancer] support partition rebalancer #5010

vagetablechicken · 2020-12-02T10:51:26Z

Proposed changes

RebalancerType could be configured via Config.rebalancer_type(BeLoad, Partition).
PartitionRebalancer is based on TwoDimensionalGreedyAlgo.
Two dims of Doris should be cluster & partition. And we only consider about the replica count, do not consider replica size.
#4845 for further details.

Types of changes

New feature (non-breaking change which adds functionality)

Checklist

I have create an issue on (Fix [Proposal] suppoort PartitionRebalancer by using TwoDimensionalGreedyAlgo #4845), and have described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If this change need a document change, I have updated the document

fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java

fe/fe-core/src/main/java/org/apache/doris/common/Config.java

docs/zh-CN/administrator-guide/operation/tablet-repair-and-balance.md

fe/fe-core/src/main/java/org/apache/doris/clone/MovesInProgressCache.java

fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java

fe/fe-core/src/main/java/org/apache/doris/clone/TwoDimensionalGreedyAlgo.java

fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java

kangkaisen

+1

killxdcj · 2020-12-28T06:47:27Z

Will there be any balance-related modifications in the future? The new balance strategy also needs to be considered when create new partition or restore data, but I don't see the relevant changes.

vagetablechicken · 2020-12-28T08:51:44Z

Will there be any balance-related modifications in the future? The new balance strategy also needs to be considered when create new partition or restore data, but I don't see the relevant changes.

It's only a rebalacer strategy, not include the creation or some else.

killxdcj · 2020-12-28T09:02:55Z

Will there be any balance-related modifications in the future? The new balance strategy also needs to be considered when create new partition or restore data, but I don't see the relevant changes.

It's only a rebalacer strategy, not include the creation or some else.

If partition create / restore does not consider the balance strategy of the cluster, then there will be such a situation: Tablets have just been created, and then balancer finds the cluster is imbalance, and then it will do a balance on the newly created tablets.

killxdcj · 2020-12-28T09:08:50Z

Will there be any balance-related modifications in the future? The new balance strategy also needs to be considered when create new partition or restore data, but I don't see the relevant changes.

It's only a rebalacer strategy, not include the creation or some else.

If partition create / restore does not consider the balance strategy of the cluster, then there will be such a situation: Tablets have just been created, and then balancer finds the cluster is imbalance, and then it will do a balance on the newly created tablets.

Another question is has the new balance strategy been verified by the production environment? We also have similar requirements and we have implemented a similar balance strategy, but now we want to switch to this community implementation.

vagetablechicken · 2020-12-28T09:37:09Z

Another question is has the new balance strategy been verified by the production environment? We also have similar requirements and we have implemented a similar balance strategy, but now we want to switch to this community implementation.

Haven't yet, only in test env. And the production envs may have some differences. If you like, plz help me to test it. BTW, our team plans to use it in production env next month.

fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java

morningman · 2020-12-28T15:51:26Z

fe/fe-core/src/main/java/org/apache/doris/clone/TwoDimensionalGreedyRebalanceAlgo.java

+        // 1. Every partition is balanced(maxPartitionSkew<=1) and any move will unbalance a partition, so there
+        // is no potential for the greedy algorithm to balance the cluster.
+        // 2. Every partition is balanced(maxPartitionSkew<=1) and the cluster as a whole is balanced(maxBeSkew<=1).
+        if (maxPartitionSkew == 0L || (maxPartitionSkew <= 1L && maxBeSkew <= 1L)) {


Is 1 too strict here?

The algo should get accurate calculation results. And I think there is no reason yet to relax the condition.

fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java

morningman · 2020-12-29T02:18:45Z

fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java

+    }
+
+    // Move completed: fromBe doesn't have a replica and toBe has a replica
+    private boolean checkMoveCompleted(TabletMove move) {


The new replica will be added to the tablet at the beginning of clone task with state CLONE.
And the "fromBE" replica may be dropped for other reason such as "data broken on BE".

So are you sure !bes.contains(move.fromBe) && bes.contains(move.toBe) can be treated as move complete?

It's hard to define, but when !bes.contains(move.fromBe) && bes.contains(move.toBe), the move is useless. So why not to treat it as completed.

fe/fe-core/src/main/java/org/apache/doris/clone/TwoDimensionalGreedyRebalanceAlgo.java

morningman · 2020-12-29T03:27:17Z

fe/fe-core/src/main/java/org/apache/doris/clone/TwoDimensionalGreedyRebalanceAlgo.java

+            }
+
+            LOG.debug("min_loaded_be: {}, max_loaded_be: {}", minLoadedBe, maxLoadedBe);
+            if (minLoadedBe.equals(maxLoadedBe)) {


If we use PICK_FIRST, in the case of fewer candidate sets, will there be a high probability of the same?

Now, we only use PICK_FIRST in UTs, to get the results that can be expected.

morningman

LGTM

RebalancerType could be configured via Config.rebalancer_type(BeLoad, Partition). PartitionRebalancer is based on TwoDimensionalGreedyAlgo. Two dims of Doris should be cluster & partition. And we only consider about the replica count, do not consider replica size. apache#4845 for further details.

…che#5010)

vagetablechicken added 4 commits December 2, 2020 18:44

[] partition rebalancer

ac63a03

[] comments & rename

dc5361d

[] fix config & add docs

f81a67b

[] comments & logs fix

700a606

morningman added area/catalog Issues or PRs related to catalog management kind/improvement labels Dec 5, 2020

gengjun-git reviewed Dec 9, 2020

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java Show resolved Hide resolved

kangkaisen reviewed Dec 21, 2020

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/common/Config.java Outdated Show resolved Hide resolved

kangkaisen reviewed Dec 21, 2020

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/common/Config.java Outdated Show resolved Hide resolved

fix by review

db0c06d

vagetablechicken requested a review from kangkaisen December 22, 2020 02:41

kangkaisen reviewed Dec 23, 2020

View reviewed changes

vagetablechicken added 2 commits December 24, 2020 16:20

[] fixed by review

6a267be

[] fix

d0ee249

vagetablechicken requested a review from kangkaisen December 24, 2020 08:25

[] fixed by review

5ef852d

kangkaisen previously approved these changes Dec 25, 2020

View reviewed changes

kangkaisen added the approved Indicates a PR has been approved by one committer. label Dec 25, 2020

morningman reviewed Dec 29, 2020

View reviewed changes

[] fix by review

29e790c

vagetablechicken dismissed kangkaisen’s stale review via 29e790c December 29, 2020 04:13

morningman approved these changes Dec 31, 2020

View reviewed changes

morningman merged commit d7a584a into apache:master Dec 31, 2020

yangzhg mentioned this pull request Feb 9, 2021

Release Notes 0.14.0 #5374

Closed

w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Dec 26, 2025

[fix](compaction) change time series threshold from 2000 to 1000 (apa…

56b8e09

…che#5010)

[Rebalancer] support partition rebalancer #5010

[Rebalancer] support partition rebalancer #5010

Uh oh!

Conversation

vagetablechicken commented Dec 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Types of changes

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangkaisen left a comment

Choose a reason for hiding this comment

Uh oh!

killxdcj commented Dec 28, 2020

Uh oh!

vagetablechicken commented Dec 28, 2020

Uh oh!

killxdcj commented Dec 28, 2020

Uh oh!

killxdcj commented Dec 28, 2020

Uh oh!

vagetablechicken commented Dec 28, 2020

Uh oh!

Uh oh!

Uh oh!

morningman Dec 28, 2020

Choose a reason for hiding this comment

Uh oh!

vagetablechicken Dec 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morningman Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

vagetablechicken Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

morningman Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

vagetablechicken Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vagetablechicken commented Dec 2, 2020 •

edited

Loading

vagetablechicken Dec 29, 2020 •

edited

Loading